Elasticsearch API key authentication breaks in predictable ways: the key expired, someone invalidated it, or it never had the right privileges for the operation you're attempting. The error messages are not always obvious about which of these happened. This article walks through the API key lifecycle, the diagnostic APIs, cross-cluster keys introduced in 8.x, and practical patterns for key rotation.
Creating API Keys with the Right Privileges
API keys are created via POST /_security/api_key. The role_descriptors object defines what the key can do. If you omit role_descriptors, the key inherits a snapshot of the creating user's permissions at creation time - not a live reference. Changes to the user's roles after key creation have no effect on the key.
POST /_security/api_key
{
"name": "ingest-pipeline-key",
"expiration": "90d",
"role_descriptors": {
"ingest_writer": {
"cluster": ["monitor", "manage_ingest_pipelines"],
"indices": [
{
"names": ["logs-*"],
"privileges": ["create_doc", "create_index", "auto_configure"]
}
]
}
}
}
The response returns an id and an encoded field. The encoded value is the base64 string you pass in the Authorization: ApiKey <encoded> header. Store this at creation time - you cannot retrieve it later. The id is what you use for all subsequent management operations.
Setting expiration is optional. Without it the key lives until explicitly invalidated. In practice, never create keys without expiration for automated systems. A 90-day or 30-day window gives you a forcing function for rotation without requiring manual intervention.
Checking Key Status
When authentication fails, the first step is determining the key's current state. The GET /_security/api_key endpoint retrieves metadata about one or more keys.
GET /_security/api_key?id=VuaCfGcBCdbkQm-e5aOx
The response includes invalidated (boolean), expiration (epoch millis, if set), and creation time. If invalidated is true, someone called the invalidate API on this key. If the current time exceeds expiration, the key is expired and Elasticsearch will reject it regardless of the invalidated flag.
For bulk inspection across the cluster, use the query API keys endpoint:
POST /_security/_query/api_key
{
"query": {
"bool": {
"must": [
{ "term": { "invalidated": false } },
{ "range": { "expiration": { "lt": "now+7d" } } }
]
}
}
}
This finds all active keys expiring within the next seven days - useful for building rotation alerts. The query requires manage_api_key or manage_security cluster privileges. With only manage_own_api_key, you can query your own keys but not others.
Cross-Cluster API Keys
Elasticsearch 8.14 introduced cross-cluster API keys for connecting remote clusters using API key authentication instead of certificate-based trust. These keys are created with POST /_security/cross_cluster/api_key and have a distinct privilege model.
POST /_security/cross_cluster/api_key
{
"name": "cluster-b-replication",
"expiration": "180d",
"access": {
"replication": [
{ "names": ["archive-*"] }
],
"search": [
{ "names": ["logs-*"], "query": { "term": { "env": "prod" } } }
]
}
}
Cross-cluster keys support two access types: search (for cross-cluster search) and replication (for cross-cluster replication). They cannot be used for regular REST API calls. The GET /_security/api_key response includes a type field with value cross_cluster to distinguish them from standard rest type keys.
One pitfall: cross-cluster keys require the remote cluster to be configured with the cluster.remote.<alias>.mode: proxy transport mode. If you're still using sniff mode, API key authentication is not supported and the connection will fail silently or with opaque transport errors.
Common Error Messages and What They Mean
The error security_exception: unable to authenticate user with reason unable to find apikey with id [...] means the key ID does not exist in the security index. This happens when the key was created on a different cluster, the .security index was restored from a snapshot that predates the key, or the key was deleted after the retention period.
The error api key has been invalidated is straightforward - someone called the invalidate endpoint. Check the audit log for invalidate_apikey events to find when and by whom.
A subtler failure is action [...] is unauthorized for API key id [...]. The key itself is valid and not expired, but its role_descriptors don't grant the required privilege. This frequently happens when keys are created with narrow index patterns and the application later starts writing to indices that don't match. It also occurs when keys are derived from other keys - a child key cannot exceed the parent key's permissions.
Key Rotation Practices
Rotation means creating a new key, updating all consumers, and invalidating the old one. The ordering matters. Invalidating first causes an outage; you need overlap where both keys are valid.
A practical rotation pattern for automated systems:
- Create a new key with the same
role_descriptorsand a fresh expiration. - Deploy the new key to all consumers (configuration management, secrets manager, Kubernetes secret).
- Verify writes and reads succeed with the new key by checking application logs.
- Invalidate the old key:
DELETE /_security/api_key
{
"ids": ["old-key-id-here"]
}
For systems where zero-downtime rotation is hard (embedded devices, third-party integrations), create the replacement key well before the current one expires. A 90-day key with rotation at day 60 gives you 30 days of buffer.
The query API keys endpoint from the earlier section is the right building block for monitoring expiration. Wire it into your alerting system - flag keys expiring in the next 14 days and treat rotation as a scheduled task rather than an emergency response when authentication starts failing at 3 AM.
