The data_streams.lifecycle.retention.max setting in Elasticsearch controls the maximum retention period for data streams. It sets an upper limit on how long data can be retained in a data stream, regardless of individual index lifecycle policies.
- Default value: None (unlimited)
- Possible values: Time units (e.g., 365d, 52w, 1y)
- Recommendations: Set this value based on your data retention requirements and compliance policies.
This setting is cluster-wide and applies to all data streams. It overrides any longer retention periods specified in individual index lifecycle policies.
This setting was introduced in Elasticsearch 7.10.0.
Example Usage
To set the maximum retention period for all data streams to 1 year:
PUT /_cluster/settings
{
"persistent": {
"data_streams.lifecycle.retention.max": "365d"
}
}
Reasons for changing this setting:
- Enforce compliance with data retention policies
- Prevent accidental over-retention of data
- Manage storage costs by limiting long-term data storage
Effects of the change:
- Data older than the specified period will be deleted, even if individual policies specify longer retention
- Helps maintain consistent data retention across all data streams
Common Issues or Misuses
- Setting too short a retention period, leading to premature data loss
- Forgetting to account for this setting when designing index lifecycle policies
- Not considering the impact on existing data streams when changing this setting
Do's and Don'ts
Do's:
- Align this setting with your organization's data retention policies
- Communicate changes to all stakeholders before implementation
- Regularly review and adjust based on changing requirements
Don'ts:
- Don't set this value without considering the needs of all data streams in your cluster
- Don't ignore this setting when designing individual index lifecycle policies
- Don't change this setting without understanding its impact on existing data
Frequently Asked Questions
Q: Can I set different maximum retention periods for different data streams?
A: No, this setting applies cluster-wide. For more granular control, use individual index lifecycle policies, but remember they cannot exceed this global maximum.
Q: What happens to existing data that exceeds the new maximum retention period when I change this setting?
A: Existing data that exceeds the new maximum retention period will be deleted during the next index lifecycle management execution.
Q: Does this setting affect snapshots or backups of my data streams?
A: No, this setting only affects the live data in your cluster. Snapshots and backups are not automatically deleted based on this setting.
Q: Can I temporarily disable this setting without losing its configured value?
A: Yes, you can set it to null to disable it temporarily. When you want to re-enable it, you can set it back to its previous value.
Q: How does this setting interact with the min_age parameter in ILM policies?
A: The min_age in ILM policies determines when an action should start, while data_streams.lifecycle.retention.max sets an absolute maximum age for data. The max retention will override any longer retention periods set by ILM policies.
