The data_streams.lifecycle.retention.max setting in Elasticsearch controls the maximum retention period for data streams. It sets an upper limit on how long data can be retained in a data stream, regardless of individual index lifecycle policies.
- Default value: None (unlimited)
- Possible values: Time units (e.g., 365d, 52w, 1y)
- Recommendations: Set this value based on your data retention requirements and compliance policies.
This setting is cluster-wide and applies to all data streams. It overrides any longer retention periods specified in individual index lifecycle policies. This setting complements the `retention.default` setting.
This setting was introduced in Elasticsearch 7.10.0.
Example Usage
To set the maximum retention period for all data streams to 1 year:
PUT /_cluster/settings
{
"persistent": {
"data_streams.lifecycle.retention.max": "365d"
}
}
Reasons for changing this setting:
- Enforce compliance with data retention policies
- Prevent accidental over-retention of data
- Manage storage costs by limiting long-term data storage
Effects of the change:
- Data older than the specified period will be deleted, even if individual policies specify longer retention
- Helps maintain consistent data retention across all data streams
Common Issues or Misuses
- Setting too short a retention period, leading to premature data loss
- Forgetting to account for this setting when designing index lifecycle policies
- Not considering the impact on existing data streams when changing this setting
Do's and Don'ts
Do's:
- Align this setting with your organization's data retention policies
- Communicate changes to all stakeholders before implementation
- Regularly review and adjust based on changing requirements
Don'ts:
- Don't set this value without considering the needs of all data streams in your cluster
- Don't ignore this setting when designing individual index lifecycle policies
- Don't change this setting without understanding its impact on existing data
Frequently Asked Questions
Q: Can I set different maximum retention periods for different data streams?
A: No, this setting applies cluster-wide. For more granular control, use individual index lifecycle policies, but remember they cannot exceed this global maximum.
Q: What happens to existing data that exceeds the new maximum retention period when I change this setting?
A: Existing data that exceeds the new maximum retention period will be deleted during the next index lifecycle management execution.
Q: Does this setting affect snapshots or backups of my data streams?
A: No, this setting only affects the live data in your cluster. Snapshots and backups are not automatically deleted based on this setting.
Q: Can I temporarily disable this setting without losing its configured value?
A: Yes, you can set it to null to disable it temporarily. When you want to re-enable it, you can set it back to its previous value.
Q: How does this setting interact with the min_age parameter in ILM policies?
A: The min_age in ILM policies determines when an action should start, while data_streams.lifecycle.retention.max sets an absolute maximum age for data. The max retention will override any longer retention periods set by ILM policies.
