The breaker.model_inference.overhead setting in Elasticsearch controls the circuit breaker for machine learning model inference operations. It determines the amount of overhead to add to the estimated memory usage of model inference requests to prevent out-of-memory errors.
- Default value: 1.0 (100% overhead)
- Possible values: Any positive decimal number
- Recommendation: Start with the default and adjust based on your specific use case and observed memory usage
This setting adds a safety margin to the estimated memory usage of model inference operations. A value of 1.0 means the circuit breaker assumes twice the estimated memory will be used. Increasing this value provides more protection against out-of-memory errors but may unnecessarily limit concurrent operations.
Example
To change the breaker.model_inference.overhead setting using the cluster settings API:
PUT /_cluster/settings
{
"persistent": {
"breaker.model_inference.overhead": 1.5
}
}
This increases the overhead to 150% of the estimated memory usage. You might want to increase this value if you're experiencing out-of-memory errors during model inference operations, or decrease it if you find the default too conservative for your workload.
Common Issues
- Setting the value too low can lead to out-of-memory errors during model inference.
- Setting the value too high can unnecessarily limit the number of concurrent model inference operations.
Do's and Don'ts
- Do monitor your cluster's memory usage and adjust this setting accordingly.
- Do consider the complexity and size of your machine learning models when configuring this setting.
- Don't set this value below 1.0 unless you're absolutely certain about your memory requirements.
- Don't ignore out-of-memory errors; they may indicate that this setting needs adjustment.
Frequently Asked Questions
Q: How does the breaker.model_inference.overhead setting affect performance?
A: This setting doesn't directly affect performance, but it can impact the number of concurrent model inference operations allowed. A higher value provides more protection against out-of-memory errors but may reduce concurrency.
Q: Can I set breaker.model_inference.overhead to 0?
A: While technically possible, setting it to 0 is not recommended as it removes the safety margin entirely, increasing the risk of out-of-memory errors.
Q: How often should I adjust this setting?
A: Adjust it when you observe out-of-memory errors during model inference or if you find the current setting too conservative. Regular monitoring of your cluster's memory usage can help determine if adjustments are needed.
Q: Does this setting apply to all types of machine learning models in Elasticsearch?
A: Yes, this setting applies to all model inference operations performed by Elasticsearch, regardless of the specific type or complexity of the machine learning model.
Q: How does this setting interact with other circuit breaker settings in Elasticsearch?
A: While this setting is specific to model inference operations, it works in conjunction with other circuit breaker settings to manage overall memory usage in Elasticsearch. It's important to consider all circuit breaker settings holistically when tuning your cluster.
