The network.breaker.inflight_requests.limit setting in Elasticsearch is part of the circuit breaker mechanism. It controls the maximum amount of memory that can be used by in-flight requests (requests that are currently being processed) before the circuit breaker is triggered.
Description
- Default Value: 100% of JVM heap
- Possible Values: Any percentage (e.g., 80%) or absolute value (e.g., 500mb)
- Recommendations: The default value is generally sufficient, but you may need to adjust it based on your specific use case and hardware resources.
This setting is crucial for preventing out-of-memory errors caused by too many concurrent requests. When the memory used by in-flight requests exceeds this limit, Elasticsearch will start rejecting new requests to protect the cluster from potential crashes.
Example
To change the network.breaker.inflight_requests.limit setting using the cluster settings API:
PUT /_cluster/settings
{
"persistent": {
"network.breaker.inflight_requests.limit": "80%"
}
}
You might want to lower this limit if you're experiencing out-of-memory errors due to a high number of concurrent requests. Reducing it to 80% of the JVM heap, for example, provides more headroom for other operations.
Common Issues and Misuses
- Setting the limit too low can lead to unnecessary request rejections during normal operation.
- Setting it too high might not provide adequate protection against memory exhaustion.
- Misinterpreting breaker trips as performance issues rather than as a protective measure.
Do's and Don'ts
- Do monitor the breaker trips and adjust the limit based on your application's needs.
- Do consider this setting in conjunction with other circuit breaker settings for a comprehensive memory management strategy.
- Don't set this value extremely low, as it may cause unnecessary request rejections.
- Don't ignore breaker trips; they are indicators of potential memory pressure.
Frequently Asked Questions
Q: How does the inflight requests breaker differ from other circuit breakers in Elasticsearch?
A: The inflight requests breaker specifically focuses on memory used by requests that are currently being processed, while other breakers (like the parent circuit breaker) deal with overall memory usage across various Elasticsearch operations.
Q: Can changing this setting impact query performance?
A: While it doesn't directly affect query performance, setting it too low might lead to request rejections, which can be perceived as performance degradation from the client's perspective.
Q: How can I monitor if this breaker is being triggered frequently?
A: You can use Elasticsearch's stats API (GET /_nodes/stats/breaker) to monitor breaker stats, including trip counts and estimated and limit sizes.
Q: Should I adjust this setting if I'm increasing my cluster's heap size?
A: If you're significantly increasing your cluster's heap size, you might want to review and potentially adjust this setting to ensure it remains appropriate for your new memory configuration.
Q: How does this setting interact with the queue size settings for thread pools?
A: While thread pool queue sizes control the number of requests that can be queued, the inflight requests breaker limits the total memory these requests can consume, providing an additional layer of protection against memory exhaustion.
