Elasticsearch's default pagination mechanism - from and size - works fine for the first few pages of results. Once you push past shallow offsets, performance degrades in ways that are predictable but frequently misunderstood. The root cause is architectural: every shard has to build a sorted result set of from + size documents, send all of them to the coordinating node, which then merges and discards everything except the final page. Requesting page 500 at 20 results per page means each shard sorts and transmits 10,000 documents so the coordinator can throw away 9,980 of them.
Why from+size Breaks at High Offsets
The index.max_result_window setting defaults to 10,000 and caps the maximum value of from + size. This is not an arbitrary limit - it reflects the linear heap cost of deep pagination. Each shard allocates a priority queue proportional to from + size, fills it by scoring and sorting documents, then ships the entire queue over the network. With 5 shards and from=9980, size=20, the coordinating node receives 50,000 scored document references, merges them, and returns 20.
Raising max_result_window to 100,000 or higher is technically possible but does not fix the underlying problem. Memory consumption on both shards and the coordinating node scales linearly with the offset. At high offsets, you will see increased GC pressure, higher query latency, and eventually rejected requests when the request circuit breaker trips. The setting exists as a guardrail; working around it by bumping the number is treating the symptom.
PUT /my_index/_settings
{
"index.max_result_window": 50000
}
This buys time but shifts the failure point. The correct response to hitting this limit is almost always to switch pagination strategies, not to increase it.
The Scroll API: Deprecated for Search
The scroll API was historically the answer for traversing large result sets. It works by creating a persistent search context on each shard, keeping a snapshot of the index state at the time the scroll was opened. Subsequent scroll requests return the next batch from that frozen snapshot without re-executing the query.
Elastic has deprecated scroll for search use cases since version 7.x. The reasons are practical: scroll contexts pin old segments in memory, preventing merges from reclaiming disk space. A long-running scroll on a high-ingestion index can cause segment count to balloon because Lucene cannot merge segments still referenced by an open context. Each open scroll also consumes file descriptors and heap on every shard involved.
Scroll remains a valid tool for reindexing and bulk export - use cases where you need to process every document exactly once and do not care about real-time consistency. For anything user-facing or interactive, it is the wrong tool.
search_after with Point in Time
The recommended approach for paginating beyond shallow offsets is search_after combined with a Point in Time (PIT). A PIT creates a lightweight, consistent snapshot of the index state. Unlike scroll, it does not pin search contexts to individual shards in a way that blocks segment merges. You open a PIT, then issue successive search requests with search_after set to the sort values of the last document from the previous page.
POST /my_index/_pit?keep_alive=2m
// Response: { "id": "46ToAwMDaWR..." }
POST /_search
{
"size": 20,
"query": { "match": { "status": "active" } },
"pit": { "id": "46ToAwMDaWR...", "keep_alive": "2m" },
"sort": [
{ "created_at": "desc" },
{ "_shard_doc": "asc" }
]
}
To fetch the next page, take the sort array from the last hit and pass it as search_after:
POST /_search
{
"size": 20,
"query": { "match": { "status": "active" } },
"pit": { "id": "46ToAwMDaWR...", "keep_alive": "2m" },
"sort": [
{ "created_at": "desc" },
{ "_shard_doc": "asc" }
],
"search_after": [1672531200000, 42]
}
The _shard_doc tiebreaker is a built-in sort field provided automatically when using PIT. It guarantees unique sort values across shards without requiring a unique field in your mapping. Each request only needs to process documents after the tiebreaker point rather than building the full offset priority queue, so the cost stays constant regardless of how deep you paginate.
Close the PIT when you are done to free resources:
DELETE /_pit
{ "id": "46ToAwMDaWR..." }
Composite Aggregation for Aggregated Pagination
When you need to paginate over aggregation buckets rather than individual documents, search_after does not apply. The composite aggregation fills this gap. It returns buckets in a deterministic order and provides an after_key in each response that you pass back to fetch the next batch.
POST /logs/_search?size=0
{
"aggs": {
"by_host_and_status": {
"composite": {
"size": 100,
"sources": [
{ "host": { "terms": { "field": "host.keyword" } } },
{ "status": { "terms": { "field": "status_code" } } }
]
}
}
}
}
The response includes an after_key object. Pass it back verbatim in the next request:
{
"aggs": {
"by_host_and_status": {
"composite": {
"size": 100,
"sources": [ ... ],
"after": { "host": "web-07", "status": 502 }
}
}
}
}
When no after_key appears in the response, you have exhausted all buckets. This pattern works well for batch export of aggregated data, building paginated dashboards over grouped metrics, and any case where iterating over all unique value combinations is the goal.
Migrating from Scroll to search_after
If you have existing code using scroll for search pagination, the migration path is straightforward. Replace the initial scroll request with a PIT open call and a search with sort defined. Replace each _search/scroll call with a new _search request carrying search_after and the PIT ID. Replace the scroll clear call with a PIT delete.
The behavioral difference to watch for: scroll returns results from a frozen snapshot created at open time. PIT with search_after also gives you a consistent view, but PIT snapshots are lighter - they do not prevent segment merges the way scroll contexts do. If your scroll-based code depends on processing every document exactly once during a long-running export while the index is actively receiving writes, PIT plus search_after preserves that guarantee while placing less strain on the cluster.
One constraint to keep in mind: search_after does not support random access to arbitrary pages. You cannot jump directly to page 50. You have to paginate sequentially from the beginning or from a previously saved cursor. For user-facing UIs that need arbitrary page jumps, the practical approach is to show only next/previous navigation, or to combine search_after with cached cursor positions for commonly accessed offsets.
