Elasticsearch Script Query: Filtering with Painless

Q: How do I avoid script recompilation errors in the script query?

Move per-request values into params so the source string stays constant. Elasticsearch caches up to script.context.filter.cache_max_size distinct script sources; changing source on every request thrashes the cache and eventually triggers general_script_exception .

The Elasticsearch script query filters documents using a Painless script that returns a boolean. Each matching candidate document executes the script; documents for which the script returns true are included. The script query produces no relevance score and is intended for filter context. Reach for it when no other DSL clause expresses the condition - for example, a comparison between two fields on the same document.

Syntax

{
  "query": {
    "script": {
      "script": {
        "source": "doc['a'].value > doc['b'].value",
        "lang": "painless",
        "params": { }
      }
    }
  }
}

Parameters

Parameter	Type	Default	Description
`script.source`	string	-	Painless source. Must return `boolean`.
`script.id`	string	-	Reference to a stored script (alternative to `source`).
`script.lang`	string	`painless`	Script language. Painless is the only supported on-the-fly language.
`script.params`	object	`{}`	Variables referenced inside the script as `params.x`. Keeps the source stable so it can be cached.

Inside the script, document fields are accessed through doc['field'] (doc values, fastest), params._source.field (raw source, slow), or via runtime fields.

Examples

Compare two numeric fields on the same document:

GET /sales/_search
{
  "query": {
    "bool": {
      "filter": {
        "script": {
          "script": "doc['price'].value > doc['cost'].value"
        }
      }
    }
  }
}

Use parameters to keep the script cacheable:

GET /products/_search
{
  "query": {
    "bool": {
      "filter": {
        "script": {
          "script": {
            "source": "doc['price'].value < params.max_price && doc['in_stock'].value == true",
            "params": { "max_price": 100 }
          }
        }
      }
    }
  }
}

Reference a stored script:

POST _scripts/min_margin
{
  "script": {
    "lang": "painless",
    "source": "doc['price'].value - doc['cost'].value >= params.margin"
  }
}

GET /orders/_search
{
  "query": {
    "script": { "script": { "id": "min_margin", "params": { "margin": 5 } } }
  }
}

Performance and Use Notes

The script query runs on every candidate document the query phase produces. It is by definition uncached at the segment level the way a term filter is, although the script source itself is compiled once and cached (parameter values are not part of the cache key, so params keeps recompilation off the hot path). Always place the script clause inside bool.filter and pair it with selective filters above it so the script only sees the documents that survive cheaper clauses.

Prefer doc values (doc['field'].value) over params._source - doc values are columnar and live in OS cache. If the field type doesn't support doc values, consider indexing a keyword/numeric mirror, or use a runtime field that exposes a script-derived value as a regular field.

Heavy use of script queries is a frequent cause of CPU saturation in Elasticsearch clusters. The manual workflow - read the slow log, decode each Painless body, decide whether the same predicate could be expressed as a cached filter or a runtime field - is exactly what Pulse automates.

Common Mistakes

Using the script query in query context (no filter wrapper); it forces score computation and disables filter-bitset caching.
Building the source string dynamically with embedded values - this defeats script compilation caching and triggers script_compilation_max errors.
Reading from params._source for a field that has doc values.
Forgetting that doc['field'].value throws when the field is missing; guard with doc['field'].size() != 0.
Trying to access external HTTP resources or filesystem from a Painless script - blocked by the security manager.

Find Slow Script Queries with Pulse

Pulse is an AI DBA for Elasticsearch and OpenSearch that continuously profiles production query traffic. For Painless script queries specifically, Pulse:

Identifies script queries running in query context instead of bool.filter, where every candidate document forces score computation and bypasses filter-bitset caching
Flags script bodies built with embedded per-request values instead of params, which thrashes the script compilation cache and risks script_compilation_max errors
Spots scripts reading from params._source on fields that have doc values, plus scripts running over candidate sets that should have been pruned by a selective filter clause
Traces each slow script query back to the calling service via slow-log and APM correlation
Recommends concrete rewrites: convert the predicate into a runtime field that can be filtered with normal DSL, materialize the derived value into an indexed field, switch to doc values access, or push a selective term/range filter ahead of the script
Tracks latency and script-compilation-rate improvement after the change ships

This converts the manual slow-log plus Painless review loop into a continuous optimization workflow.

Try Pulse on your cluster.

Frequently Asked Questions

Q: When should I use script query vs runtime fields in Elasticsearch?
A: Runtime fields expose a script-derived value as a queryable field, which can then be filtered with normal DSL (term, range). Use runtime fields when the same derived value is queried repeatedly; use the script query for one-off boolean conditions that are not reused.

Q: Does the script query support languages other than Painless?
A: Painless is the only inline scripting language supported in production Elasticsearch. Expression, Mustache, and Lucene Expressions exist for specific contexts (template rendering, sorting), but the script query body should be Painless.

Q: How do I avoid script recompilation errors in the script query?
A: Move per-request values into params so the source string stays constant. Elasticsearch caches up to script.context.filter.cache_max_size distinct script sources; changing source on every request thrashes the cache and eventually triggers general_script_exception.

Q: Can the Elasticsearch script query return a numeric score?
A: No. The script query expects a boolean. To produce a custom numeric score, use the script_score query or the function_score query with a script_score function.

Q: Why is my script query slow?
A: Either it runs against too large a candidate set, or it reads from _source instead of doc values. Add selective filters before the script clause, and switch to doc['field'].value where the field has doc values enabled.

Q: Is the script query cached like other filters?
A: The compiled script is cached, but the per-document evaluation result is not stored in the request cache or the segment filter cache. That is why narrowing the candidate set with cacheable filters first matters.

Q: How do I debug expensive Painless script queries in production Elasticsearch?
A: Pulse profiles Elasticsearch and OpenSearch slow logs, identifies script queries running outside filter context, those with non-parameterized sources, and those reading from _source instead of doc values, attributes each one to the calling service, and recommends a runtime-field, indexed-field, or filter-first rewrite.

Elasticsearch Script Score Query: scoring-side script query.
Elasticsearch Function Score Query: scoring with script_score and other functions.
Elasticsearch Bool Query: the correct wrapper for placing scripts in filter context.
Elasticsearch Term Query: exact-value filters that should usually precede a script clause.
Elasticsearch Range Query: numeric range filtering without scripts.

Elasticsearch Script Query: Filtering with Painless - Syntax, Example, and Tips