Logstash kv Filter Plugin

The Logstash kv filter parses key=value pairs from a single source field and writes them as event fields. It handles firewall logs, URL query strings, syslog key/value formats, and ad-hoc structured text. Configure field_split (separator between pairs) and value_split (separator between key and value) to match the input. Without an include_keys allowlist, every key in the source becomes a top-level event field, which can blow up the Elasticsearch mapping.

Syntax

filter {
  kv {
    source         => "message"
    target         => "kv"
    field_split    => " "
    value_split    => "="
    include_brackets => true
    trim_key       => " "
    trim_value     => "\""
    include_keys   => [ "user", "action", "status" ]
    allow_duplicate_values => false
  }
}

Parameters

Examples

Parse a URL query string into a qs sub-field:

filter {
  kv {
    source      => "query_string"
    target      => "qs"
    field_split => "&"
    value_split => "="
  }
}

Parse a Cisco ASA firewall log with key=value pairs separated by spaces, restricted to known keys:

filter {
  kv {
    source       => "message"
    target       => "asa"
    include_keys => [ "src", "dst", "sport", "dport", "proto", "action" ]
  }
}

Parse syslog-style application logs that mix quoted and unquoted values:

filter {
  kv {
    source     => "message"
    target     => "app"
    trim_value => "\""
    field_split_pattern => "[\\s,]"
  }
}

Common Issues

The most common production failure is mapping explosion: unbounded keys in the source produce hundreds of unique event fields, which become hundreds of unique Elasticsearch fields, eventually hitting the index.mapping.total_fields.limit (default 1000). Always set include_keys or use target plus a dynamic_templates mapping on the receiving index.

Values containing the value_split character (=) are split on the first occurrence only by default, so key=a=b=c becomes key => "a=b=c". Values with embedded spaces require quoted values plus trim_value => "\"", or a custom field_split_pattern that only splits on whitespace outside quotes.

Duplicate keys are coalesced into arrays by default; set allow_duplicate_values => false to keep only the last occurrence. This matters for replayed log lines where a duplicate key indicates a logging bug rather than a real array.

Performance Notes

The kv filter is implemented in Ruby and is slower than json or dissect for equivalent structured data. For high-throughput pipelines parsing logs that you control end-to-end, switch the upstream application to emit JSON and use the json filter instead. When kv is unavoidable, the biggest wins are: an include_keys allowlist (skips work entirely for unwanted keys), a field_split_pattern simple enough to compile to a fast regex, and a target sub-field so downstream filters operate on a known shape.

Monitoring Logstash kv Pipelines with Pulse

Pulse is the only tool built specifically for monitoring and optimizing Logstash pipelines. Unrestricted kv parsing is a classic cause of "Elasticsearch suddenly rejecting events" because the mapping limit was hit overnight. Pulse correlates Logstash output failures with the upstream filter and identifies kv-driven mapping explosions before they block ingestion. It also tracks per-filter throughput so you can spot kv filters that have become CPU hotspots after a log format change.

Frequently Asked Questions

Q: How does the Logstash kv filter differ from grok?
A: The kv filter is purpose-built for repeating key=value pairs and runs faster than grok for that shape because it does not use full regex matching per field. Grok is more flexible (it handles positional and irregular formats) but slower. Use kv first when the input is genuinely a list of pairs.

Q: Can the Logstash kv filter parse URL query strings?
A: Yes. Set field_split => "&" and value_split => "=" and the filter behaves like a URL-decoded query string parser. Note that the kv filter does not URL-decode percent-encoded values - run a urldecode filter afterwards if you need decoded text.

Q: How do I prevent the Logstash kv filter from exploding Elasticsearch mappings?
A: Set include_keys to an explicit allowlist, write parsed output to a target sub-field rather than the event root, and configure the destination index with dynamic_templates that map unknown sub-fields as keyword with ignore_above.

Q: Does the Logstash kv filter handle nested key=value pairs?
A: Set recursive => true to apply the same kv parsing to values that themselves contain pairs. This is useful for formats like outer=foo inner="a=1 b=2".

Q: How does the Logstash kv filter handle quoted values with spaces?
A: Set trim_value => "\"" to strip surrounding quotes, and use a field_split_pattern that does not match whitespace inside quoted regions. For complex quoting rules, a grok or dissect stage upstream is often easier than tuning kv.

Q: What is the Logstash kv filter's default field_split?
A: A single space character. Set field_split => "&" for URL query strings, field_split => "," for comma-separated pairs, or use field_split_pattern for anything more complex.

Related Reading

• Logstash Grok Filter Plugin: handles non-key/value patterns and positional fields.
• Logstash Dissect Filter Plugin: faster delimiter-based parsing for fixed-position fields.
• Logstash JSON Filter Plugin: preferred when the source is structured JSON.
• Logstash GeoIP Filter Plugin: enrich IP addresses parsed from kv output.
• Logstash Mutate Filter Plugin: cast string values from kv into numeric or boolean types.