Logstash kv Filter Plugin

The kv (key-value) filter plugin in Logstash is used to automatically parse key-value pairs from event fields. It's particularly useful for processing logs that contain data in key=value format, allowing you to extract structured data from unstructured or semi-structured log entries.

Syntax

The basic syntax for the kv filter is:

kv {
  source => "message"
  field_split => "&"
  value_split => "="
}

For detailed configuration options, refer to the official Logstash kv filter documentation.

Example Use Case

Consider a log line containing key-value pairs:

user=john id=123 action=login status=success

You can use the kv filter to parse this into structured fields:

filter {
  kv {
    source => "message"
    field_split => " "
    include_keys => [ "user", "id", "action", "status" ]
  }
}

This configuration will create new fields in your event: user, id, action, and status, with their respective values.

Common Issues and Best Practices

1. Performance: The kv filter can be CPU-intensive for large logs. Use the include_keys or exclude_keys options to limit processing to necessary fields.

2. Field Conflicts: Be cautious of overwriting existing fields. Use the prefix option to avoid conflicts.

3. Complex Values: If values contain spaces or special characters, you may need to adjust the field_split and value_split options or use regex.

4. Nested Data: For nested key-value pairs, consider using the recursive option.

5. Type Casting: By default, all extracted values are strings. Use the transform_key and transform_value options for type casting.

Frequently Asked Questions

Q: How can I handle key-value pairs with spaces in the values?
A: You can use regex for more complex parsing. For example, field_split => "(?<=[a-zA-Z0-9])\s+(?=[a-zA-Z0-9]+=)" can handle spaces in values.

Q: Can the kv filter parse JSON-like structures?
A: While kv is primarily for simple key-value pairs, you can use it in conjunction with the json filter for more complex structures.

Q: How do I deal with duplicate keys in my log data?
A: Use the allow_duplicate_values option set to false to keep only the last value for a given key.

Q: Can I use kv filter to parse URL query parameters?
A: Yes, it's great for this. Set field_split => "&" and value_split => "=" to parse query strings.

Q: How can I validate the keys extracted by the kv filter?
A: Use the include_keys option to specify allowed keys, or exclude_keys to remove unwanted ones. You can also use the validate_keys option for more advanced validation.