The kv (key-value) filter plugin in Logstash is used to automatically parse key-value pairs from event fields. It's particularly useful for processing logs that contain data in key=value format, allowing you to extract structured data from unstructured or semi-structured log entries.
Syntax
The basic syntax for the kv filter is:
kv {
source => "message"
field_split => "&"
value_split => "="
}
For detailed configuration options, refer to the official Logstash kv filter documentation.
Example Use Case
Consider a log line containing key-value pairs:
user=john id=123 action=login status=success
You can use the kv filter to parse this into structured fields:
filter {
kv {
source => "message"
field_split => " "
include_keys => [ "user", "id", "action", "status" ]
}
}
This configuration will create new fields in your event: user
, id
, action
, and status
, with their respective values.
Common Issues and Best Practices
Performance: The kv filter can be CPU-intensive for large logs. Use the
include_keys
orexclude_keys
options to limit processing to necessary fields.Field Conflicts: Be cautious of overwriting existing fields. Use the
prefix
option to avoid conflicts.Complex Values: If values contain spaces or special characters, you may need to adjust the
field_split
andvalue_split
options or use regex.Nested Data: For nested key-value pairs, consider using the
recursive
option.Type Casting: By default, all extracted values are strings. Use the
transform_key
andtransform_value
options for type casting.
Frequently Asked Questions
Q: How can I handle key-value pairs with spaces in the values?
A: You can use regex for more complex parsing. For example, field_split => "(?<=[a-zA-Z0-9])\s+(?=[a-zA-Z0-9]+=)"
can handle spaces in values.
Q: Can the kv filter parse JSON-like structures?
A: While kv is primarily for simple key-value pairs, you can use it in conjunction with the json filter for more complex structures.
Q: How do I deal with duplicate keys in my log data?
A: Use the allow_duplicate_values
option set to false to keep only the last value for a given key.
Q: Can I use kv filter to parse URL query parameters?
A: Yes, it's great for this. Set field_split => "&"
and value_split => "="
to parse query strings.
Q: How can I validate the keys extracted by the kv filter?
A: Use the include_keys
option to specify allowed keys, or exclude_keys
to remove unwanted ones. You can also use the validate_keys
option for more advanced validation.