Logstash tld Filter Plugin

The Logstash tld filter plugin is used to extract top-level domain (TLD) information from URLs or hostnames. It's particularly useful when processing log data containing domain names and you need to categorize or analyze based on the TLD.

Syntax

filter {
  tld {
    source => "field_containing_domain"
    target => "field_to_store_tld"
  }
}

For more detailed information, refer to the official Logstash tld filter plugin documentation.

Example Use Case

Suppose you have log entries with URLs, and you want to extract the TLD for analysis. Here's an example configuration:

filter {
  tld {
    source => "url"
    target => "tld"
  }
}

If the input event has a field url with value "https://www.example.com/page", the plugin will add a new field tld with value "com".

Common Issues and Best Practices

1. Ensure that the source field contains valid domain names or URLs.
2. The plugin may not recognize newer or less common TLDs. Keep the plugin updated to the latest version for better coverage.
3. Consider using this plugin in combination with the geoip filter for more comprehensive domain analysis.
4. Be aware that some domains use multi-level TLDs (e.g., .co.uk). The plugin will correctly identify these as well.

Frequently Asked Questions

Q: Can the tld filter handle internationalized domain names (IDNs)?
A: Yes, the tld filter can handle IDNs. It will extract the TLD correctly for both ASCII and Unicode representations of domain names.

Q: How does the tld filter handle subdomains?
A: The tld filter focuses on extracting the top-level domain. It doesn't directly handle subdomains, but you can use other Logstash filters like grok or dissect to extract subdomain information if needed.

Q: Is it possible to get both the TLD and the domain name?
A: The tld filter itself only extracts the TLD. To get both the TLD and the domain name, you might need to combine it with other filters or use a more complex configuration.

Q: How often is the TLD database updated in the plugin?
A: The TLD database in the plugin is typically updated with each new release of the plugin. It's recommended to keep your Logstash and its plugins up to date to ensure you have the latest TLD information.

Q: Can the tld filter handle URLs with IP addresses instead of domain names?
A: The tld filter is designed to work with domain names. If you pass an IP address, it won't extract a TLD. You might need to use conditional logic or other filters to handle IP addresses separately.