Logstash Documentation

Welcome to the comprehensive Logstash documentation hub. This guide covers all aspects of Logstash operations, from filter plugins and configuration to error troubleshooting and performance optimization.

Filter Plugins

Data Processing and Transformation

Mutate Filter Plugin - Perform general mutations on fields
Clone Filter Plugin - Create copies of events
Alter Filter Plugin - Modify field values
Split Filter Plugin - Split events into multiple events
Prune Filter Plugin - Remove unwanted fields
Drop Filter Plugin - Drop events based on conditions
Truncate Filter Plugin - Truncate field values
Dedot Filter Plugin - Remove dots from field names
Age Filter Plugin - Calculate age from timestamps
Bytes Filter Plugin - Convert byte values
Range Filter Plugin - Check if values fall within ranges
ExtractNumbers Filter Plugin - Extract numeric values

Parsing and Text Processing

Grok Filter Plugin - Parse unstructured data using patterns
JSON Filter Plugin - Parse JSON data
CSV Filter Plugin - Parse CSV data
XML Filter Plugin - Parse XML data
KV Filter Plugin - Parse key-value pairs
Dissect Filter Plugin - Extract structured fields using delimiters
Date Filter Plugin - Parse dates from fields
Syslog Pri Filter Plugin - Parse syslog priority values
URLDecode Filter Plugin - Decode URL-encoded strings
UserAgent Filter Plugin - Parse user agent strings
I18n Filter Plugin - Handle internationalization
TLD Filter Plugin - Extract top-level domain information

Enrichment and External Data

GeoIP Filter Plugin - Add geographic information
DNS Filter Plugin - Perform DNS lookups
Translate Filter Plugin - Replace field values using dictionaries
JDBC Static Filter Plugin - Enrich with database data
JDBC Streaming Filter Plugin - Stream data from databases
Elasticsearch Filter Plugin - Query Elasticsearch for enrichment
HTTP Filter Plugin - Make HTTP requests for data enrichment
Memcached Filter Plugin - Query Memcached for data

Specialized and Utility Filters

Cipher Filter Plugin - Encrypt or decrypt fields
Fingerprint Filter Plugin - Generate fingerprints for events
UUID Filter Plugin - Generate UUID values
Java UUID Filter Plugin - Generate Java-style UUIDs
Metrics Filter Plugin - Generate metrics from events
Metricize Filter Plugin - Convert events to metrics
Sleep Filter Plugin - Add delays to processing
Environment Filter Plugin - Access environment variables
CIDR Filter Plugin - Work with CIDR network ranges
Aggregate Filter Plugin - Aggregate data across events
Elapsed Filter Plugin - Measure time between events
WURFL Device Detection Filter Plugin - Detect device information
Threats Classifier Filter Plugin - Classify security threats
Elastic Integration Filter Plugin - Integrate with Elastic products
Ruby Filter Plugin - Execute custom Ruby code
JSON Encode Filter Plugin - Encode data as JSON

Common Errors and Troubleshooting

Configuration and Pipeline Issues

Pipeline Aborted Configuration Error - Configuration syntax issues
Expected One of Hash Input Filter Output - Configuration structure errors
Configuration Error Expected Opening Brace Found - Syntax errors
Field Subfield Not Defined - Field reference errors
Pipeline Aborted Due to Error - General pipeline failures
Pipeline Worker Thread Died - Worker thread issues
Pipeline is Blocked - Pipeline blocking issues
Timeout Executing Filter - Filter execution timeouts

Memory and Performance Issues

Persistent Queue is Full - Queue capacity issues
Max Inflight Events Reached - Event processing limits
Detected Corrupt Queue File - Queue corruption
Event Rate Exceeded Threshold - Performance thresholds
Metrics Collector Threw Exception - Metrics collection issues
Could Not Write Event to DLQ - Dead letter queue problems

Data Processing and Parsing Errors

Grok Parse Failure - Grok pattern matching failures
Failed Parsing Date from Field - Date parsing errors
Mutate Filter Failed - Mutate filter execution issues
Cannot Cast Value to Type - Type conversion errors
Cannot Decode Event - Event decoding issues

Network and Connectivity Issues

Address Already in Use - Port binding conflicts
Read Timed Out - Network read timeouts
Connection Reset by Peer - Connection issues
Could Not Resolve Host - DNS resolution failures
SSL Handshake Error - SSL/TLS connection issues
DNS Lookup Timeout - DNS resolution timeouts
Fetching Metadata - Metadata retrieval issues

External Service Integration

Elasticsearch Unreachable - Elasticsearch connectivity
Elasticsearch Rejected Documents - Document rejection issues
MapperParsingException - Elasticsearch parsing errors
Invalid Logstash Operation - Invalid operations
Mapping Conflict for Field - Elasticsearch mapping conflicts
401 Unauthorized - Authentication failures
Timed Out Connecting Kafka - Kafka connection timeouts
Kafka Leader Not Available - Kafka leadership issues
Kafka Consumer Lag Detected - Consumer lag problems
Cannot Connect to Redis Server - Redis connectivity
JDBC Error - Database connectivity issues

Installation and System Issues

Plugin Not Found - Missing plugin errors
Bundler Install Error - Installation failures
Failed to Start Logstash Service - Service startup issues
No Executable Java Binary - Java runtime issues
No Such File or Directory - File access errors
Permission Denied - File permission issues
File Already Processed - File input tracking
Log File Rotation Failed - Log rotation issues
License Expired - License expiration problems

Input-Specific Errors

SNMP Input Error - SNMP input issues
Beats Input Unknown Client - Beats input problems
GeoIP Filter Could Not Locate Database - GeoIP database issues

This documentation serves as your comprehensive guide to Logstash operations. Each linked article provides detailed information about specific plugins, errors, and troubleshooting steps.