Guide for Using the Elastic Agent

Elastic Agent is a unified agent for collecting logs, metrics, and security data from your infrastructure and applications. It simplifies data collection by replacing multiple Beats with a single agent that can be centrally managed through Fleet. This guide covers installation, configuration, integrations, and best practices.

What is Elastic Agent?

Elastic Agent provides:

Single agent replacing Filebeat, Metricbeat, Packetbeat, and others
Unified management through Fleet in Kibana
Pre-built integrations for common services and applications
Automatic updates and configuration management
Simplified deployment with centralized policies

Key Components

1. Elastic Agent

The lightweight agent installed on your hosts that collects and ships data.

2. Fleet

The management interface in Kibana for:

Deploying agents
Managing agent policies
Monitoring agent status
Updating configurations

3. Integrations

Pre-packaged configurations for:

System monitoring
Application logs
Cloud services
Security tools
Custom data sources

Installation Methods

Fleet-Managed Installation (Recommended)

Prerequisites

Elasticsearch and Kibana running
Fleet Server set up in Kibana
Network connectivity between agents and Fleet Server

Step 1: Set Up Fleet

Navigate to Kibana > Management > Fleet
Click Add Fleet Server
Choose deployment type:
- Quick start: For testing
- Advanced: For production

Quick Start Setup:

# Install and enroll Fleet Server
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.x.x-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.x.x-linux-x86_64.tar.gz
cd elastic-agent-8.x.x-linux-x86_64

sudo ./elastic-agent install \
  --fleet-server-es=https://elasticsearch:9200 \
  --fleet-server-service-token=<service-token> \
  --fleet-server-policy=fleet-server-policy

Step 2: Create Agent Policy

In Fleet, click Agent policies > Create agent policy
Name your policy (e.g., "Linux Servers", "Windows Endpoints")
Select system monitoring integration
Save policy

Step 3: Install Elastic Agent

Click Add agent in Fleet
Select the agent policy
Copy the installation command

Linux/macOS:

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.x.x-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.x.x-linux-x86_64.tar.gz
cd elastic-agent-8.x.x-linux-x86_64

sudo ./elastic-agent install \
  --url=https://fleet-server:8220 \
  --enrollment-token=<enrollment-token>

Windows:

$ProgressPreference = 'SilentlyContinue'
Invoke-WebRequest -Uri https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.x.x-windows-x86_64.zip -OutFile elastic-agent-8.x.x-windows-x86_64.zip
Expand-Archive .\elastic-agent-8.x.x-windows-x86_64.zip
cd elastic-agent-8.x.x-windows-x86_64

.\elastic-agent.exe install `
  --url=https://fleet-server:8220 `
  --enrollment-token=<enrollment-token>

Standalone Installation

For environments without Fleet Server:

Download and extract:

curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.x.x-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.x.x-linux-x86_64.tar.gz
cd elastic-agent-8.x.x-linux-x86_64

Create elastic-agent.yml:

outputs:
  default:
    type: elasticsearch
    hosts: ["https://localhost:9200"]
    username: "elastic"
    password: "changeme"

inputs:
  - type: system/metrics
    data_stream.namespace: default
    streams:
      - metricset: cpu
        data_stream.dataset: system.cpu
      - metricset: memory
        data_stream.dataset: system.memory
      - metricset: network
        data_stream.dataset: system.network

Run standalone:

sudo ./elastic-agent run -c elastic-agent.yml

Docker Installation

Fleet-managed:

docker run \
  --env FLEET_ENROLL=1 \
  --env FLEET_URL=https://fleet-server:8220 \
  --env FLEET_ENROLLMENT_TOKEN=<token> \
  --rm docker.elastic.co/beats/elastic-agent:8.x.x

Standalone:

docker run \
  -v $(pwd)/elastic-agent.yml:/usr/share/elastic-agent/elastic-agent.yml \
  --rm docker.elastic.co/beats/elastic-agent:8.x.x

Managing Integrations

Adding Integrations to Agent Policy

Go to Fleet > Agent policies
Select your policy
Click Add integration
Browse or search for integration
Configure integration settings
Save integration

Popular Integrations

System Integration

Collects system metrics and logs:

CPU, memory, disk usage
Network statistics
System logs
Process information

Configuration:

- type: system/metrics
  streams:
    - metricset: cpu
    - metricset: memory
    - metricset: filesystem
    - metricset: network

Nginx Integration

Monitors Nginx web servers:

Access and error logs
Stub status metrics
Performance monitoring

Docker Integration

Monitors Docker containers:

Container metrics
Container logs
Docker events

AWS Integration

Collects AWS service data:

CloudWatch metrics
VPC Flow Logs
S3 access logs
CloudTrail events

Custom Logs Integration

Collects custom application logs:

File paths configuration
Log parsing
Multiline support
Processors

Example configuration:

- type: logfile
  paths:
    - /var/log/myapp/*.log
  processors:
    - add_fields:
        target: ''
        fields:
          service.name: myapp
          service.environment: production

Agent Management with Fleet

Viewing Agent Status

Navigate to Fleet > Agents
View agent list with:
- Health status (Healthy, Offline, Unhealthy)
- Last check-in time
- Applied policy
- Agent version

Updating Agent Policies

Policy changes are automatically applied to enrolled agents:

Go to Fleet > Agent policies
Select policy to modify
Edit integration settings
Save changes
Agents receive updates automatically

Upgrading Agents

Bulk upgrade:

Go to Fleet > Agents
Select agents to upgrade
Click Actions > Upgrade agent
Confirm upgrade

Individual upgrade:

Click on specific agent
Click Actions > Upgrade agent

Monitoring Agent Health

Kibana UI:

Fleet dashboard shows agent status
Click individual agents for details
View error logs and diagnostics

Agent logs location:

Linux: /var/lib/elastic-agent/data/elastic-agent-*/logs/
Windows: C:\Program Files\Elastic\Agent\data\elastic-agent-*\logs\
macOS: /Library/Elastic/Agent/data/elastic-agent-*/logs/

Agent Policy Best Practices

1. Policy Organization

Create separate policies for:

Different environments (dev, staging, prod)
Different server types (web, database, application)
Different operating systems
Different security requirements

Example structure:

production-linux-servers
production-windows-servers
staging-all-servers
security-endpoints

2. Namespace Convention

Use meaningful namespaces:

data_stream.namespace: production-web

Benefits:

Easier data segmentation
Simplified index management
Better access control

3. Resource Management

Set appropriate limits:

agent.limits:
  go_max_procs: 0
agent.monitoring:
  enabled: true
  logs: true
  metrics: true

4. Data Collection Optimization

Filter unnecessary data:

processors:
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.id"]
  - drop_event:
      when:
        equals:
          http.response.status_code: 200

Reduce collection frequency:

- type: system/metrics
  period: 60s  # Collect every 60 seconds instead of default 10s

Troubleshooting

Agent Not Appearing in Fleet

Check:

Network connectivity to Fleet Server
Enrollment token validity
Fleet Server status
Firewall rules (port 8220)

Verify Fleet Server:

curl -k https://fleet-server:8220/api/status

Agent Status Unhealthy

Common causes:

Configuration errors
Permission issues
Network connectivity
Resource constraints

View agent logs:

# Linux
sudo cat /var/lib/elastic-agent/data/elastic-agent-*/logs/elastic-agent-json.log

# View diagnostics
sudo elastic-agent diagnostics

Integration Not Collecting Data

Verify:

Integration properly configured
Required permissions granted
Data source accessible
No conflicting configurations

Test connection:

sudo elastic-agent test config

High Resource Usage

Solutions:

Reduce collection frequency
Filter unnecessary fields
Limit number of integrations
Adjust resource limits
Use processors to drop events

Security Best Practices

1. Secure Enrollment

Use short-lived enrollment tokens
Rotate tokens regularly
Limit token permissions

2. Network Security

Use TLS for all communications
Configure certificate validation
Implement network segmentation
Use firewall rules

3. Access Control

Use role-based access control
Limit Fleet user permissions
Separate policies by sensitivity
Audit agent access

4. Agent Updates

Enable automatic security updates
Monitor CVE announcements
Test updates in staging
Maintain version consistency

Advanced Configuration

Custom Output Configuration

Send data to multiple destinations:

outputs:
  default:
    type: elasticsearch
    hosts: ["https://es-cluster1:9200"]

  monitoring:
    type: elasticsearch
    hosts: ["https://monitoring-cluster:9200"]

Processor Configuration

Transform data before sending:

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - dissect:
      tokenizer: "%{timestamp} %{log_level} %{message}"
      field: "message"
  - drop_fields:
      fields: ["temp_field"]

Proxy Configuration

Route through proxy:

outputs:
  default:
    type: elasticsearch
    hosts: ["https://elasticsearch:9200"]
    proxy_url: "http://proxy-server:8080"
    proxy_headers:
      X-Custom-Header: value

Frequently Asked Questions

Q: What's the difference between Elastic Agent and Beats?
A: Elastic Agent is a unified agent that can replace multiple Beats (Filebeat, Metricbeat, etc.) and is managed centrally through Fleet.

Q: Can I run both Elastic Agent and Beats on the same host?
A: Yes, but it's not recommended as it may cause duplicate data collection and increased resource usage.

Q: How do I migrate from Beats to Elastic Agent?
A: Install Elastic Agent with equivalent integrations, verify data collection, then uninstall Beats. Use the same data streams for continuity.

Q: Does Elastic Agent require internet access?
A: Only to the Fleet Server and Elasticsearch endpoints. It doesn't require public internet access.

Q: Can Elastic Agent collect Windows Event Logs?
A: Yes, through the Windows integration which collects event logs, performance counters, and more.

Q: How much disk space does Elastic Agent need?
A: Typically 500MB-1GB for the agent and temporary data buffers, depending on configuration.

Q: Can I use Elastic Agent in air-gapped environments?
A: Yes, using standalone mode without Fleet, though you lose centralized management capabilities.

Q: How do I uninstall Elastic Agent?
A: Use sudo elastic-agent uninstall (Linux/macOS) or run the uninstaller from Control Panel (Windows).

Q: Can Elastic Agent collect data from cloud services?
A: Yes, through cloud integrations for AWS, Azure, GCP, and other cloud providers.

Q: What happens if the agent loses connection to Fleet Server?
A: The agent continues operating with its last known configuration and buffers data until connection is restored.