Guide to Elastic APM

Elastic APM (Application Performance Monitoring) is a real-time monitoring solution that helps you track application performance, identify bottlenecks, and troubleshoot errors. Built on the Elastic Stack, APM collects detailed performance metrics, distributed traces, and error information to provide complete visibility into your applications.

What is Elastic APM?

Elastic APM consists of four main components:

APM Agents: Lightweight libraries that instrument your application code
APM Server: Receives data from agents and processes it
Elasticsearch: Stores and indexes APM data
Kibana APM UI: Visualizes and analyzes performance data

Key Features

1. Distributed Tracing

Track requests across microservices
Visualize service dependencies
Identify slow components in transaction flows
Understand inter-service communication

2. Performance Metrics

Response times and throughput
CPU and memory usage
Database query performance
External HTTP request latency
Custom metrics

3. Error Tracking

Automatic error capture
Stack traces and context
Error rates and trends
Exception grouping
Source code integration

4. Service Maps

Visual representation of service architecture
Real-time health indicators
Dependency visualization
Performance bottleneck identification

5. Alerting

Threshold-based alerts
Anomaly detection
Custom alert conditions
Integration with notification channels

Supported Languages and Frameworks

Elastic APM provides official agents for:

Java: Spring Boot, Servlet API, JAX-RS, and more
Node.js: Express, Koa, Fastify, Next.js
Python: Django, Flask, FastAPI, Tornado
Ruby: Rails, Sinatra, Grape
Go: net/http, Gin, Echo, Gorilla
.NET: ASP.NET Core, Entity Framework
PHP: Laravel, Symfony, WordPress
JavaScript (RUM): Browser-side monitoring

Getting Started with Elastic APM

Step 1: Set Up APM Server

Using Elastic Cloud

Create an Elastic Cloud deployment
Enable APM from deployment settings
Note the APM Server URL and secret token

Self-Hosted Installation

Download and Install:

curl -L -O https://artifacts.elastic.co/downloads/apm-server/apm-server-8.x.x-linux-x86_64.tar.gz
tar xzvf apm-server-8.x.x-linux-x86_64.tar.gz
cd apm-server-8.x.x-linux-x86_64/

Configure apm-server.yml:

apm-server:
  host: "0.0.0.0:8200"

output.elasticsearch:
  hosts: ["localhost:9200"]
  username: "elastic"
  password: "changeme"

setup.kibana:
  host: "localhost:5601"

Start APM Server:

./apm-server -e

Step 2: Install APM Agent

Java Application

Add Maven dependency:

<dependency>
    <groupId>co.elastic.apm</groupId>
    <artifactId>elastic-apm-agent</artifactId>
    <version>1.x.x</version>
</dependency>

Start with agent:

java -javaagent:/path/to/elastic-apm-agent.jar \
     -Delastic.apm.service_name=my-service \
     -Delastic.apm.server_urls=http://localhost:8200 \
     -Delastic.apm.secret_token=your-token \
     -jar my-application.jar

Node.js Application

Install agent:

npm install elastic-apm-node --save

Initialize in code:

// Must be at the very beginning of your app
const apm = require('elastic-apm-node').start({
  serviceName: 'my-service',
  serverUrl: 'http://localhost:8200',
  secretToken: 'your-token',
  environment: 'production'
})

Python Application

Install agent:

pip install elastic-apm

Django integration:

# settings.py
INSTALLED_APPS = (
   'elasticapm.contrib.django',
   # other apps
)

ELASTIC_APM = {
    'SERVICE_NAME': 'my-service',
    'SERVER_URL': 'http://localhost:8200',
    'SECRET_TOKEN': 'your-token',
    'ENVIRONMENT': 'production',
}

MIDDLEWARE = (
    'elasticapm.contrib.django.middleware.TracingMiddleware',
    # other middleware
)

Go Application

Install agent:

go get go.elastic.co/apm/v2

Instrument HTTP server:

import (
    "go.elastic.co/apm/module/apmhttp/v2"
    "go.elastic.co/apm/v2"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/", handleRequest)

    handler := apmhttp.Wrap(mux)
    http.ListenAndServe(":8080", handler)
}

Step 3: Configure Agent Settings

Common configuration options:

Environment Variables:

export ELASTIC_APM_SERVICE_NAME="my-service"
export ELASTIC_APM_SERVER_URL="http://localhost:8200"
export ELASTIC_APM_SECRET_TOKEN="your-token"
export ELASTIC_APM_ENVIRONMENT="production"
export ELASTIC_APM_TRANSACTION_SAMPLE_RATE="0.5"

Key Configuration Parameters:

service_name: Identifies your application
server_url: APM Server endpoint
secret_token: Authentication token
environment: deployment environment (dev, staging, prod)
transaction_sample_rate: Percentage of requests to trace (0.0-1.0)
capture_body: Whether to capture request/response bodies
log_level: Logging verbosity

Using Kibana APM UI

Accessing APM

Open Kibana
Navigate to Observability > APM
Select your service from the list

APM Dashboard Sections

Services Overview

List of all monitored services
Transaction throughput
Average response time
Error rates
Health status indicators

Service Details

Transaction timeline
Top transactions by impact
Error distribution
Dependencies and service map
Infrastructure metrics

Transactions View

Transaction types (request, background job, etc.)
Latency distribution
Throughput over time
Transaction traces

Trace Timeline

Waterfall view of distributed traces
Span duration and timing
Service dependencies
Database queries
External HTTP calls

Errors

Error occurrences over time
Error messages and types
Stack traces
Affected users
Error grouping

Metrics

CPU usage
Memory consumption
JVM heap (Java)
Garbage collection
Custom metrics

Advanced Features

Custom Spans and Transactions

Java:

import co.elastic.apm.api.ElasticApm;
import co.elastic.apm.api.Span;

Span span = ElasticApm.currentSpan()
    .startSpan("external", "http", "GET");
try {
    // Your code here
} finally {
    span.end();
}

Node.js:

const span = apm.startSpan('custom-operation')
if (span) {
  // Your code here
  span.end()
}

Python:

from elasticapm import capture_span

@capture_span('custom-operation')
def my_function():
    # Your code here
    pass

Custom Metrics

Node.js:

apm.setCustomContext({
  userId: '12345',
  accountType: 'premium',
  region: 'us-east-1'
})

Error Tracking

Manual error capture:

// Node.js
apm.captureError(new Error('Something went wrong'), {
  custom: {
    errorCode: 'ERR_001',
    userId: user.id
  }
})

Real User Monitoring (RUM)

Monitor browser-side performance:

Install RUM agent:

npm install @elastic/apm-rum --save

Initialize:

import { init as initApm } from '@elastic/apm-rum'

const apm = initApm({
  serviceName: 'my-frontend',
  serverUrl: 'http://localhost:8200',
  serviceVersion: '1.0.0'
})

Performance Best Practices

1. Sampling Strategy

Use appropriate sample rates (0.1-1.0)
Higher rates for critical services
Lower rates for high-traffic services
Dynamic sampling based on load

2. Reduce Agent Overhead

Disable body capture if not needed
Limit stack trace depth
Filter sensitive data
Use async processing

3. Optimize APM Server

Scale horizontally for high loads
Use appropriate hardware sizing
Configure queue settings
Monitor APM Server metrics

4. Data Retention

Implement ILM policies
Archive old traces
Delete unnecessary data
Use different retention for errors vs. metrics

5. Alerting Strategy

Set meaningful thresholds
Avoid alert fatigue
Use anomaly detection
Combine multiple conditions

Troubleshooting Common Issues

Agent Not Sending Data

Check:

APM Server is running and accessible
Correct server URL and secret token
Network connectivity
Agent logs for errors
Firewall rules

Verify:

curl -X GET "http://localhost:8200/"

High CPU Usage

Solutions:

Reduce sampling rate
Disable unnecessary features
Optimize instrumentation
Update to latest agent version

Missing Transactions

Causes:

Sampling rate too low
Transaction below threshold
Agent not properly initialized
Framework not supported

Data Not Appearing in Kibana

Check:

Elasticsearch is receiving data
Index templates are correct
Time range in Kibana
Data stream health

Frequently Asked Questions

Q: Does Elastic APM support microservices architectures?
A: Yes, Elastic APM excels at monitoring microservices with distributed tracing that tracks requests across multiple services.

Q: What is the performance overhead of APM agents?
A: APM agents are designed to be lightweight, typically adding less than 1-3% overhead with default sampling rates.

Q: Can I monitor serverless functions with Elastic APM?
A: Yes, APM agents support AWS Lambda, Azure Functions, and Google Cloud Functions with specific configurations.

Q: How long is APM data retained?
A: Data retention depends on your ILM policies. Default is typically 30 days, but it's configurable based on your needs.

Q: Can I correlate APM data with logs?
A: Yes, APM automatically adds trace IDs to logs, enabling correlation between traces and logs in Kibana.

Q: Is Elastic APM suitable for production environments?
A: Yes, Elastic APM is production-ready and used by many enterprises to monitor critical applications at scale.

Q: How do I secure APM data transmission?
A: Use secret tokens for authentication, enable TLS/SSL for encryption, and implement network security policies.

Q: Can I create custom dashboards with APM data?
A: Yes, APM data is stored in Elasticsearch, so you can create custom visualizations and dashboards in Kibana.

Q: Does Elastic APM support database query monitoring?
A: Yes, APM agents automatically capture database queries, including timing, query text, and database type.

Q: How do I monitor third-party API calls?
A: APM agents automatically instrument HTTP clients and capture external API calls, including timing and response codes.