Monitoring OpenSearch with AWS CloudWatch: Complete Setup Guide

Amazon OpenSearch Service provides built-in integration with AWS CloudWatch for monitoring cluster health, performance, and operational metrics. While some metrics are provided out of the box, they are often not sufficient.

This guide walks you through setting up a complete monitoring and alerting solution using CloudWatch to ensure your OpenSearch clusters remain healthy and performant.

Prerequisites

Before setting up monitoring, ensure you have:

AWS Account Access: Administrative or appropriate IAM permissions for CloudWatch and OpenSearch Service
OpenSearch Domain: An active Amazon OpenSearch Service domain
IAM Permissions: The following permissions for your user/role:
- cloudwatch:PutMetricData
- cloudwatch:GetMetricData
- cloudwatch:PutDashboard
- cloudwatch:DescribeAlarms
- es:DescribeElasticsearchDomain
- es:ListTags

Step 1: Enable CloudWatch Monitoring

1.1 Access Your OpenSearch Domain

Log into the AWS Management Console
Navigate to Amazon OpenSearch Service
Select your domain from the list
Click on the Monitoring tab

1.2 Configure CloudWatch Integration

CloudWatch monitoring is enabled by default for Amazon OpenSearch Service domains. To verify and configure:

In the Monitoring tab, you'll see the CloudWatch metrics section
Ensure Enhanced monitoring is enabled (recommended for production)
Set the Monitoring interval to 1 minute for detailed metrics or 5 minutes for standard monitoring

# Using AWS CLI to enable enhanced monitoring
aws es update-elasticsearch-domain-config \
  --domain-name your-domain-name \
  --domain-endpoint-options '{"EnforceHTTPS":true,"TLSSecurityPolicy":"Policy-Min-TLS-1-2-2019-07"}' \
  --ebs-options '{"EBSEnabled":true,"VolumeType":"gp3","VolumeSize":100}' \
  --encryption-at-rest-options '{"Enabled":true}' \
  --node-to-node-encryption-options '{"Enabled":true}' \
  --advanced-options '{"rest.action.multi.allow_explicit_index":"true"}'

Step 2: Key Metrics to Monitor

2.1 Cluster Health Metrics

Monitor these critical cluster-level metrics:

ClusterStatus: Overall cluster health (Green, Yellow, Red)
Nodes: Number of data nodes in the cluster
ClusterIndexWritesBlocked: Whether writes are blocked
ClusterIndexWritesBlocked: Whether reads are blocked

2.2 Performance Metrics

Track performance-related metrics:

SearchLatency: Average search query latency
IndexingLatency: Average indexing latency
CPUUtilization: CPU usage across nodes
JVMMemoryPressure: JVM heap memory pressure
FreeStorageSpace: Available storage space

2.3 Operational Metrics

Monitor operational aspects:

AutomatedSnapshotFailure: Failed automated snapshots
ClusterUsedSpace: Total used storage space
ClusterIndexWritesBlocked: Write operations blocked
ClusterReadOnly: Cluster in read-only mode

Step 3: Create CloudWatch Dashboards

3.1 Create a Basic Monitoring Dashboard

Navigate to CloudWatch in the AWS Console
Click Dashboards in the left sidebar
Click Create dashboard
Name your dashboard: OpenSearch-Cluster-Monitoring

3.2 Add Key Widgets

Cluster Health Widget

{
  "metrics": [
    [ "AWS/ES", "ClusterStatus", "DomainName", "your-domain-name", "ClientId", "your-account-id" ]
  ],
  "period": 300,
  "stat": "Average",
  "region": "us-east-1",
  "title": "Cluster Status"
}

Performance Metrics Widget

{
  "metrics": [
    [ "AWS/ES", "SearchLatency", "DomainName", "your-domain-name", "ClientId", "your-account-id" ],
    [ ".", "IndexingLatency", ".", ".", ".", "." ]
  ],
  "period": 300,
  "stat": "Average",
  "region": "us-east-1",
  "title": "Search and Indexing Latency"
}

Resource Utilization Widget

{
  "metrics": [
    [ "AWS/ES", "CPUUtilization", "DomainName", "your-domain-name", "ClientId", "your-account-id" ],
    [ ".", "JVMMemoryPressure", ".", ".", ".", "." ],
    [ ".", "FreeStorageSpace", ".", ".", ".", "." ]
  ],
  "period": 300,
  "stat": "Average",
  "region": "us-east-1",
  "title": "Resource Utilization"
}

Step 4: Set Up CloudWatch Alarms

4.1 Critical Alarms

Cluster Status Alarm

Go to CloudWatch → Alarms
Click Create alarm
Select Metric → OpenSearch Service
Choose ClusterStatus metric
Configure alarm:
- Threshold type: Static
- Condition: Less than 1 (Red status)
- Evaluation period: 1 out of 1 datapoints
- Period: 5 minutes

High CPU Utilization Alarm

Create new alarm for CPUUtilization
Configure:
- Threshold type: Static
- Condition: Greater than 80%
- Evaluation period: 2 out of 3 datapoints
- Period: 5 minutes

Low Storage Space Alarm

Create alarm for FreeStorageSpace
Configure:
- Threshold type: Static
- Condition: Less than 10GB
- Evaluation period: 1 out of 1 datapoints
- Period: 5 minutes

4.2 Performance Alarms

High Search Latency Alarm

# Using AWS CLI to create search latency alarm
aws cloudwatch put-metric-alarm \
  --alarm-name "OpenSearch-High-Search-Latency" \
  --alarm-description "Alert when search latency exceeds 1000ms" \
  --metric-name "SearchLatency" \
  --namespace "AWS/ES" \
  --statistic "Average" \
  --period 300 \
  --threshold 1000 \
  --comparison-operator "GreaterThanThreshold" \
  --evaluation-periods 2 \
  --alarm-actions "arn:aws:sns:us-east-1:123456789012:your-sns-topic"

High Indexing Latency Alarm

aws cloudwatch put-metric-alarm \
  --alarm-name "OpenSearch-High-Indexing-Latency" \
  --alarm-description "Alert when indexing latency exceeds 500ms" \
  --metric-name "IndexingLatency" \
  --namespace "AWS/ES" \
  --statistic "Average" \
  --period 300 \
  --threshold 500 \
  --comparison-operator "GreaterThanThreshold" \
  --evaluation-periods 2 \
  --alarm-actions "arn:aws:sns:us-east-1:123456789012:your-sns-topic"

Step 5: Configure Notifications

5.1 Set Up SNS Topic

Navigate to SNS in AWS Console
Click Create topic
Choose Standard topic type
Name: OpenSearch-Alerts
Add subscribers (email, SMS, or other endpoints)

5.2 Configure Alarm Actions

For each alarm you create:

In the alarm configuration, add Alarm actions
Select your SNS topic: OpenSearch-Alerts
Optionally add OK actions to notify when issues are resolved

Step 6: Advanced Monitoring Setup

6.1 Custom Metrics

You can publish custom metrics using the CloudWatch API:

import boto3
import time

cloudwatch = boto3.client('cloudwatch')

def publish_custom_metric(metric_name, value, unit='Count'):
    cloudwatch.put_metric_data(
        Namespace='OpenSearch/Custom',
        MetricData=[
            {
                'MetricName': metric_name,
                'Value': value,
                'Unit': unit,
                'Timestamp': time.time()
            }
        ]
    )

6.2 Log Monitoring

Enable CloudWatch Logs for OpenSearch:

In your OpenSearch domain settings, go to Logs
Enable Error logs and Search slow logs
Set retention period (7-30 days recommended)
Create log-based alarms for critical errors

6.3 Anomaly Detection

Set up anomaly detection for key metrics:

In CloudWatch, go to Anomaly Detection
Select your OpenSearch metrics
Configure sensitivity and training period
Create alarms based on anomalies

Step 7: Best Practices

7.1 Alarm Configuration

Avoid alert fatigue: Set appropriate thresholds and evaluation periods
Use different severity levels: Critical, Warning, and Info
Implement escalation: Different actions for different severity levels
Test alarms: Regularly test your alarm configurations

7.2 Dashboard Organization

Group related metrics: Cluster health, performance, and operational metrics
Use appropriate time ranges: 1 hour for real-time, 24 hours for trends
Add annotations: Document incidents and maintenance windows
Share dashboards: Make dashboards available to relevant teams

7.3 Cost Optimization

Monitor CloudWatch costs: Set up billing alarms
Use appropriate granularity: 5-minute periods for most metrics
Archive old data: Use CloudWatch Insights for historical analysis
Optimize custom metrics: Batch metric publishing when possible

Step 8: Troubleshooting Common Issues

8.1 Missing Metrics

If metrics aren't appearing:

Verify enhanced monitoring is enabled
Check IAM permissions
Ensure domain is active and healthy
Wait for metrics to populate (can take 5-15 minutes)

8.2 Alarm Not Triggering

If alarms aren't working:

Check alarm configuration and thresholds
Verify SNS topic and subscriptions
Test alarm actions manually
Review CloudWatch logs for errors

8.3 High False Positives

To reduce false positives:

Increase evaluation periods
Adjust thresholds based on historical data
Use anomaly detection instead of static thresholds
Implement hysteresis (different thresholds for alarm and OK states)

Additional Resources

Frequently Asked Questions

Q: How often should I review and update my monitoring setup?
A: Review your monitoring configuration monthly, and update thresholds and alarms based on changing usage patterns and business requirements. Conduct quarterly reviews of dashboard effectiveness and alarm accuracy.

Q: Can I monitor multiple OpenSearch domains with a single dashboard?
A: Yes, you can create a multi-domain dashboard by adding multiple metrics with different domain names. Use CloudWatch's multi-metric widgets to display all domains in a single view.

Q: What's the difference between basic and enhanced monitoring?
A: Basic monitoring provides metrics at 5-minute intervals, while enhanced monitoring provides metrics at 1-minute intervals and includes additional per-instance metrics. Enhanced monitoring incurs additional costs but provides better granularity.

Q: How do I set up monitoring for OpenSearch Serverless?
A: OpenSearch Serverless uses different metrics and monitoring approaches. Refer to the AWS documentation for serverless-specific monitoring setup, which includes collection-level metrics and different CloudWatch integration patterns.

Q: Can I export CloudWatch metrics to external monitoring tools?
A: Yes, you can use CloudWatch APIs, AWS CLI, or third-party tools to export metrics to external monitoring solutions like Grafana, Prometheus, or Datadog for additional analysis and visualization.