OpenSearch Cross-Cluster Replication (CCR): Complete Setup Guide

Pulse - Elasticsearch Operations Done Right

On this page

What is Cross-Cluster Replication? Prerequisites Step 1: Configure Network Connectivity Step 2: Set Up Authentication and Authorization Step 3: Configure Remote Cluster Connection Step 4: Set Up Cross-Cluster Replication Step 5: Advanced CCR Configuration Step 6: Monitoring and Management Step 7: Troubleshooting Common Issues Best Practices

Cross-cluster replication (CCR) in OpenSearch enables you to replicate indices from a leader cluster to one or more follower clusters. This feature is essential for disaster recovery, geographic distribution, and maintaining data consistency across multiple environments. This guide provides step-by-step instructions for setting up CCR on self-managed OpenSearch clusters.

For managed clusters on Amazon OpenSearch Service, see our guide for Amazon OpenSearch Service Cross-Cluster Replication which covers AWS-specific configurations and best practices.

What is Cross-Cluster Replication?

Cross-cluster replication allows you to:

  • Replicate data from a leader cluster to follower clusters
  • Enable disaster recovery by maintaining copies in different locations
  • Reduce latency by placing data closer to users
  • Separate read and write operations across clusters
  • Maintain data consistency with automatic synchronization

Prerequisites

Before setting up CCR, ensure you have:

  1. Two or more OpenSearch clusters (leader and follower)
  2. OpenSearch version 1.0.0 or later (CCR was introduced in OpenSearch 1.0.0)
  3. Network connectivity between clusters
  4. Security plugin enabled on both clusters
  5. Appropriate permissions for CCR operations
  6. Sufficient storage on follower clusters

Step 1: Configure Network Connectivity

1.1 Enable Remote Cluster Connections

On both leader and follower clusters, configure the elasticsearch.yml file to allow remote cluster connections:

# On leader cluster elasticsearch.yml
cluster.remote.leader_cluster.seeds: ["follower-cluster-node1:9300", "follower-cluster-node2:9300"]
cluster.remote.leader_cluster.skip_unavailable: true

# On follower cluster elasticsearch.yml
cluster.remote.follower_cluster.seeds: ["leader-cluster-node1:9300", "leader-cluster-node2:9300"]
cluster.remote.follower_cluster.skip_unavailable: true

1.2 Configure Security Settings

Ensure security is properly configured for cross-cluster communication:

# On both clusters elasticsearch.yml
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.pemcert_filepath: /path/to/cert.pem
plugins.security.ssl.transport.pemkey_filepath: /path/to/key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /path/to/ca.pem

Step 2: Set Up Authentication and Authorization

2.1 Create CCR-Specific Roles

Create roles with appropriate permissions for CCR operations:

# On leader cluster - create role for follower cluster access
curl -X POST "https://leader-cluster:9200/_security/role/ccr_leader_role" \
  -H "Content-Type: application/json" \
  -u admin:admin \
  -k \
  -d '{
    "cluster": [
      "cluster:monitor/state",
      "cluster:monitor/health",
      "cluster:admin/remote_info"
    ],
    "indices": [
      {
        "names": ["*"],
        "privileges": [
          "read",
          "view_index_metadata"
        ]
      }
    ]
  }'

# On follower cluster - create role for CCR operations
curl -X POST "https://follower-cluster:9200/_security/role/ccr_follower_role" \
  -H "Content-Type: application/json" \
  -u admin:admin \
  -k \
  -d '{
    "cluster": [
      "cluster:monitor/state",
      "cluster:admin/remote_info",
      "cluster:admin/ccr/follow_index",
      "cluster:admin/ccr/auto_follow_pattern",
      "cluster:admin/ccr/pause_follow",
      "cluster:admin/ccr/resume_follow",
      "cluster:admin/ccr/forget_follower"
    ],
    "indices": [
      {
        "names": ["*"],
        "privileges": [
          "create_index",
          "write",
          "read",
          "view_index_metadata"
        ]
      }
    ]
  }'

2.2 Create Users and Assign Roles

# Create user on leader cluster
curl -X POST "https://leader-cluster:9200/_security/user/ccr_user" \
  -H "Content-Type: application/json" \
  -u admin:admin \
  -k \
  -d '{
    "password": "secure_password",
    "roles": ["ccr_leader_role"],
    "full_name": "CCR User"
  }'

# Create user on follower cluster
curl -X POST "https://follower-cluster:9200/_security/user/ccr_user" \
  -H "Content-Type: application/json" \
  -u admin:admin \
  -k \
  -d '{
    "password": "secure_password",
    "roles": ["ccr_follower_role"],
    "full_name": "CCR User"
  }'

Step 3: Configure Remote Cluster Connection

3.1 Add Remote Cluster on Follower

On the follower cluster, add the leader cluster as a remote cluster:

curl -X PUT "https://follower-cluster:9200/_cluster/settings" \
  -H "Content-Type: application/json" \
  -u ccr_user:secure_password \
  -k \
  -d '{
    "persistent": {
      "cluster.remote.leader_cluster.seeds": ["leader-cluster-node1:9300", "leader-cluster-node2:9300"],
      "cluster.remote.leader_cluster.skip_unavailable": true
    }
  }'

3.2 Verify Remote Cluster Connection

# Check remote cluster info
curl -X GET "https://follower-cluster:9200/_remote/info" \
  -u ccr_user:secure_password \
  -k

# Expected response should show leader_cluster as connected

Step 4: Set Up Cross-Cluster Replication

4.1 Create Index on Leader Cluster

First, create an index on the leader cluster that you want to replicate:

# Create test index on leader
curl -X PUT "https://leader-cluster:9200/test-index" \
  -H "Content-Type: application/json" \
  -u ccr_user:secure_password \
  -k \
  -d '{
    "settings": {
      "index.number_of_shards": 3,
      "index.number_of_replicas": 1
    }
  }'

# Add some test data
curl -X POST "https://leader-cluster:9200/test-index/_doc" \
  -H "Content-Type: application/json" \
  -u ccr_user:secure_password \
  -k \
  -d '{
    "message": "Test document for CCR",
    "timestamp": "2024-01-01T00:00:00Z"
  }'

4.2 Start Following the Index

On the follower cluster, start following the leader index:

curl -X PUT "https://follower-cluster:9200/test-index/_ccr/follow" \
  -H "Content-Type: application/json" \
  -u ccr_user:secure_password \
  -k \
  -d '{
    "remote_cluster": "leader_cluster",
    "leader_index": "test-index",
    "settings": {
      "index.number_of_replicas": 0
    }
  }'

4.3 Verify Replication Status

Check the status of the CCR follow operation:

# Check follow stats
curl -X GET "https://follower-cluster:9200/test-index/_ccr/stats" \
  -u ccr_user:secure_password \
  -k

# Check index status
curl -X GET "https://follower-cluster:9200/test-index/_stats" \
  -u ccr_user:secure_password \
  -k

Step 5: Advanced CCR Configuration

5.1 Auto-Follow Patterns

Set up auto-follow patterns to automatically replicate new indices:

curl -X PUT "https://follower-cluster:9200/_ccr/auto_follow/leader_cluster" \
  -H "Content-Type: application/json" \
  -u ccr_user:secure_password \
  -k \
  -d '{
    "remote_cluster": "leader_cluster",
    "leader_index_patterns": ["logs-*", "metrics-*"],
    "follow_index_pattern": ""
  }'

5.2 Configure Replication Settings

Customize replication behavior with advanced settings:

curl -X PUT "https://follower-cluster:9200/test-index/_ccr/follow" \
  -H "Content-Type: application/json" \
  -u ccr_user:secure_password \
  -k \
  -d '{
    "remote_cluster": "leader_cluster",
    "leader_index": "test-index",
    "settings": {
      "index.number_of_replicas": 0
    },
    "max_read_request_operation_count": 5120,
    "max_read_request_size": "32mb",
    "max_outstanding_read_requests": 12,
    "max_read_request_retries": 3,
    "read_poll_timeout": "1m"
  }'

Step 6: Monitoring and Management

6.1 Monitor CCR Status

Regularly check the health of your CCR setup:

# Get all CCR stats
curl -X GET "https://follower-cluster:9200/_ccr/stats" \
  -u ccr_user:secure_password \
  -k

# Get specific index CCR info
curl -X GET "https://follower-cluster:9200/test-index/_ccr/info" \
  -u ccr_user:secure_password \
  -k

6.2 Pause and Resume Replication

# Pause replication
curl -X POST "https://follower-cluster:9200/test-index/_ccr/pause_follow" \
  -u ccr_user:secure_password \
  -k

# Resume replication
curl -X POST "https://follower-cluster:9200/test-index/_ccr/resume_follow" \
  -u ccr_user:secure_password \
  -k

6.3 Stop Following

To stop following an index and make it a regular index:

curl -X POST "https://follower-cluster:9200/test-index/_ccr/unfollow" \
  -u ccr_user:secure_password \
  -k

Step 7: Troubleshooting Common Issues

7.1 Connection Issues

If remote cluster connection fails:

# Check cluster health
curl -X GET "https://leader-cluster:9200/_cluster/health" \
  -u ccr_user:secure_password \
  -k

# Check remote cluster info
curl -X GET "https://follower-cluster:9200/_remote/info" \
  -u ccr_user:secure_password \
  -k

# Verify network connectivity
telnet leader-cluster-node1 9300

7.2 Authentication Issues

If authentication fails:

# Test authentication
curl -X GET "https://leader-cluster:9200/_cluster/health" \
  -u ccr_user:secure_password \
  -k

# Check user permissions
curl -X GET "https://leader-cluster:9200/_security/user/ccr_user" \
  -u admin:admin \
  -k

7.3 Replication Lag

Monitor replication lag and performance:

# Check detailed CCR stats
curl -X GET "https://follower-cluster:9200/test-index/_ccr/stats?pretty" \
  -u ccr_user:secure_password \
  -k

# Look for these key metrics:
# - "follower_global_checkpoint": Current checkpoint
# - "leader_global_checkpoint": Leader checkpoint
# - "last_requested_seq_no": Last requested sequence number

Best Practices

8.1 Performance Optimization

  1. Monitor network latency between clusters
  2. Adjust batch sizes based on your data volume
  3. Use appropriate shard counts for your data size
  4. Monitor memory usage on follower clusters

8.2 Security Considerations

  1. Use dedicated users for CCR operations
  2. Implement least privilege access control
  3. Enable SSL/TLS for all cross-cluster communication
  4. Regularly rotate passwords and certificates

8.3 Monitoring and Alerting

  1. Set up alerts for replication lag
  2. Monitor cluster health of both leader and follower
  3. Track CCR metrics in your monitoring system
  4. Set up dashboards for CCR status visualization

In summary, Cross-Cluster Replication in OpenSearch provides a robust solution for data replication across multiple clusters. By following this guide, you can set up a reliable CCR infrastructure that supports disaster recovery, geographic distribution, and operational flexibility. Remember to monitor your CCR setup regularly and adjust configurations based on your specific requirements and performance needs.

Pulse - Elasticsearch Operations Done Right

Pulse can solve your OpenSearch issues

Subscribe to the Pulse Newsletter

Get early access to new Pulse features, insightful blogs & exclusive events , webinars, and workshops.

We use cookies to provide an optimized user experience and understand our traffic. To learn more, read our use of cookies; otherwise, please choose 'Accept Cookies' to continue using our website.