Cross-cluster replication (CCR) in OpenSearch enables you to replicate indices from a leader cluster to one or more follower clusters. This feature is essential for disaster recovery, geographic distribution, and maintaining data consistency across multiple environments. This guide provides step-by-step instructions for setting up CCR on self-managed OpenSearch clusters.
For managed clusters on Amazon OpenSearch Service, see our guide for Amazon OpenSearch Service Cross-Cluster Replication which covers AWS-specific configurations and best practices.
What is Cross-Cluster Replication?
Cross-cluster replication allows you to:
- Replicate data from a leader cluster to follower clusters
- Enable disaster recovery by maintaining copies in different locations
- Reduce latency by placing data closer to users
- Separate read and write operations across clusters
- Maintain data consistency with automatic synchronization
Prerequisites
Before setting up CCR, ensure you have:
- Two or more OpenSearch clusters (leader and follower)
- OpenSearch version 1.0.0 or later (CCR was introduced in OpenSearch 1.0.0)
- Network connectivity between clusters
- Security plugin enabled on both clusters
- Appropriate permissions for CCR operations
- Sufficient storage on follower clusters
Step 1: Configure Network Connectivity
1.1 Enable Remote Cluster Connections
On both leader and follower clusters, configure the elasticsearch.yml
file to allow remote cluster connections:
# On leader cluster elasticsearch.yml
cluster.remote.leader_cluster.seeds: ["follower-cluster-node1:9300", "follower-cluster-node2:9300"]
cluster.remote.leader_cluster.skip_unavailable: true
# On follower cluster elasticsearch.yml
cluster.remote.follower_cluster.seeds: ["leader-cluster-node1:9300", "leader-cluster-node2:9300"]
cluster.remote.follower_cluster.skip_unavailable: true
1.2 Configure Security Settings
Ensure security is properly configured for cross-cluster communication:
# On both clusters elasticsearch.yml
plugins.security.ssl.transport.enabled: true
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.pemcert_filepath: /path/to/cert.pem
plugins.security.ssl.transport.pemkey_filepath: /path/to/key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /path/to/ca.pem
Step 2: Set Up Authentication and Authorization
2.1 Create CCR-Specific Roles
Create roles with appropriate permissions for CCR operations:
# On leader cluster - create role for follower cluster access
curl -X POST "https://leader-cluster:9200/_security/role/ccr_leader_role" \
-H "Content-Type: application/json" \
-u admin:admin \
-k \
-d '{
"cluster": [
"cluster:monitor/state",
"cluster:monitor/health",
"cluster:admin/remote_info"
],
"indices": [
{
"names": ["*"],
"privileges": [
"read",
"view_index_metadata"
]
}
]
}'
# On follower cluster - create role for CCR operations
curl -X POST "https://follower-cluster:9200/_security/role/ccr_follower_role" \
-H "Content-Type: application/json" \
-u admin:admin \
-k \
-d '{
"cluster": [
"cluster:monitor/state",
"cluster:admin/remote_info",
"cluster:admin/ccr/follow_index",
"cluster:admin/ccr/auto_follow_pattern",
"cluster:admin/ccr/pause_follow",
"cluster:admin/ccr/resume_follow",
"cluster:admin/ccr/forget_follower"
],
"indices": [
{
"names": ["*"],
"privileges": [
"create_index",
"write",
"read",
"view_index_metadata"
]
}
]
}'
2.2 Create Users and Assign Roles
# Create user on leader cluster
curl -X POST "https://leader-cluster:9200/_security/user/ccr_user" \
-H "Content-Type: application/json" \
-u admin:admin \
-k \
-d '{
"password": "secure_password",
"roles": ["ccr_leader_role"],
"full_name": "CCR User"
}'
# Create user on follower cluster
curl -X POST "https://follower-cluster:9200/_security/user/ccr_user" \
-H "Content-Type: application/json" \
-u admin:admin \
-k \
-d '{
"password": "secure_password",
"roles": ["ccr_follower_role"],
"full_name": "CCR User"
}'
Step 3: Configure Remote Cluster Connection
3.1 Add Remote Cluster on Follower
On the follower cluster, add the leader cluster as a remote cluster:
curl -X PUT "https://follower-cluster:9200/_cluster/settings" \
-H "Content-Type: application/json" \
-u ccr_user:secure_password \
-k \
-d '{
"persistent": {
"cluster.remote.leader_cluster.seeds": ["leader-cluster-node1:9300", "leader-cluster-node2:9300"],
"cluster.remote.leader_cluster.skip_unavailable": true
}
}'
3.2 Verify Remote Cluster Connection
# Check remote cluster info
curl -X GET "https://follower-cluster:9200/_remote/info" \
-u ccr_user:secure_password \
-k
# Expected response should show leader_cluster as connected
Step 4: Set Up Cross-Cluster Replication
4.1 Create Index on Leader Cluster
First, create an index on the leader cluster that you want to replicate:
# Create test index on leader
curl -X PUT "https://leader-cluster:9200/test-index" \
-H "Content-Type: application/json" \
-u ccr_user:secure_password \
-k \
-d '{
"settings": {
"index.number_of_shards": 3,
"index.number_of_replicas": 1
}
}'
# Add some test data
curl -X POST "https://leader-cluster:9200/test-index/_doc" \
-H "Content-Type: application/json" \
-u ccr_user:secure_password \
-k \
-d '{
"message": "Test document for CCR",
"timestamp": "2024-01-01T00:00:00Z"
}'
4.2 Start Following the Index
On the follower cluster, start following the leader index:
curl -X PUT "https://follower-cluster:9200/test-index/_ccr/follow" \
-H "Content-Type: application/json" \
-u ccr_user:secure_password \
-k \
-d '{
"remote_cluster": "leader_cluster",
"leader_index": "test-index",
"settings": {
"index.number_of_replicas": 0
}
}'
4.3 Verify Replication Status
Check the status of the CCR follow operation:
# Check follow stats
curl -X GET "https://follower-cluster:9200/test-index/_ccr/stats" \
-u ccr_user:secure_password \
-k
# Check index status
curl -X GET "https://follower-cluster:9200/test-index/_stats" \
-u ccr_user:secure_password \
-k
Step 5: Advanced CCR Configuration
5.1 Auto-Follow Patterns
Set up auto-follow patterns to automatically replicate new indices:
curl -X PUT "https://follower-cluster:9200/_ccr/auto_follow/leader_cluster" \
-H "Content-Type: application/json" \
-u ccr_user:secure_password \
-k \
-d '{
"remote_cluster": "leader_cluster",
"leader_index_patterns": ["logs-*", "metrics-*"],
"follow_index_pattern": ""
}'
5.2 Configure Replication Settings
Customize replication behavior with advanced settings:
curl -X PUT "https://follower-cluster:9200/test-index/_ccr/follow" \
-H "Content-Type: application/json" \
-u ccr_user:secure_password \
-k \
-d '{
"remote_cluster": "leader_cluster",
"leader_index": "test-index",
"settings": {
"index.number_of_replicas": 0
},
"max_read_request_operation_count": 5120,
"max_read_request_size": "32mb",
"max_outstanding_read_requests": 12,
"max_read_request_retries": 3,
"read_poll_timeout": "1m"
}'
Step 6: Monitoring and Management
6.1 Monitor CCR Status
Regularly check the health of your CCR setup:
# Get all CCR stats
curl -X GET "https://follower-cluster:9200/_ccr/stats" \
-u ccr_user:secure_password \
-k
# Get specific index CCR info
curl -X GET "https://follower-cluster:9200/test-index/_ccr/info" \
-u ccr_user:secure_password \
-k
6.2 Pause and Resume Replication
# Pause replication
curl -X POST "https://follower-cluster:9200/test-index/_ccr/pause_follow" \
-u ccr_user:secure_password \
-k
# Resume replication
curl -X POST "https://follower-cluster:9200/test-index/_ccr/resume_follow" \
-u ccr_user:secure_password \
-k
6.3 Stop Following
To stop following an index and make it a regular index:
curl -X POST "https://follower-cluster:9200/test-index/_ccr/unfollow" \
-u ccr_user:secure_password \
-k
Step 7: Troubleshooting Common Issues
7.1 Connection Issues
If remote cluster connection fails:
# Check cluster health
curl -X GET "https://leader-cluster:9200/_cluster/health" \
-u ccr_user:secure_password \
-k
# Check remote cluster info
curl -X GET "https://follower-cluster:9200/_remote/info" \
-u ccr_user:secure_password \
-k
# Verify network connectivity
telnet leader-cluster-node1 9300
7.2 Authentication Issues
If authentication fails:
# Test authentication
curl -X GET "https://leader-cluster:9200/_cluster/health" \
-u ccr_user:secure_password \
-k
# Check user permissions
curl -X GET "https://leader-cluster:9200/_security/user/ccr_user" \
-u admin:admin \
-k
7.3 Replication Lag
Monitor replication lag and performance:
# Check detailed CCR stats
curl -X GET "https://follower-cluster:9200/test-index/_ccr/stats?pretty" \
-u ccr_user:secure_password \
-k
# Look for these key metrics:
# - "follower_global_checkpoint": Current checkpoint
# - "leader_global_checkpoint": Leader checkpoint
# - "last_requested_seq_no": Last requested sequence number
Best Practices
8.1 Performance Optimization
- Monitor network latency between clusters
- Adjust batch sizes based on your data volume
- Use appropriate shard counts for your data size
- Monitor memory usage on follower clusters
8.2 Security Considerations
- Use dedicated users for CCR operations
- Implement least privilege access control
- Enable SSL/TLS for all cross-cluster communication
- Regularly rotate passwords and certificates
8.3 Monitoring and Alerting
- Set up alerts for replication lag
- Monitor cluster health of both leader and follower
- Track CCR metrics in your monitoring system
- Set up dashboards for CCR status visualization
In summary, Cross-Cluster Replication in OpenSearch provides a robust solution for data replication across multiple clusters. By following this guide, you can set up a reliable CCR infrastructure that supports disaster recovery, geographic distribution, and operational flexibility. Remember to monitor your CCR setup regularly and adjust configurations based on your specific requirements and performance needs.