Managing Elasticsearch clusters in environments is a common challenge for organizations that run search and analytics work that is vital to the business. Whether you are moving to infrastructure, setting up disaster recovery or changing operating systems, keeping Elasticsearch data consistent and minimizing downtime are the top priorities.<\/p>\n
The primary objectives were to:<\/p>\n
In this blog I will explain:<\/p>\n
Cross-cluster replication is an Elasticsearch feature that enables you to replicate indices from one cluster (the leader cluster) to another cluster (the follower cluster). We have seen that the follower index is constantly pulling the changes from the leader index to keep the data up to time.<\/p>\n
Cross-Cluster Replication is great for:<\/p>\n
Cross-Cluster Replication works at the shard level. Every follower shard pulls changes from its leader shard. Two main phases exist:<\/p>\n
The transaction log is very important. It records every write operation. It serves as the source for synchronization.<\/p>\n
Both clusters must be compatible versions before CCR is set up.<\/p>\n
If possible, keep both clusters on the same major version. When upgrading major versions or the final minor of the older version, upgrade the leader first.<\/p>\n
First install Elasticsearch on your cluster nodes. Start by installing the required Java runtime. Then install the Elasticsearch RPM package. After installation, reload the system daemon. Enable the Elasticsearch service so that it starts automatically when the system boots.<\/p>\n
Finally reset the password, for the superuser. For any nodes generate an enrollment token from the node. Use that token to reconfigure and add the nodes to the Elasticsearch cluster.<\/p>\n
Both clusters need a license that supports CCR. We started a trial license on both sides using the license API and verified it was active. This is a step but easy to forget if you’re moving fast.<\/p>\n
\/\/ Start trial license on both clusters\r\ncurl -XPOST http:\/\/es-ip:9200\/_license\/start_trial?acknowledge=true -u elastic:password\r\n\r\n\/\/Verify license status\r\ncurl -X GET \"http:\/\/es-ip:9200\/_license\" -u elastic:password<\/code><\/pre>\nStep 3 Configure TLS Certificates<\/h3>\n
Secure communication between clusters means you must exchange CA certificates. We found this step essential because CCR communication happens over the transport layer and that layer is encrypted by default in Elasticsearch versions.<\/p>\n
On the Leader Cluster<\/strong><\/p>\nExport the CA certificate from the transport keystore with OpenSSL. Transfer it to follower cluster, we have used S3 bucket for that. Make sure to do this so the follower cluster can retrieve the file.<\/p>\n
\/\/ Export the CA certificate from the transport keystore\r\n openssl pkcs12 -in \/etc\/elasticsearch\/certs\/transport.p12 -cacerts \r\n -nokey -out cluster-a-ca.crt\r\n\r\n \/\/ Upload to shared S3 location\r\n aws s3 cp cluster-a-ca.crt s3:\/\/bucket-name\/ --region ap-south-1<\/code><\/pre>\nOn the Follower Cluster<\/strong><\/p>\nDownload the leader clusters CA certificate fetch the truststore password, from the Elasticsearch keystore. Then import the certificate into the local transport keystore using the keytool utility.<\/p>\n
\/\/ Download leader cluster's CA certificate\r\n aws s3 cp s3:\/\/bucket-name\/cluster-a-ca.crt --region ap-south-1 .\r\n\r\n \/\/ Get the truststore password\r\n \/bin\/elasticsearch-keystore show xpack.security.transport.ssl.truststore.secure_password\r\n\r\n \/\/ Import the certificate into the local transport keystore\r\n keytool -importcert -keystore \/etc\/elasticsearch\/certs\/transport.p12 \r\n -storetype PKCS12 -alias cluster-a-ca -file cluster-a-ca.crt \r\n -noprompt -storepass <truststore_password><\/code><\/pre>\nRepeat the same steps so the leader cluster also trusts the follower clusters CA certificate. Also double-check that the reverse steps are correct.<\/p>\n
Configure Node Roles<\/strong><\/p>\nOn the follower cluster add remote_cluster_client to the node roles in elasticsearch.yml. This role is needed for any node that needs to talk to a cluster. This role is needed for any node that needs to talk to a cluster. Restart Elasticsearch on each node one, at a time to make the change take effect without stopping the cluster.<\/p>\n
node.roles: [master, data, remote_cluster_client]<\/code><\/pre>\nStep 4: Add Remote Cluster<\/h3>\n
On the follower cluster set up the connection to the leader cluster using the cluster settings API. Give the seed nodes of the leader cluster turn on transport compression to save bandwidth and set skip_unavailable to false if you want availability.<\/p>\n
curl -X PUT \"http:\/\/es-ip:9200\/_cluster\/settings\" \r\n -H 'Content-Type: application\/json' -u elastic:<password> \r\n -d '{\r\n \"persistent\": {\r\n \"cluster\": {\r\n \"remote\": {\r\n \"leader-cluster\": {\r\n \"seeds\": [\r\n \"leader-node-1:9300\",\r\n \"leader-node-2:9300\"\r\n ],\r\n \"transport.compress\": true,\r\n \"skip_unavailable\": false\r\n }\r\n }\r\n }\r\n }\r\n }'<\/code><\/pre>\nStep 5: Start Replication<\/h3>\n
Begin Replication Create a follower index on the follower cluster that follows the leader index. This is done with the CCR follow API, where you enter the name of the cluster and the name of the leader index.<\/p>\n
curl -XPUT \"http:\/\/es-ip:9200\/<follower_index>\/_ccr\/follow?pretty\" \r\n-H 'Content-Type: application\/json' -u elastic:<password> \r\n-d '{\r\n \"remote_cluster\": \"leader-cluster\",\r\n \"leader_index\": \"<leader_index>\"\r\n}'<\/code><\/pre>\nOnce replication starts check the index status and replication details. The follower index should show up in the list of indexes. The CCR stats API will show how the replication is going including any delay, between the leader and the follower.<\/p>\n
\/\/ Check indices\r\ncurl -X GET \"http:\/\/es-ip:9200\/_cat\/indices?pretty\" -u elastic:<password>\r\n\r\n\/\/ Check CCR stats\r\ncurl -X GET \"http:\/\/es-ip:9200\/_ccr\/stats?pretty\" -u elastic:<password><\/code><\/pre>\nStep 6: Monitor Replication<\/h3>\n
Monitoring must be done. The metrics that matter are Replication lag, the gap between the leader and follower checkpoints. Also watch for reads and writes because they show connectivity or permission problems. Then check sync status to see if Replication is actively paused or in an error state.<\/p>\n
\/\/ Check follower stats for a specific index\r\ncurl -XGET http:\/\/es-ip:9200\/<follower_index>\/_ccr\/stats?pretty -u elastic:password<\/code><\/pre>\nCheck out these metrics. It’s so much easier to catch a problem early than to fix it after it’s been quietly failing for hours.<\/p>\n
Disabling CCR on the Follower Cluster<\/h2>\n
To disable CCR, there are four step process which need to perform:<\/p>\n
Step 1: Pause the Follower Index<\/strong><\/p>\nThis stops ongoing replication without deleting anything.<\/p>\n
curl -XPOST http:\/\/es-ip:9200\/<follower_index>\/_ccr\/pause_follow -u elastic:password<\/code><\/pre>\nStep 2: Close the Follower Index<\/strong><\/p>\nClose the follower index to prepare it for the operation also you cannot unfollow an open index.<\/p>\n
curl -X POST \"http:\/\/es-ip:9200\/<follower_index>\/_close\" -u elastic:password<\/code><\/pre>\nStep 3: Unfollow the Index<\/strong><\/p>\nThis converts the follower into a regular read-write index. This step is irreversible. Once you unfollow it there is no way to go back to following without setting it up from scratch.<\/p>\n
curl -XPOST http:\/\/es-ip:9200\/<follower_index>\/_ccr\/unfollow -u elastic:password<\/code><\/pre>\nStep 4: Access the Index again<\/strong><\/p>\nReopen the index to make it available for reads and writes.<\/p>\n
curl -XPOST \"http:\/\/es-ip:9200\/<follower_index>\/_open\" -u elastic:password<\/code><\/pre>\nThen we went back to the Basic license, as we didn’t need the trial anymore. We also commented out remote_cluster_client from the node roles on all nodes and restarted them one by one. Leaving that role enabled when you’re not using CCR is just unnecessary overhead.<\/p>\n
\/\/ Revert to Basic license\r\ncurl -XPOST http:\/\/es-ip:9200\/_license\/start_basic?acknowledge=true -u elastic:password\r\n\r\n\/\/ verify data\r\ncurl -XGET \"http:\/\/es-ip:9200\/_cat\/indices?v&s=index&pretty\" -u elastic:password<\/code><\/pre>\nBest Practices<\/h2>\n\n- Use dedicated network links between datacenters if you can It makes a difference<\/li>\n
- Configure your timeouts based on your actual network latency not the defaults<\/li>\n
- Size your follower cluster properly It’s not just a passive receiver Its doing real work<\/li>\n
- Use TLS Create CCR users with minimal privileges instead of using the elastic superuser<\/li>\n
- Rotate certificates regularly<\/li>\n<\/ul>\n
Conclusion<\/h2>\n
Cross-Cluster Replication gives a way to move Elasticsearch data between clusters with very little downtime. Here’s what we learned from our setup:<\/p>\n
\n- Cross-Cluster Replication replicates indices automatically from the leader cluster to the follower cluster.<\/li>\n
- You must set up TLS certificates correctly to keep communication, between clusters<\/li>\n
- The remote_cluster_client role needs to be set on the follower cluster nodes.<\/li>\n
- Unfollow actions can’t be undone so think carefully before switching over.<\/li>\n<\/ul>\n
Following this guide helps you build a CCR migration plan. It keeps your data consistent. Your business runs smoothly during the move.<\/p>\n","protected":false},"excerpt":{"rendered":"
Introduction Managing Elasticsearch clusters in environments is a common challenge for organizations that run search and analytics work that is vital to the business. Whether you are moving to infrastructure, setting up disaster recovery or changing operating systems, keeping Elasticsearch data consistent and minimizing downtime are the top priorities. The primary objectives were to: Achieve […]<\/p>\n","protected":false},"author":2382,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":0,"footnotes":""},"categories":[2348],"tags":[9033,9035,9036,7788,1524],"class_list":["post-83323","post","type-post","status-publish","format-standard","hentry","category-devops-technology","tag-ccr","tag-ccrsetup","tag-clusterreplication","tag-datamigration","tag-elasticsearch-2"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t