{"id":83298,"date":"2026-10-11T00:43:31","date_gmt":"2026-10-10T19:13:31","guid":{"rendered":"https:\/\/2thenew.xyz\/blog\/?p=83298"},"modified":"2026-10-27T23:13:11","modified_gmt":"2026-10-27T17:43:11","slug":"elasticsearch-has-20000-shards-what-actually-happens-when-you-restart-a-node","status":"publish","type":"post","link":"https:\/\/2thenew.xyz\/blog\/elasticsearch-has-20000-shards-what-actually-happens-when-you-restart-a-node\/","title":{"rendered":"Elasticsearch Has 20,000 Shards: What Actually Happens When You Restart a Node?"},"content":{"rendered":"

Elasticsearch Has 20,000 Shards: What Actually Happens When You Restart a Node?<\/h1>\n

Introduction<\/h2>\n

Restarting an Elasticsearch node sounds simple:<\/p>\n

\"Elasticsearch

Node Restart Flow<\/p><\/div>\n

That assumption changes when a cluster contains thousands of shards.<\/p>\n

In one production environment I worked with, the Elasticsearch cluster had approximately 20,000 shards<\/strong>. A node restart could trigger extensive shard recovery, allocation, disk I\/O, network traffic and cluster-state changes.<\/p>\n

The server could be healthy while Elasticsearch still took significant time to return to a stable state.<\/p>\n

This article explains what happens during a restart, how I troubleshoot slow recovery, and what I learned from operating a high-shard cluster.<\/p>\n

1. Why Shard Count Matters<\/h2>\n

A shard is more than a number in Elasticsearch. Each shard represents a Lucene index with its own segments, files and metadata.<\/p>\n

For example:<\/p>\n

2,000 indices \u00d7 10 shards = 20,000 shards<\/pre>\n
A node restart can therefore trigger a large number of allocation and recovery operations.<\/p>\n
The impact depends on:<\/p>\n
    \n
  • Shard size<\/li>\n
  • Number of shards on the affected node<\/li>\n
  • Primary\/replica configuration<\/li>\n
  • Disk and network performance<\/li>\n
  • CPU and memory<\/li>\n
  • Current indexing\/search workload<\/li>\n
  • Allocation rules<\/li>\n<\/ul>\n
    Two clusters with the same shard count can therefore behave very differently during recovery.<\/p>\n
     <\/p>\n
    2. What Happens When a Node Restarts?<\/h2>\n
    When Elasticsearch detects that a node has left the cluster, it evaluates the shards that were hosted there.<\/p>\n
    \"ES
    Restart Checklist<\/p><\/div>\n
    For each affected shard, Elasticsearch determines:<\/p>\n
      \n
    • Whether it is a primary or replica<\/li>\n
    • Whether another valid copy exists<\/li>\n
    • Where it can be allocated<\/li>\n
    • Whether allocation rules allow placement<\/li>\n
    • Whether sufficient disk space is available<\/li>\n
    • Whether recovery is being throttled<\/li>\n<\/ul>\n
      If a valid copy is available, Elasticsearch can recover the shard locally or from another node.<\/p>\n
      With thousands of shards, these operations can happen at significant scale.<\/p>\n
      A node restart in a large Elasticsearch cluster is not just a service restart. It is a distributed recovery event.<\/strong><\/p>\n
      3. How I Troubleshoot Slow Recovery<\/h2>\n
      I avoid changing Elasticsearch settings immediately. First, I determine whether the bottleneck is recovery, allocation, disk performance or another resource constraint.<\/p>\n
      Step 1: Check Cluster Health<\/h4>\n
      curl -s \"http:\/\/localhost:9200\/_cluster\/health?pretty\"<\/pre>\n
      I mainly check:<\/p>\n
      status\r\nnumber_of_nodes\r\nactive_shards\r\ninitializing_shards\r\nrelocating_shards\r\nunassigned_shards<\/pre>\n
      For example:<\/p>\n
      initializing_shards: 300\r\nunassigned_shards: 0<\/pre>\n
      indicates active recovery.<\/p>\n
      If unassigned shards remain high while very few shards are initializing, allocation needs investigation.<\/p>\n
      Step 2: Check Shard Distribution<\/h4>\n
      curl -s \"http:\/\/localhost:9200\/_cat\/shards?v\"<\/pre>\n
      For focused output:<\/p>\n
      curl -s \"http:\/\/localhost:9200\/_cat\/shards?v&h=index,shard,prirep,state,node,store\"<\/pre>\n
      Look for:<\/p>\n
      UNASSIGNED\r\nINITIALIZING\r\nRELOCATING<\/pre>\n
      Also check whether one node is carrying significantly more shards than others.<\/p>\n
      Step 3: Check Recovery<\/h4>\n
      curl -s \"http:\/\/localhost:9200\/_cat\/recovery?v\"<\/pre>\n
      Useful fields include:<\/p>\n
      index\r\nshard\r\nstage\r\nsource_host\r\ntarget_host\r\nfiles_percent\r\nbytes_percent\r\ntime<\/pre>\n
      This helps determine whether recovery is actually progressing.<\/p>\n
      A cluster can appear stuck while recovery is steadily moving forward.<\/p>\n
      Step 4: Explain Unassigned Shards<\/h4>\n
      For unassigned shards, I use:<\/p>\n
      curl -X GET \"http:\/\/localhost:9200\/_cluster\/allocation\/explain?pretty\" \r\n-H \"Content-Type: application\/json\"<\/pre>\n
      One important lesson from production troubleshooting is:<\/p>\n
      UNASSIGNED does not automatically mean ALLOCATION_FAILED.<\/strong><\/p>\n
      For example, a sample response might show:<\/p>\n
      {\r\n  \"index\" : \"sample-data-processor-2026.09.10\",\r\n  \"shard\" : 0,\r\n  \"primary\" : true,\r\n  \"current_state\" : \"unassigned\",\r\n  \"unassigned_info\" : {\r\n    \"reason\" : \"CLUSTER_RECOVERED\",\r\n    \"last_allocation_status\" : \"throttled\"\r\n  },\r\n  \"can_allocate\" : \"yes\",\r\n  \"allocate_explanation\" : \"Elasticsearch can allocate the shard.\",\r\n  \"target_node\" : {\r\n    \"name\" : \"node-1\"\r\n  },\r\n  \"node_allocation_decisions\" : [\r\n    {\r\n      \"node_name\" : \"node-1\",\r\n      \"node_decision\" : \"yes\",\r\n      \"store\" : {\r\n        \"in_sync\" : true\r\n      }\r\n    }\r\n  ]\r\n}<\/pre>\n
      The important clues are:<\/p>\n
        \n
      • CLUSTER_RECOVERED<\/strong> \u2014 the shard became unassigned during cluster recovery.<\/li>\n
      • throttled<\/strong> \u2014 the previous allocation status was throttled.<\/li>\n
      • can_allocate: yes<\/strong> \u2014 Elasticsearch can currently allocate the shard.<\/li>\n
      • node_decision: yes<\/strong> \u2014 a suitable target node exists.<\/li>\n
      • in_sync: true<\/strong> \u2014 a valid shard copy is available.<\/li>\n<\/ul>\n
        This means an unassigned shard is not necessarily stuck because of an allocation failure. Before forcing allocation or changing recovery settings, check whether recovery is progressing.<\/p>\n
        For a specific shard, provide the target explicitly:<\/p>\n
        curl -X GET \"http:\/\/localhost:9200\/_cluster\/allocation\/explain?pretty\" \r\n-H \"Content-Type: application\/json\" \r\n-d '{\r\n  \"index\": \"sample-data-processor-2026.09.10\",\r\n  \"shard\": 0,\r\n  \"primary\": true\r\n}'<\/pre>\n
        4. Recovery Is a Resource-Balancing Problem<\/h2>\n
        Faster recovery is not always better.<\/p>\n
        Recovery consumes:<\/p>\n
          \n
        • Disk I\/O<\/li>\n
        • Network bandwidth<\/li>\n
        • CPU<\/li>\n
        • Memory<\/li>\n
        • File descriptors<\/li>\n<\/ul>\n
          At the same time, Elasticsearch may still be handling indexing, searches, replication, merges and Kibana queries.<\/p>\n
          The objective should therefore be:<\/p>\n
          \"Recovery
          Balance Recovery<\/p><\/div>\n
          Maximum safe recovery speed, not maximum recovery speed.<\/strong><\/p>\n
          5. Recovery Settings That Matter<\/h2>\n
          Two settings became particularly relevant when investigating recovery behavior.<\/p>\n
          cluster.routing.allocation.node_initial_primaries_recoveries<\/h4>\n
          This controls concurrent initial primary shard recoveries on a node.<\/p>\n
          Check the current configuration with:<\/p>\n
          curl -s \"http:\/\/localhost:9200\/_cluster\/settings?include_defaults=true&pretty\"<\/pre>\n
          A persistent value can be configured through the cluster settings API:<\/p>\n
          {\r\n  \"persistent\": {\r\n    \"cluster.routing.allocation.node_initial_primaries_recoveries\": 20\r\n  }\r\n}<\/pre>\n
          The value should be tuned according to the environment rather than copied blindly.<\/p>\n
          indices.recovery.max_bytes_per_sec<\/h4>\n
          This controls recovery bandwidth.<\/p>\n
          For example:<\/p>\n
          {\r\n  \"persistent\": {\r\n    \"indices.recovery.max_bytes_per_sec\": \"125mb\"\r\n  }\r\n}<\/pre>\n
          Increasing it may shorten recovery time, but can also increase resource contention.<\/p>\n
          My approach is:<\/p>\n
          Measure\r\n\u2193\r\nChange one setting\r\n\u2193\r\nObserve\r\n\u2193\r\nMeasure again\r\n\u2193\r\nKeep \/ revert<\/pre>\n
          Measure first, tune second.<\/strong><\/p>\n
          6. Don’t Forget the Operating System<\/h2>\n
          Elasticsearch recovery is closely tied to the underlying server.<\/p>\n
          Disk<\/h4>\n
          iostat -xz 1<\/pre>\n
          Check utilization, latency, throughput and queue depth.<\/p>\n
          If the disk is already saturated, increasing recovery concurrency can make recovery slower rather than faster.<\/p>\n
          File Descriptors<\/h4>\n
          Check:<\/p>\n
          ulimit -n<\/pre>\n
          For systemd:<\/p>\n
          systemctl show elasticsearch | grep LimitNOFILE<\/pre>\n
          For the running process:<\/p>\n
          cat \/proc\/$(pgrep -f elasticsearch | head -1)\/limits | grep \"open files\"<\/pre>\n
          We encountered Too many open files<\/strong> during Elasticsearch operations.<\/p>\n
          The lesson was simple:<\/p>\n
          Increasing OS limits can remove a constraint, but it does not fix excessive shard counts.<\/strong><\/p>\n
          7. Shard Count Is an Architectural Problem Too<\/h2>\n
          After troubleshooting recovery repeatedly, another question becomes unavoidable:<\/p>\n
          Why do we have so many shards?<\/strong><\/p>\n
          Consider:<\/p>\n
          10,000 indices \u00d7 2 shards = 20,000 shards<\/pre>\n
          versus:<\/p>\n
          2,000 indices \u00d7 10 shards = 20,000 shards<\/pre>\n
          Both produce the same shard count, but their operational behavior can differ significantly depending on shard size and workload.<\/p>\n
          There is no universal answer to:<\/p>\n
          “How many shards should one node have?”<\/strong><\/p>\n
          It depends on shard size, hardware, heap, workload, recovery requirements and cluster architecture.<\/p>\n
          Adding more nodes can distribute the workload, but it does not reduce the total number of shards.<\/p>\n
          Recovery tuning is an operational optimization. Reducing unnecessary shard creation is an architectural optimization.<\/strong><\/p>\n
          8. Lessons From Elasticsearch Upgrades<\/h2>\n
          Recovery issues also became more complicated during Elasticsearch and Kibana upgrades.<\/p>\n
          Across multiple upgrades, we encountered:<\/p>\n
            \n
          • Older indices created under Elasticsearch 7.x<\/li>\n
          • Index write blocks<\/li>\n
          • Unassigned shards<\/li>\n
          • Allocation failures<\/li>\n
          • Disk pressure<\/li>\n
          • File descriptor limits<\/li>\n
          • Java\/JVM requirements<\/li>\n
          • Kibana compatibility issues<\/li>\n<\/ul>\n
            This reinforced an important principle:<\/p>\n
            Do not treat an Elasticsearch upgrade as a package installation.<\/strong><\/p>\n
            Before an upgrade, verify cluster health, snapshots, index compatibility, disk capacity, JVM requirements, Elasticsearch\/Kibana compatibility and recovery status.<\/p>\n
            A healthy cluster before an upgrade is one of the strongest safeguards against a difficult upgrade.<\/p>\n
            9. A Safer Node Restart Checklist<\/h2>\n
            Before Restart<\/h4>\n
              \n
            • Check cluster health<\/li>\n
            • Check unassigned shards<\/li>\n
            • Check active recovery<\/li>\n
            • Check disk, CPU and memory<\/li>\n
            • Check recent Elasticsearch errors<\/li>\n
            • Verify snapshots\/recovery strategy<\/li>\n<\/ul>\n
              During Restart<\/h4>\n
                \n
              • Stop Elasticsearch cleanly<\/li>\n
              • Confirm the process has stopped<\/li>\n
              • Start Elasticsearch<\/li>\n
              • Verify the node rejoins the cluster<\/li>\n<\/ul>\n
                After Restart<\/h4>\n
                  \n
                • Check _cluster\/health<\/code><\/li>\n
                • Check _cat\/shards<\/code><\/li>\n
                • Check _cat\/recovery<\/code><\/li>\n
                • Check unassigned shards<\/li>\n
                • Use _cluster\/allocation\/explain<\/code> if required<\/li>\n
                • Monitor disk I\/O, CPU and network<\/li>\n
                • Check Elasticsearch logs<\/li>\n<\/ul>\n
                  Recommended troubleshooting flow:<\/strong><\/p>\n
                  \"
                  Shard Recovery<\/p><\/div>\n
                  10. Final Takeaway<\/h2>\n
                  A small Elasticsearch cluster can make a node restart look like a simple service operation.<\/p>\n
                  A large cluster makes it a distributed recovery event<\/strong>.<\/p>\n
                  With approximately 20,000 shards, a restart can trigger thousands of allocation and recovery operations while the cluster continues serving production traffic.<\/p>\n
                  When recovery is slow, ask:<\/p>\n
                  Is the bottleneck:\r\n- Disk?\r\n- CPU?\r\n- Network?\r\n- Allocation?\r\n- Recovery throttling?\r\n- File descriptors?\r\n- Shard count?\r\n- Shard size?<\/pre>\n
                  Once the bottleneck is understood, recovery settings can be tuned safely.<\/p>\n
                  The goal is not maximum recovery speed.<\/p>\n
                  The goal is to restore cluster health as quickly as possible without taking the production workload down with it.<\/strong><\/p>\n
                  If high shard counts repeatedly make routine maintenance difficult, recovery tuning is only part of the solution. The longer-term question is whether the index and shard architecture itself needs to change.<\/p>\n
                  References<\/h2>\n
                    \n
                  • Elastic Documentation \u2014 Index Recovery Settings
                    \nhttps:\/\/www.elastic.co\/guide\/en\/elasticsearch\/reference\/current\/recovery.html<\/li>\n
                  • Elastic Documentation \u2014 Diagnose Unassigned Shards
                    \nhttps:\/\/www.elastic.co\/docs\/troubleshoot\/elasticsearch\/diagnose-unassigned-shards<\/li>\n
                  • Elastic Documentation \u2014 Cluster Allocation Explain API
                    \nhttps:\/\/www.elastic.co\/docs\/api\/doc\/elasticsearch\/v9\/operation\/operation-cluster-allocation-explain<\/li>\n
                  • Elastic Documentation \u2014 Total Number of Shards Per Node
                    \nhttps:\/\/www.elastic.co\/docs\/troubleshoot\/elasticsearch\/increase-cluster-shard-limit<\/li>\n<\/ul>\n
                     <\/p>\n","protected":false},"excerpt":{"rendered":"
                    Elasticsearch Has 20,000 Shards: What Actually Happens When You Restart a Node? Introduction Restarting an Elasticsearch node sounds simple: That assumption changes when a cluster contains thousands of shards. In one production environment I worked with, the Elasticsearch cluster had approximately 20,000 shards. A node restart could trigger extensive shard recovery, allocation, disk I\/O, network […]<\/p>\n","protected":false},"author":1500,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":0,"footnotes":""},"categories":[2348],"tags":[248,1892,1524,288],"class_list":["post-83298","post","type-post","status-publish","format-standard","hentry","category-devops-technology","tag-aws","tag-devops","tag-elasticsearch-2","tag-logging"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t