Elasticsearch Shard Sizing Best Practices

Proper shard sizing is crucial for Elasticsearch performance. Shards that are too small waste resources, while shards that are too large impact recovery times and query performance. This guide provides best practices for optimal shard sizing.

Optimal Shard Size Guidelines

General Recommendation

10-50 GB per shard is usually optimal.

More specifically:

Upper Limit

Elasticsearch recommends a soft upper limit of 50 GB per shard:

• Larger shards take longer to recover
• Query performance may degrade
• Rebalancing becomes more disruptive

Lower Limit

Avoid shards smaller than 1 GB in production:

• Small shards waste memory (overhead per shard)
• Coordination costs outweigh benefits
• Many small shards hurt query performance

Calculating Shard Size

For New Indices

Number of shards = Expected data size / Target shard size

Example:
Expected data: 200 GB
Target shard size: 40 GB
Primary shards needed: 200 / 40 = 5 shards

For Time-Series Data

Shards per time period = Data volume per period / Target shard size

Example:
Daily ingest: 80 GB
Target shard size: 40 GB
Shards per day: 80 / 40 = 2 primary shards per daily index

Checking Current Shard Sizes

GET /_cat/shards?v&h=index,shard,prirep,store&s=store:desc

Implementation Strategies

Strategy 1: ILM with Size-Based Rollover

The best approach for time-series data:

PUT _ilm/policy/optimal_shard_size
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "40gb",
            "max_age": "7d"
          }
        }
      }
    }
  }
}

This creates new indices when shards reach 40 GB or after 7 days.

Strategy 2: Fixed Shard Count with Templates

For predictable data volumes:

PUT _index_template/sized_indices
{
  "index_patterns": ["data-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1
    }
  }
}

Strategy 3: Single Shard for Small Indices

For indices under 50 GB total:

PUT /small-index
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  }
}

Workload-Specific Recommendations

High-Volume Logging

Characteristics: High write rate, time-based queries, eventual deletion

PUT _ilm/policy/logging_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": {
            "number_of_shards": 1
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      }
    }
  }
}

Full-Text Search

Characteristics: Complex queries, lower write rate, long retention

PUT /search-index
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 2
  }
}

Aim for 10-30 GB shards for better search latency.

Analytics/Aggregations

Characteristics: Large aggregations, wide time ranges

PUT _ilm/policy/analytics_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_primary_shard_size": "40gb"
          }
        }
      }
    }
  }
}

High-Availability Requirements

If you need to spread shards across more nodes:

PUT /ha-index
{
  "settings": {
    "number_of_shards": 6,  // Spread across 6+ nodes
    "number_of_replicas": 2,
    "index.routing.allocation.total_shards_per_node": 2
  }
}

Adjusting Existing Indices

Shrinking Over-Sharded Indices

// 1. Make index read-only
PUT /my-index/_settings
{
  "index.blocks.write": true
}

// 2. Shrink to fewer shards
POST /my-index/_shrink/my-index-shrunk
{
  "settings": {
    "index.number_of_shards": 1
  }
}

Splitting Under-Sharded Indices

// 1. Make index read-only
PUT /my-index/_settings
{
  "index.blocks.write": true
}

// 2. Split to more shards (must be multiple of original)
POST /my-index/_split/my-index-split
{
  "settings": {
    "index.number_of_shards": 4
  }
}

Monitoring Shard Size

Regular Health Check

# Large shards (>50 GB)
GET /_cat/shards?v&h=index,shard,store&bytes=gb | awk '$3 > 50'

# Small shards (<1 GB)
GET /_cat/shards?v&h=index,shard,store&bytes=gb | awk '$3 < 1'

Alerting Thresholds

Common Mistakes to Avoid

Mistake 1: One Shard Per Document Type

// Wrong: Too many small indices
PUT /users, /products, /orders, /reviews, /comments...

// Better: Consolidate related data
PUT /application-data

Mistake 2: Fixed Daily Shards Regardless of Volume

// Wrong: 10 shards per day for 1 GB/day data
// Results in tiny shards

// Better: Use size-based rollover
"rollover": {
  "max_primary_shard_size": "40gb"
}

Mistake 3: Ignoring Replica Impact

Remember: Total shards = Primary shards × (1 + Replicas)

5 primaries × 2 replicas = 15 total shards

Sizing Decision Tree

What's your data volume?
│
├── < 50 GB total
│   └── Use 1 primary shard
│
├── 50 GB - 500 GB total
│   └── Use 2-10 primary shards (target 30-50 GB each)
│
└── > 500 GB or time-series
    └── Use ILM with size-based rollover
        └── max_primary_shard_size: 40gb

Related Topics

• Elasticsearch Too Many Shards Performance
• Elasticsearch Shard Imbalance Performance Issues
• Elasticsearch Capacity Planning Guide
• Elasticsearch Sizing Calculator