Elasticsearch Forcemerge Operation: Complete Guide to Safe Usage

The forcemerge operation in Elasticsearch is a powerful maintenance API that manually triggers the merging of index segments. While it can significantly reduce disk usage and improve search performance in specific scenarios, it must be used carefully to avoid performance degradation and cluster instability.

You should consider using forcemerge when:

1. Working with read-only indices (logs rolled over, time-based indices, frozen data)
2. Indexing has fully stopped or can be paused safely
3. Disk usage is high due to deleted documents, not active growth
4. Preparing data for long-term retention or snapshots
5. Reducing segment count after bulk delete operations

What is Forcemerge?

Forcemerge is an Elasticsearch API that manually triggers the merging of Lucene segments within an index. Normally, Elasticsearch automatically merges segments in the background according to its merge policy. However, the forcemerge API allows you to explicitly control this process.

The operation consolidates multiple smaller segments into fewer, larger segments, which can:

• Reduce the number of segments
• Reclaim disk space from deleted or updated documents
• Improve search performance by reducing the number of files to search through

What Happens Behind the Scenes

When you execute a forcemerge operation, Elasticsearch performs the following steps:

1. Segment Selection: Identifies segments that can be merged based on the parameters provided
2. Merge Process: Reads data from multiple segments and writes it to new, consolidated segments
3. Document Cleanup: Physically removes documents marked as deleted (which normally only occupy space)
4. Segment Replacement: Replaces old segments with the new merged segments
5. Cleanup: Deletes the old segment files once they're no longer needed

During this process:

• CPU usage increases as the merge operation is computationally intensive
• Disk I/O spikes due to reading existing segments and writing new ones
• Temporary disk space is required (up to 2x the index size in worst cases)
• Search operations continue but may be slower due to resource contention
• Indexing is blocked while merging occurs on write-active indices

When to Run Forcemerge

Recommended Scenarios

1. Read-Only Indices

# After rolling over a time-based index
POST /logs-2025-01-*/_forcemerge?max_num_segments=1

Ideal for indices that will no longer receive writes, such as historical logs or archived data.

2. After Bulk Delete Operations

# Clean up deleted documents
POST /my-index/_forcemerge?only_expunge_deletes=true

When you've deleted a significant portion of documents and need to reclaim disk space.

3. Before Snapshotting

# Optimize before backup
POST /important-index/_forcemerge?max_num_segments=1

Reduces snapshot size and speeds up snapshot creation for archived data.

4. Indexing Pause Windows

# During maintenance window
POST /active-index/_forcemerge?max_num_segments=5

When you can temporarily pause indexing without impacting operations.

Scenarios to Avoid

• Active write-heavy indices: Forcemerge blocks writes and can cause severe performance issues
• Continuously updated indices: The benefits will be quickly negated by new segments
• Resource-constrained clusters: Can overwhelm CPU and I/O capacity
• During peak hours: Will impact query performance when users need it most

Risks and Downsides

Performance Impact

CPU and I/O Spike: Forcemerge is resource-intensive and can saturate CPU and disk I/O, degrading search and indexing performance across the cluster.

Indexing Delays: On active indices, writes are blocked during the merge process, leading to indexing backlogs and potential timeouts.

Memory Pressure: Large merge operations can increase heap usage and potentially trigger garbage collection pauses.

Operational Risks

Temporary Disk Space: Requires significant temporary disk space (potentially double the index size), which can cause disk full errors if not carefully planned.

Long-Running Operations: On large indices, forcemerge can take hours to complete, during which the cluster may be vulnerable to failures.

Shard Relocation Issues: If a node fails during forcemerge, recovery can be slower due to large segment sizes.

Counterproductive Effects

Wasted Effort on Active Indices: If indexing continues, new segments will be created immediately, negating the benefits of the merge.

Oversized Segments: Merging to a single segment creates very large files that are harder to manage and slower to recover.

Merge Policy Conflicts: Can interfere with Elasticsearch's automatic merge policy, potentially creating suboptimal segment distributions.

Benefits of Forcemerge

When used appropriately, forcemerge provides several significant benefits:

Disk Space Recovery

• Deleted Document Cleanup: Physically removes documents marked as deleted, reclaiming disk space
• Reduced Overhead: Fewer segments mean less filesystem overhead and metadata
• Snapshot Efficiency: Smaller, optimized indices create faster, more compact snapshots

Search Performance

• Fewer Segments to Search: Reduces the number of files Lucene must open and search through
• Better Cache Utilization: Larger segments can be cached more efficiently
• Improved Query Speed: Particularly beneficial for read-only indices with complex queries

Cluster Management

• Simplified Shard Management: Fewer segments make shard allocation and relocation faster
• Reduced File Handles: Lower file descriptor usage across the cluster
• Predictable Performance: Read-only optimized indices have more consistent response times

How to Perform a Forcemerge

Basic Forcemerge

Merge to a specific number of segments:

POST /my-index/_forcemerge?max_num_segments=1

Expunge Deletes Only

Clean up deleted documents without full merging (safer for active indices):

POST /my-index/_forcemerge?only_expunge_deletes=true

Forcemerge Multiple Indices

Use wildcards to merge multiple indices:

POST /logs-2025-01-*/_forcemerge?max_num_segments=1

Forcemerge with Flush

Ensure all data is persisted before merging:

POST /my-index/_forcemerge?max_num_segments=1&flush=true

Common Parameters

• max_num_segments: Target number of segments (default: dynamic)
• only_expunge_deletes: Only merge segments with deletions (default: false)
• flush: Perform a flush before merging (default: true)

Using only_expunge_deletes

The only_expunge_deletes=true parameter is a safer alternative for active indices:

When to Use

High Delete Ratio: When deleted documents occupy significant space but you still need to index

POST /active-index/_forcemerge?only_expunge_deletes=true

Disk Pressure: When approaching disk watermarks and need quick space recovery

# Target indices with high delete ratios
POST /high-delete-index/_forcemerge?only_expunge_deletes=true

Minimal Performance Impact: When you can't afford full merge overhead

# During business hours on active index
POST /busy-index/_forcemerge?only_expunge_deletes=true

Benefits of only_expunge_deletes

• Lower Resource Usage: Only merges segments containing deleted documents
• Faster Execution: Completes more quickly than full forcemerge
• Continued Indexing: Less likely to block writes for extended periods
• Incremental Benefit: Can be run periodically without major disruption

Best Practices for Running Forcemerge

Timing and Execution

Run Off-Peak

# Schedule during maintenance window
# 2 AM example
0 2 * * * curl -X POST "localhost:9200/logs-*/_forcemerge?max_num_segments=1"

One Index at a Time

# Sequential approach
POST /index-1/_forcemerge?max_num_segments=1
# Wait for completion
POST /index-2/_forcemerge?max_num_segments=1

Avoid Cluster-Wide Concurrent Merges

• Never run forcemerge on all indices simultaneously
• Limit concurrent operations to prevent resource exhaustion
• Monitor cluster health between operations

Segment Count Guidelines

Read-Only Indices: Merge to 1 segment for maximum efficiency

POST /readonly-index/_forcemerge?max_num_segments=1

Occasionally Updated Indices: Keep 5-10 segments for flexibility

POST /semi-active-index/_forcemerge?max_num_segments=5

Active Indices: Avoid forcemerge or use only_expunge_deletes

POST /active-index/_forcemerge?only_expunge_deletes=true

Preventive Measures

Avoid on Active Indices

• Never use _forcemerge on write-active indices unless absolutely necessary
• Prefer only_expunge_deletes=true for active indices requiring cleanup
• Consider if the performance issue can be solved another way

Investigate Root Causes

• Heavy deletes may indicate data lifecycle issues
• Consider Index Lifecycle Management (ILM) for automatic optimization
• Review delete-by-query operations and their frequency
• Analyze if updates can be structured differently

Monitor Cluster Resources

• Check disk space before starting (ensure 2x free space)
• Monitor CPU and I/O during operation
• Watch heap usage for signs of memory pressure
• Set up alerts for long-running merge operations

Monitoring Forcemerge Operations

Check Merge Status

View ongoing merge operations:

GET /_cat/tasks?detailed=true&v&actions=*forcemerge*

Monitor merge statistics:

GET /my-index/_stats/merge

Check segment information:

GET /my-index/_segments

Using Pulse for Comprehensive Monitoring

For real-time visibility into forcemerge operations and their impact on cluster performance, use Pulse.

Pulse provides:

• Real-Time Merge Tracking: Visualize ongoing forcemerge operations across your cluster
• Performance Impact Analysis: See how merges affect query latency, indexing throughput, and resource usage
• Segment Visualization: Monitor segment counts and sizes before, during, and after merge operations
• Disk Space Monitoring: Track disk usage changes as deleted documents are expunged
• Resource Metrics: CPU, I/O, and heap usage during intensive merge operations
• Historical Trends: Analyze the effectiveness of forcemerge operations over time
• Alert Configuration: Set up notifications for long-running or problematic merges

With Pulse, you can make data-driven decisions about when to run forcemerge, which indices benefit most, and whether the operation achieved its goals without negatively impacting cluster health.

Task Management API

Cancel a running forcemerge if needed:

# First, find the task ID
GET /_tasks?detailed=true&actions=*forcemerge*

# Then cancel it
POST /_tasks/TASK_ID/_cancel

Frequently Asked Questions

Q: How long does a forcemerge operation take?
A: Duration varies based on index size, segment count, and available resources. Small indices (< 10GB) may complete in minutes, while large indices (> 100GB) can take hours. Monitor progress using the _tasks API and plan accordingly.

Q: Can I safely forcemerge an index that receives occasional writes?
A: It depends on write frequency. If writes are rare (a few documents per hour), you can use only_expunge_deletes=true safely. For more frequent writes, avoid forcemerge or schedule it during a maintenance window when writes can be paused.

Q: Should I always merge to max_num_segments=1?
A: Only for truly read-only indices. For indices that might receive occasional writes or require shard relocation, keeping 5-10 segments provides better flexibility. Single-segment indices are harder to manage during failures and relocations.

Q: What's the difference between forcemerge and optimize?
A: They're the same operation. The _optimize API was deprecated in favor of _forcemerge to better reflect its nature as a forced, manual operation rather than an automatic optimization.

Q: How much disk space do I need before running forcemerge?
A: Plan for at least 2x the index size in available disk space. Forcemerge creates new segments before deleting old ones, temporarily doubling storage requirements. Running out of disk during forcemerge can cause node failures.

Q: Can forcemerge improve indexing performance?
A: No, forcemerge typically degrades indexing performance. It's designed for read-only or read-heavy workloads. For active indices with heavy writes, rely on Elasticsearch's automatic merge policy instead.

Q: How often should I run forcemerge?
A: Only when needed, not on a schedule. Run it on indices after they become read-only (e.g., after ILM rollover), after major bulk delete operations, or when preparing snapshots. Regular automatic merging handles most optimization needs.

Q: What happens if a node fails during forcemerge?
A: The forcemerge operation will fail, but data integrity is maintained. However, recovery may be slower due to partially merged segments. If the primary shard was being merged, Elasticsearch will promote a replica and retry shard recovery.

Q: Does forcemerge affect cluster health status?
A: The operation itself doesn't change cluster health status, but resource exhaustion from poorly planned forcemerge operations can lead to cascading failures, slow queries, and potential yellow/red cluster status if nodes become unresponsive.

Q: Can I run forcemerge on system indices?
A: Generally avoid forcemerging system indices (.kibana, .security, etc.) unless specifically recommended by Elastic documentation. These indices are usually active and small enough that automatic merging suffices.