Elasticsearch 8.x Breaking Changes and Cluster Issues

Elasticsearch 8.x introduced significant changes that can cause issues during upgrades or when configuring new clusters. This guide covers the most impactful breaking changes and how to address them.

Major Breaking Changes

Security Enabled by Default

Change: Security features (authentication, TLS) are enabled by default in 8.x.

Common Issues:

Connection refused without authentication
TLS certificate errors
"unable to authenticate user" errors

Solutions:

Get auto-generated credentials (during first start):

# Password printed at first startup
grep "elastic user" /var/log/elasticsearch/elasticsearch.log

Reset password:

bin/elasticsearch-reset-password -u elastic

Disable security (not recommended for production):

# elasticsearch.yml
xpack.security.enabled: false
xpack.security.enrollment.enabled: false

Configure TLS certificates:

bin/elasticsearch-certutil ca
bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12

Java 17+ Required

Change: Elasticsearch 8.x requires Java 17 or later (bundled JDK recommended).

Issues:

Startup failures with older Java
GC behavior differences

Solution: Use bundled JDK or ensure Java 17+:

# Use bundled JDK (recommended)
export ES_JAVA_HOME=  # Unset to use bundled

# Or verify Java version
java -version  # Must be 17+

API Changes

Removal of Types

Change: Document types removed (deprecated since 7.x).

Old:

PUT /my-index/my-type/1

New:

PUT /my-index/_doc/1

Query DSL Changes

Removed: indices_boost format changed

// 7.x
"indices_boost": [{"my-index": 1.4}]

// 8.x
"indices_boost": [{"my-index": 1.4}]  // Same, but some internal changes

Discovery Configuration

Change: discovery.type: single-node behavior changed.

For single-node development:

# elasticsearch.yml
discovery.type: single-node

For multi-node clusters:

# elasticsearch.yml
discovery.seed_hosts: ["host1", "host2", "host3"]
# cluster.initial_master_nodes only for initial bootstrap

Deprecation of Features

REST API Deprecations

_type in document APIs removed
Some _cat API columns renamed
Response format changes in some APIs

Setting Deprecations

Check for deprecated settings:

GET /_cluster/settings?include_defaults=true&flat_settings=true

Common Upgrade Issues

Issue 1: Cluster Won't Form

Symptoms: Nodes can't discover each other after upgrade

Causes:

TLS mismatch between nodes
Incompatible security settings

Solution:

# Ensure consistent TLS settings on all nodes
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /path/to/certs/transport.p12
xpack.security.transport.ssl.truststore.path: /path/to/certs/transport.p12

Issue 2: Client Connection Failures

Symptoms: Applications can't connect after upgrade

Causes:

Security enabled (auth required)
TLS enabled (HTTPS required)
API compatibility issues

Solutions:

Update client configuration:

# Python example
from elasticsearch import Elasticsearch

es = Elasticsearch(
    "https://localhost:9200",
    basic_auth=("elastic", "password"),
    ca_certs="/path/to/http_ca.crt"
)

Enable compatibility mode (temporary):

# Allow 7.x API behavior
PUT /_cluster/settings
{
  "persistent": {
    "cluster.deprecation_indexing.enabled": true
  }
}

Issue 3: Index Compatibility

Symptoms: Old indices not accessible

Check index compatibility:

GET /_cat/indices?v&h=index,version.created

Solution: Indices from 6.x must be reindexed:

POST _reindex
{
  "source": {"index": "old-index"},
  "dest": {"index": "new-index"}
}

Issue 4: Plugin Incompatibility

Symptoms: Plugins fail to load

Solution:

Remove old plugins
Install 8.x compatible versions:

bin/elasticsearch-plugin remove old-plugin
bin/elasticsearch-plugin install compatible-plugin

Security Configuration Guide

Minimal Security Setup

# elasticsearch.yml

# Enable security
xpack.security.enabled: true

# HTTP layer TLS
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12

# Transport layer TLS
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: transport.p12
xpack.security.transport.ssl.truststore.path: transport.p12

Generate Certificates

# Create CA
bin/elasticsearch-certutil ca --out elastic-stack-ca.p12

# Create node certificates
bin/elasticsearch-certutil cert \
  --ca elastic-stack-ca.p12 \
  --out elastic-certificates.p12

# Create HTTP certificates
bin/elasticsearch-certutil http

Enrollment Tokens (New in 8.x)

For adding nodes:

# On existing node
bin/elasticsearch-create-enrollment-token -s node

# On new node
bin/elasticsearch --enrollment-token <token>

Migration Checklist

Pre-Upgrade

Review 8.x release notes
Check plugin compatibility
Verify indices created in 7.x (6.x indices need reindex)
Plan security configuration
Test in non-production environment
Update client libraries
Review and update queries using deprecated features

During Upgrade

Follow rolling upgrade procedure
Monitor cluster health at each step
Verify security certificates distributed
Update load balancer/proxy configuration for HTTPS

Post-Upgrade

Verify all nodes joined cluster
Test application connectivity
Check for deprecation warnings
Update monitoring/alerting
Document new credentials and certificates

Deprecation Warnings

Check for Deprecations

GET /_migration/deprecations

Enable Deprecation Logging

PUT /_cluster/settings
{
  "persistent": {
    "cluster.deprecation_indexing.enabled": true
  }
}

Review in:

GET /.logs-deprecation*/_search