OneLinersCommand workbench
Guides
Databases & Data / Security / Backup & Recovery / Observability & Monitoring

Secure Elasticsearch with TLS, users, snapshots, and safe heap sizing

Enforce transport mTLS and HTTPS, replace superuser applications with scoped roles, preserve automatic heap sizing unless measurements justify change, and prove snapshots through an isolated restore.

240 min15 stepsHigh-impact changeRevision 1
Save or explore
#elasticsearch#tls#roles#api-keys#snapshot#slm#restore#jvm#heap
Save to collectionCreate a collection in the sidebar first.
0 of 15 steps completed
Goal

Operate a self-managed Elasticsearch 9.4 cluster whose identity, authorization, memory boundary, recovery points, and restore evidence are explicit and monitored.

Supported environments
  • Elasticsearch 9.4.x
  • Linux package installation Current supported distributions
Prerequisites
  • Healthy supported cluster All intended nodes are green, version-aligned, monitored, and have sufficient disk/memory headroom.
  • PKI and DNS Unique valid node certificates, trusted CA chains, stable DNS SANs, offline CA protection, and rotation monitoring exist.
  • Short-lived API-key handling Create separate mode-0600 curl config files for the operator and application API keys through the approved secret manager. Do not place Authorization headers in command arguments, shell history, screenshots, or the tutorial form.
  • Repository A supported repository with consistent semantics, independent access/retention, and capacity is mounted or provisioned.
  • Recovery window A maintenance window, application validation manifest, recovery owner, and configuration backups are approved.
Operating boundary

OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.

Full guide

What you will build

System
  • A self-managed Elasticsearch cluster with authenticated transport mTLS, hostname-validated HTTPS, and least-privilege application access.
  • A monitored snapshot lifecycle and an isolated, behaviorally verified restore drill with explicit heap and filesystem-cache boundaries.
Observable outcome
  • All node and client traffic is encrypted with validated identity; application credentials cannot administer snapshots or unrelated indices.
  • A completed SUCCESS snapshot restores under isolated names, passes deterministic data checks, and is removed without touching production.
  • Heap sizing remains automatic unless measured evidence proves a safe equal-Xms/Xmx override within the node memory budget.

Architecture

How the parts fit together

Three self-managed nodes form one transport-mTLS cluster and expose HTTPS to authenticated clients. Native roles or scoped API keys enforce resource boundaries. Master/data nodes share a protected snapshot repository, SLM creates retained recovery points, and a restore-drill namespace proves data usability without taking over live aliases.

Elasticsearch nodesHold shards, elect cluster management leadership, enforce transport/HTTP security, and execute snapshots/restores.
Transport and HTTP PKISeparate node-membership trust from client endpoint identity and lifecycle.
Application role/API keyLimits client operations to required indices and non-administrative cluster monitoring.
Snapshot repository and SLMStore incremental recovery points under scheduled retention and observable outcomes.
Restore-drill namespaceValidates real recovery without colliding with production index or alias names.
  1. Clients validate HTTPS, authenticate, and are authorized only for the approved index namespace.
  2. Nodes authenticate peers over transport TLS and coordinate shard and cluster state.
  3. SLM writes repository metadata and immutable segment references for available primary shards.
  4. Operators restore a named snapshot under a unique prefix, validate known data, then clean only drill indices.

Assumptions

  • A three-node self-managed Elasticsearch 9.4 cluster on supported Linux packages is already installed and currently healthy.
  • An internal PKI can issue unique transport and HTTP certificates, and private DNS names are stable.
  • A supported shared-filesystem or official cloud repository exists with independent access protection and sufficient capacity.
  • Application owners provide an exact operation matrix, deterministic validation query, expected count, and synthetic known document ID.

Key concepts

Transport TLS
Mutually authenticated encryption between Elasticsearch nodes; required for production multi-node security.
HTTP TLS
Server-authenticated HTTPS for clients and Kibana; it protects credentials, requests, and responses.
Role
A named set of cluster and index privileges assigned to users or used to derive API-key permissions.
Snapshot
A repository-managed logical recovery point composed of cluster metadata and immutable index segments.
SLM
Snapshot Lifecycle Management scheduling, retention, execution history, and statistics.
Filesystem cache
Operating-system page cache used heavily by Lucene; it competes with JVM heap for RAM.

Before you copy

Values used in this guide

{{elasticHost}}

DNS name of the HTTPS endpoint whose certificate is validated.

Example: search.example.net
{{elasticCaFile}}

Trusted HTTP CA bundle.

Example: /etc/ssl/certs/elastic-http-ca.pem
{{elasticAdminCurlConfig}}secret

Mode-0600 curl config containing only the short-lived operator Authorization header; delete it after the maintenance window.

Example: /run/user/1000/elastic-admin.curlrc
{{elasticAppCurlConfig}}secret

Mode-0600 curl config containing only the scoped application Authorization header used for the behavior test.

Example: /run/user/1000/elastic-app.curlrc
{{elasticHttpCertificate}}

HTTP leaf certificate inspected before activation.

Example: /etc/elasticsearch/certs/http.crt
{{elasticTransportKeystore}}

Transport PKCS#12 keystore relative to ES_PATH_CONF.

Example: certs/transport.p12
{{elasticTransportTruststore}}

Transport PKCS#12 truststore relative to ES_PATH_CONF.

Example: certs/transport-ca.p12
{{elasticHttpKeystore}}

HTTP PKCS#12 keystore relative to ES_PATH_CONF.

Example: certs/http.p12
{{elasticHttpTruststore}}

HTTP truststore relative to ES_PATH_CONF.

Example: certs/http-ca.p12
{{elasticAppRole}}

Least-privilege role name.

Example: search_reader
{{elasticIndexPrefix}}

Approved production index/data-stream prefix.

Example: search-prod-
{{elasticRepoPath}}

Identical repository filesystem root on all master/data nodes.

Example: /srv/elasticsearch-snapshots
{{elasticRepoLocation}}

Relative repository directory beneath path.repo.

Example: search-prod
{{elasticRepository}}

Registered snapshot repository name.

Example: prod-fs-repository
{{elasticSlmPolicy}}

SLM policy identifier.

Example: daily-search
{{elasticSnapshotPrefix}}

Prefix used for scheduled snapshot names.

Example: search-prod
{{elasticSnapshot}}

Unique completed snapshot selected for the drill.

Example: prechange-20260728t1130z
{{elasticRestorePrefix}}

Unique prefix prepended to every restored index.

Example: restore-drill-
{{elasticValidationIndex}}

Source index name expected in the selected snapshot.

Example: search-prod-000042
{{elasticKnownId}}

Non-sensitive synthetic document ID captured in the snapshot manifest.

Example: synthetic-20260728

Security and production boundaries

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Stop before continuing if

  • Stop before a disruptive test when monitoring, a named incident commander, a current backup, or a tested recovery channel is missing.
  • Stop when nodes disagree about identity, time, version, topology, or trust anchors; do not use force, trust bypasses, or quorum overrides to conceal the inconsistency.
  • Stop when the application does not implement the retry, idempotency, timeout, and connection-discovery behavior required by the tested database.
  • Stop after any unexplained data divergence, failed integrity check, authentication bypass, certificate validation failure, or loss of a majority.
01

verification

Inventory the cluster and record a clean baseline

read-only

Query the supported HTTPS endpoint with a short-lived administrative API key and record versions, node roles, master identity, shard health, and allocation before changing security or recovery settings.

Why this step matters

Security, snapshot, and heap changes should never begin from an unknown or degraded cluster. The exact version and role layout determine supported settings and safe rolling behavior.

What to understand

Capture `_cat/nodes`, `_cat/indices`, pending tasks, allocation settings, and disk watermarks in the private change record.

Use an API key limited to the administrative calls needed for the maintenance window and invalidate it afterward.

System changes

  • Read-only HTTPS requests; the baseline changes no cluster or index state.

Syntax explained

--cacert
Pins trust to the approved CA while retaining hostname validation.
--fail --show-error
Makes HTTP errors fail the command while preserving a useful diagnostic message.
Command
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/'; curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/_cluster/health?pretty'
Example output / evidence
{ "name": "es-a", "cluster_name": "search-prod", "version": { "number": "9.4.2" }, "tagline": "You Know, for Search" }
{ "cluster_name": "search-prod", "status": "green", "number_of_nodes": 3, "number_of_data_nodes": 3, "active_primary_shards": 18, "unassigned_shards": 0 }

Checkpoint: Checkpoint: Inventory the cluster and record a clean baseline

Continue whenAll intended nodes run the approved supported release, cluster health is green, and no shards are unassigned.

Stop whenThe cluster is yellow/red, versions or node roles differ unexpectedly, or a snapshot/restore/rebalance is already active.

If this step fails

Cluster health remains yellow or red after a rolling restart.

Likely causeA node did not rejoin, shard allocation is blocked, disk watermarks are exceeded, TLS identity differs, or allocation exclusions remain.

Safe checks
  • GET /_cluster/health?pretty
  • GET /_cat/nodes?v
  • GET /_cluster/allocation/explain

ResolutionStop the rollout, restore the missing node/configuration, clear only understood temporary exclusions, and wait for shard recovery before touching another node.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Rehearse this step on a disposable environment with production-shaped data and topology before changing production.

Stop conditions

  • Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
  • Stop when the independent recovery path is unavailable or an unrelated production incident is active.
02

verification

Measure heap, filesystem cache, disk, and workload pressure

read-only

Capture JVM maximum/used heap, GC, filesystem capacity, circuit-breaker activity, and rejected tasks on every node before deciding whether any sizing override is justified.

Why this step matters

Elasticsearch normally sizes heap automatically from node roles and memory. Manual sizing without measurements can starve the filesystem cache, lengthen GC pauses, or trigger host OOM termination.

What to understand

Available memory means the container limit when containerized, not physical host RAM.

When overriding, Xms must equal Xmx and the total heap of all nodes on a host must remain at most 50% of available memory; smaller can be better.

System changes

  • Read-only node statistics; no JVM or cluster setting changes.

Syntax explained

filter_path
Limits the response to reviewable capacity and pressure evidence.
heap_max_in_bytes
The active maximum heap, which may come from automatic sizing rather than a custom file.
Command
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/_nodes/stats/jvm,fs,breaker,thread_pool?filter_path=nodes.*.name,nodes.*.jvm.mem,nodes.*.jvm.gc,nodes.*.fs.total,nodes.*.breakers,nodes.*.thread_pool.*.rejected&pretty'
Example output / evidence
{ "nodes": { "a1": { "name": "es-a", "jvm": { "mem": { "heap_used_percent": 31, "heap_max_in_bytes": 8589934592 } }, "fs": { "total": { "available_in_bytes": 751619276800 } } } } }

Checkpoint: Checkpoint: Measure heap, filesystem cache, disk, and workload pressure

Continue whenEvery node has stable heap/GC, disk headroom, no unexplained breaker trips, and no growing rejection counter.

Stop whenAny node is pressured, OOM-killed, above disk watermarks, or materially asymmetric without an approved role-based reason.

If this step fails

Heap pressure, long GC pauses, or OOM kills increase after manual sizing.

Likely causeHeap exceeds the container/host budget, Xms and Xmx differ, filesystem cache was starved, or custom JVM options conflict with automatic sizing.

Safe checks
  • GET /_nodes/stats/jvm?filter_path=nodes.*.jvm.mem,nodes.*.jvm.gc
  • GET /_nodes/_all/jvm?filter_path=nodes.*.jvm.using_compressed_ordinary_object_pointers,nodes.*.jvm.mem.heap_max_in_bytes
  • journalctl -u elasticsearch -n 200 --no-pager

ResolutionReturn to automatic sizing unless measured workload evidence requires an override. If overriding, keep Xms equal to Xmx and no more than half of available node/container memory.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Rehearse this step on a disposable environment with production-shaped data and topology before changing production.

Stop conditions

  • Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
  • Stop when the independent recovery path is unavailable or an unrelated production incident is active.
03

verification

Validate dedicated transport and HTTP certificate identities

read-only

Inspect the planned CA and node certificates before touching configuration. Transport membership and client HTTPS are separate trust boundaries and should use appropriate identities and SANs.

Why this step matters

A certificate must be valid for the exact endpoint clients use and readable by the Elasticsearch service without exposing the private key to unrelated users.

What to understand

Elastic recommends a dedicated CA for transport membership so unrelated certificates cannot authorize nodes to join the cluster.

Protect CA private keys offline; nodes need only their signed keypair and public trust chain.

System changes

  • Reads certificate metadata and chain; does not install or alter keys.

Syntax explained

openssl verify -CAfile
Checks the leaf certificate chain against the explicit trusted CA.
subjectAltName
Lists DNS/IP identities accepted by hostname-verifying clients.
Command
Fill variables0/2 ready

Values stay on this page and are never sent or saved.

openssl verify -CAfile {{elasticCaFile}} {{elasticHttpCertificate}}; openssl x509 -in {{elasticHttpCertificate}} -noout -subject -issuer -dates -ext subjectAltName
Example output / evidence
{{elasticHttpCertificate}}: OK
subject=CN=es-a.example.net
issuer=CN=Elastic HTTP CA
notBefore=Jul 01 00:00:00 2026 GMT
notAfter=Jun 30 23:59:59 2027 GMT
X509v3 Subject Alternative Name: DNS:es-a.example.net,DNS:search.example.net

Checkpoint: Checkpoint: Validate dedicated transport and HTTP certificate identities

Continue whenChain verification is OK, SANs cover approved endpoint names, and lifetime exceeds the monitored rotation window.

Stop whenA key is world-readable, a certificate is expired/near expiry, a SAN is missing, or one CA unexpectedly signs both unrelated trust domains.

If this step fails

HTTPS works only with curl -k or a trust-server-certificate option.

Likely causeThe CA is not trusted, the node certificate lacks the requested DNS SAN, or an intermediary is presenting a different certificate.

Safe checks
  • openssl s_client -connect {{elasticHost}}:9200 -servername {{elasticHost}} -CAfile {{elasticCaFile}} -verify_return_error </dev/null
  • openssl x509 -in {{elasticHttpCertificate}} -noout -subject -issuer -dates -ext subjectAltName

ResolutionCorrect DNS, issue a matching certificate from the approved CA, and distribute the CA. Remove every validation bypass before proceeding.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Rehearse this step on a disposable environment with production-shaped data and topology before changing production.

Stop conditions

  • Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
  • Stop when the independent recovery path is unavailable or an unrelated production incident is active.
04

config

Configure transport mTLS and HTTPS

danger

Place keystores under the Elasticsearch configuration directory, store passwords in the Elasticsearch keystore, and apply TLS settings consistently to every node before the coordinated full restart required when converting an existing plaintext transport cluster.

Why this step matters

Production multi-node clusters require encrypted authenticated transport. HTTPS also protects client credentials, queries, and results and must validate server identity.

What to understand

Nodes configured for TLS transport cannot communicate with plaintext transport nodes; converting an existing cluster requires a coordinated full restart, not a mixed rolling state.

Store secure keystore passwords with `elasticsearch-keystore`, never in elasticsearch.yml. Use unique host certificates for full hostname verification.

System changes

  • Changes every node's transport and HTTP listeners and requires a coordinated restart; malformed settings can prevent cluster formation.

Syntax explained

verification_mode: full
Validates certificate chain and peer hostname rather than only CA membership.
client_authentication: required
Requires peer certificates on the internal transport layer.
File /etc/elasticsearch/elasticsearch.yml
Configuration
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: {{elasticTransportKeystore}}
xpack.security.transport.ssl.truststore.path: {{elasticTransportTruststore}}
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: {{elasticHttpKeystore}}
xpack.security.http.ssl.truststore.path: {{elasticHttpTruststore}}
Example output / evidence
[2026-07-28T11:02:14,322][INFO ][o.e.x.s.t.n.SecurityNetty4Transport] [es-a] publish_address {es-a.example.net:9300}, bound_addresses {[::]:9300}
[2026-07-28T11:02:15,108][INFO ][o.e.h.AbstractHttpServerTransport] [es-a] publish_address {es-a.example.net:9200}, bound_addresses {[::]:9200}

Checkpoint: Checkpoint: Configure transport mTLS and HTTPS

Continue whenAll nodes form the same cluster over TLS, HTTPS answers with the intended certificate, and cluster health returns green.

Stop whenAny node cannot join, plaintext transport remains, hostname verification fails, or the cluster cannot recover all primary shards.

If this step fails

A node refuses to start after transport TLS is enabled.

Likely causeA keystore path, password, certificate chain, file permission, cluster name, or peer-verification setting is wrong; a mixed encrypted/unencrypted transport cluster cannot form.

Safe checks
  • systemctl status elasticsearch --no-pager
  • journalctl -u elasticsearch -n 200 --no-pager
  • sudo -u elasticsearch test -r {{elasticTransportKeystore}}

ResolutionKeep the node out of service, correct certificate and keystore settings, and re-run local validation. Restore the reviewed configuration if the maintenance window cannot safely continue.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • For a new Elasticsearch 8+ installation, retain the automatic security bootstrap and rotate its generated material into the organization's certificate lifecycle instead of disabling security.

Stop conditions

  • Stop before a disruptive test when monitoring, a named incident commander, a current backup, or a tested recovery channel is missing.
  • Stop when nodes disagree about identity, time, version, topology, or trust anchors; do not use force, trust bypasses, or quorum overrides to conceal the inconsistency.
  • Stop when the application does not implement the retry, idempotency, timeout, and connection-discovery behavior required by the tested database.
  • Stop after any unexplained data divergence, failed integrity check, authentication bypass, certificate validation failure, or loss of a majority.
05

verification

Prove HTTPS identity and encryption in the live session

read-only

Connect without insecure flags, inspect the negotiated certificate, then ask Elasticsearch whether the authenticated HTTP session uses TLS.

Why this step matters

Configuration text does not prove the endpoint presents the intended identity or that the operator is using the expected principal.

What to understand

Test every published DNS alias because a certificate may validate on the node name but fail on the application endpoint.

Inspect proxy/load-balancer TLS separately if it terminates HTTPS; end-to-end policy must state where plaintext can exist.

System changes

  • Read-only TLS handshake and authentication introspection.

Syntax explained

_security/_authenticate
Returns the current authenticated subject and realm without exposing the credential.
Command
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

openssl s_client -connect {{elasticHost}}:9200 -servername {{elasticHost}} -CAfile {{elasticCaFile}} -verify_return_error </dev/null 2>/dev/null | grep 'Verify return code'; curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/_security/_authenticate?pretty'
Example output / evidence
Verify return code: 0 (ok)
{ "username": "maintenance-api-key", "authentication_type": "api_key", "authentication_realm": { "type": "_es_api_key" } }

Checkpoint: Checkpoint: Prove HTTPS identity and encryption in the live session

Continue whenCertificate verification returns 0 and the intended short-lived identity is authenticated.

Stop whenThe endpoint works only with -k, the wrong identity appears, or any HTTP endpoint remains unintentionally plaintext.

If this step fails

HTTPS works only with curl -k or a trust-server-certificate option.

Likely causeThe CA is not trusted, the node certificate lacks the requested DNS SAN, or an intermediary is presenting a different certificate.

Safe checks
  • openssl s_client -connect {{elasticHost}}:9200 -servername {{elasticHost}} -CAfile {{elasticCaFile}} -verify_return_error </dev/null
  • openssl x509 -in {{elasticHttpCertificate}} -noout -subject -issuer -dates -ext subjectAltName

ResolutionCorrect DNS, issue a matching certificate from the approved CA, and distribute the CA. Remove every validation bypass before proceeding.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Rehearse this step on a disposable environment with production-shaped data and topology before changing production.

Stop conditions

  • Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
  • Stop when the independent recovery path is unavailable or an unrelated production incident is active.
06

command

Create a least-privilege application role

caution

Define only the cluster and index privileges the application actually needs. Separate read, write, monitoring, snapshot, and administrative duties instead of sharing the elastic superuser.

Why this step matters

Role-based access makes compromise and mistakes bounded. A role should reflect a tested application operation matrix, not a convenient broad wildcard.

What to understand

Use separate roles for ingestion and search when their privileges differ; snapshot administration belongs to an operator, not the application.

Review aliases and data streams because privileges apply to the names clients address and can change meaning as templates evolve.

System changes

  • Creates or replaces one named native Elasticsearch role.

Syntax explained

cluster: monitor
Allows monitoring operations without cluster configuration administration.
read + view_index_metadata
Allows queries and metadata needed by common clients only within the approved index prefix.
Command
Fill variables0/5 ready

Values stay on this page and are never sent or saved.

curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} -H 'Content-Type: application/json' -X PUT 'https://{{elasticHost}}:9200/_security/role/{{elasticAppRole}}' --data '{"cluster":["monitor"],"indices":[{"names":["{{elasticIndexPrefix}}*"],"privileges":["read","view_index_metadata"]}]}'
Example output / evidence
{ "role": { "created": true } }

Checkpoint: Checkpoint: Create a least-privilege application role

Continue whenThe role API confirms creation and its returned definition matches the approved operation/resource matrix.

Stop whenThe application requires superuser, all-index wildcard access, snapshot administration, or permissions that cannot be explained.

If this step fails

The application receives 403 after the new role is applied.

Likely causeThe role omits a required cluster or index privilege, the index pattern is wrong, or the application previously depended on accidental superuser access.

Safe checks
  • Use the authenticate API with the application identity.
  • Use the has-privileges API for the exact required operations.
  • Inspect audit logs without logging credentials or document bodies.

ResolutionAdd only the missing operation and resource scope after application-owner review. Do not grant superuser or broad wildcard privileges as a shortcut.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Rehearse this step on a disposable environment with production-shaped data and topology before changing production.

Stop conditions

  • Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
  • Stop when the independent recovery path is unavailable or an unrelated production incident is active.
07

verification

Assign and behaviorally test the application identity

read-only

Create or update the application identity through the approved secret workflow, assign the narrow role, then prove allowed and denied operations with that identity.

Why this step matters

A role definition is not accepted until the real principal proves required operations and is denied an explicitly forbidden, read-only administrative request.

What to understand

Prefer scoped API keys for services when lifecycle and ownership are clear; store them in the workload secret manager and rotate them.

Test negative permissions using a harmless request whose denial cannot modify production, and verify audit evidence.

System changes

  • The shown tests read one approved index and request read-only security-administration data that the application identity must not be allowed to inspect.

Syntax explained

--config {{elasticAppCurlConfig}}
Reads an Authorization header from a mode-0600 curl config so the API key is not expanded into process arguments.
--write-out %{http_code}
Captures the expected authorization result without printing a verbose error body.
Command
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

curl --config {{elasticAppCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/{{elasticIndexPrefix}}sample/_search?size=0'; curl --config {{elasticAppCurlConfig}} --silent --output /dev/null --write-out '%{http_code}\n' --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/_security/user'
Example output / evidence
{ "took": 3, "timed_out": false, "hits": { "total": { "value": 1240, "relation": "eq" } } }
403

Checkpoint: Checkpoint: Assign and behaviorally test the application identity

Continue whenThe read succeeds and the harmless forbidden administration request returns HTTP 403.

Stop whenRequired reads fail, security-administration data is returned, or the identity is actually a superuser/broad administrator.

If this step fails

The application receives 403 after the new role is applied.

Likely causeThe role omits a required cluster or index privilege, the index pattern is wrong, or the application previously depended on accidental superuser access.

Safe checks
  • Use the authenticate API with the application identity.
  • Use the has-privileges API for the exact required operations.
  • Inspect audit logs without logging credentials or document bodies.

ResolutionAdd only the missing operation and resource scope after application-owner review. Do not grant superuser or broad wildcard privileges as a shortcut.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Rehearse this step on a disposable environment with production-shaped data and topology before changing production.

Stop conditions

  • Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
  • Stop when the independent recovery path is unavailable or an unrelated production incident is active.
08

decision

Keep automatic heap sizing unless measurements justify change

read-only

Inspect the active heap and compressed-oops state. Retain Elastic's automatic role-aware sizing by default; create a jvm.options.d override only from load-test evidence and a complete memory budget.

Why this step matters

Heap competes with the operating-system filesystem cache that Lucene relies on. More heap is not automatically more performance and can increase garbage-collection pauses.

What to understand

If an override is required, put equal Xms/Xmx values in a new `.options` file; never edit the root jvm.options file.

Keep the heap no greater than 50% of memory available to that node/container and below the compressed-oops threshold; account for co-located processes.

System changes

  • The documented default path makes no change. A separately approved override would alter JVM memory at the next node restart.

Syntax explained

heap_max_in_bytes
Reports the effective maximum after automatic sizing and overrides.
compressed ordinary object pointers
A JVM memory-layout optimization that should remain enabled.
Command
Fill variables0/3 ready

Values stay on this page and are never sent or saved.

curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/_nodes/_all/jvm?filter_path=nodes.*.name,nodes.*.jvm.mem.heap_max_in_bytes,nodes.*.jvm.using_compressed_ordinary_object_pointers&pretty'
Example output / evidence
{ "nodes": { "a1": { "name": "es-a", "jvm": { "mem": { "heap_max_in_bytes": 8589934592 }, "using_compressed_ordinary_object_pointers": "true" } } } }

Checkpoint: Checkpoint: Keep automatic heap sizing unless measurements justify change

Continue whenAutomatic sizing fits the measured budget, compressed ordinary object pointers are true, and GC/pressure remain healthy.

Stop whenA custom override has no benchmark evidence, exceeds half of available memory, differs between Xms/Xmx, or disables compressed pointers.

If this step fails

Heap pressure, long GC pauses, or OOM kills increase after manual sizing.

Likely causeHeap exceeds the container/host budget, Xms and Xmx differ, filesystem cache was starved, or custom JVM options conflict with automatic sizing.

Safe checks
  • GET /_nodes/stats/jvm?filter_path=nodes.*.jvm.mem,nodes.*.jvm.gc
  • GET /_nodes/_all/jvm?filter_path=nodes.*.jvm.using_compressed_ordinary_object_pointers,nodes.*.jvm.mem.heap_max_in_bytes
  • journalctl -u elasticsearch -n 200 --no-pager

ResolutionReturn to automatic sizing unless measured workload evidence requires an override. If overriding, keep Xms equal to Xmx and no more than half of available node/container memory.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Rehearse this step on a disposable environment with production-shaped data and topology before changing production.

Stop conditions

  • Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
  • Stop when the independent recovery path is unavailable or an unrelated production incident is active.
09

config

Expose a protected shared snapshot repository

danger

Mount the supported repository at the same path on every master and data node, add its parent to path.repo, and perform a one-node-at-a-time restart only while the cluster is green between nodes.

Why this step matters

Filesystem repositories require consistent path.repo configuration and storage semantics on every master/data node. A path visible on only one node creates partial or unverifiable backups.

What to understand

The Elasticsearch service identity must create, open, rename, list, and lock files, and changes must be immediately visible across nodes.

Only one cluster may have write access to a repository. Register it read-only on any secondary cluster to prevent corruption.

System changes

  • Adds a repository path and requires a rolling restart, temporarily reducing cluster redundancy one node at a time.

Syntax explained

path.repo
Allowlist of filesystem roots from which fs repositories may be registered.
File /etc/elasticsearch/elasticsearch.yml
Configuration
Fill variables0/1 ready

Values stay on this page and are never sent or saved.

path.repo:
  - {{elasticRepoPath}}
Example output / evidence
/srv/elasticsearch-snapshots nfs4 rw,nosuid,nodev,noexec,relatime 0 0
elasticsearch.service active

Checkpoint: Checkpoint: Expose a protected shared snapshot repository

Continue whenAfter each node restart the cluster returns green; the identical mount is readable/writable by Elasticsearch on all required nodes.

Stop whenA node fails to rejoin, shard health is not green, mount semantics differ, or another cluster writes to the same repository.

If this step fails

Repository verification fails on some nodes.

Likely causeThe shared mount is absent, mounted at a different path, not listed in path.repo, not writable by Elasticsearch, or lacks linearizable filesystem semantics.

Safe checks
  • findmnt {{elasticRepoPath}}
  • sudo -u elasticsearch test -r {{elasticRepoPath}} -a -w {{elasticRepoPath}}
  • Call POST /_snapshot/{{elasticRepository}}/_verify.

ResolutionMake the same supported repository visible with correct ownership on every master and data node, then repeat repository verification before taking a snapshot.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Use an official S3, Azure, or GCS repository plugin with workload identity when shared filesystem semantics cannot be guaranteed.

Stop conditions

  • Stop before a disruptive test when monitoring, a named incident commander, a current backup, or a tested recovery channel is missing.
  • Stop when nodes disagree about identity, time, version, topology, or trust anchors; do not use force, trust bypasses, or quorum overrides to conceal the inconsistency.
  • Stop when the application does not implement the retry, idempotency, timeout, and connection-discovery behavior required by the tested database.
  • Stop after any unexplained data divergence, failed integrity check, authentication bypass, certificate validation failure, or loss of a majority.
10

command

Register and verify the repository from every node

caution

Create the fs repository with a relative location beneath path.repo, then execute repository verification and preserve the participating node list.

Why this step matters

Repository registration alone does not prove all nodes can perform the atomic storage operations snapshot metadata requires. Verification turns a configuration claim into cluster evidence.

What to understand

Repository contents are Elasticsearch-managed. Never modify, copy selectively, or clean files behind Elasticsearch's back.

Protect repository access and retention independently so a compromised cluster identity cannot erase every recovery point.

System changes

  • Creates cluster repository metadata and temporary verification artifacts in the repository.

Syntax explained

compress: true
Compresses snapshot metadata, not already-compressed Lucene segment files.
_verify
Checks repository access from relevant nodes and returns their identities.
Command
Fill variables0/5 ready

Values stay on this page and are never sent or saved.

curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} -H 'Content-Type: application/json' -X PUT 'https://{{elasticHost}}:9200/_snapshot/{{elasticRepository}}' --data '{"type":"fs","settings":{"location":"{{elasticRepoLocation}}","compress":true}}'; curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} -X POST 'https://{{elasticHost}}:9200/_snapshot/{{elasticRepository}}/_verify?pretty'
Example output / evidence
{ "acknowledged": true }
{ "nodes": { "a1": { "name": "es-a" }, "b2": { "name": "es-b" }, "c3": { "name": "es-c" } } }

Checkpoint: Checkpoint: Register and verify the repository from every node

Continue whenRepository registration is acknowledged and verification lists every intended master/data node.

Stop whenAny node is absent, verification fails, or repository ownership/retention is not independently protected.

If this step fails

Repository verification fails on some nodes.

Likely causeThe shared mount is absent, mounted at a different path, not listed in path.repo, not writable by Elasticsearch, or lacks linearizable filesystem semantics.

Safe checks
  • findmnt {{elasticRepoPath}}
  • sudo -u elasticsearch test -r {{elasticRepoPath}} -a -w {{elasticRepoPath}}
  • Call POST /_snapshot/{{elasticRepository}}/_verify.

ResolutionMake the same supported repository visible with correct ownership on every master and data node, then repeat repository verification before taking a snapshot.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Rehearse this step on a disposable environment with production-shaped data and topology before changing production.

Stop conditions

  • Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
  • Stop when the independent recovery path is unavailable or an unrelated production incident is active.
11

command

Create a retention-aware SLM policy

caution

Schedule snapshots under unique names, include required indices and cluster state, and set retention that meets RPO while preserving independently protected copies.

Why this step matters

A repository without automated scheduling, retention, and monitoring silently becomes a stale or unbounded archive. Policy must map to a documented RPO and legal retention requirement.

What to understand

SLM retention is a convenience inside the cluster trust boundary; use storage-side immutability or separate credentials for ransomware resilience.

Including global state affects feature state and restore scope. Document exactly which indices and features the service needs.

System changes

  • Creates or replaces one Snapshot Lifecycle Management policy.

Syntax explained

min_count / max_count
Retention guardrails that preserve a floor while bounding repository growth.
include_global_state
Includes persistent cluster settings, templates, and eligible feature states; review restore impact.
Command
Fill variables0/7 ready

Values stay on this page and are never sent or saved.

curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} -H 'Content-Type: application/json' -X PUT 'https://{{elasticHost}}:9200/_slm/policy/{{elasticSlmPolicy}}' --data '{"schedule":"0 30 1 * * ?","name":"<{{elasticSnapshotPrefix}}-{now/d}>","repository":"{{elasticRepository}}","config":{"indices":["{{elasticIndexPrefix}}*"],"include_global_state":true},"retention":{"expire_after":"30d","min_count":7,"max_count":60}}'
Example output / evidence
{ "acknowledged": true }

Checkpoint: Checkpoint: Create a retention-aware SLM policy

Continue whenThe policy is acknowledged, its schedule/indices/retention match RPO and capacity, and monitoring covers failures.

Stop whenThe policy omits required data, includes secrets or unrelated indices unexpectedly, or retention can delete the only valid recovery point.

If this step fails

The SLM policy exists but no successful snapshot appears.

Likely causeThe schedule has not fired, repository access failed, the policy is disabled/mis-scoped, or master scheduling was interrupted.

Safe checks
  • GET /_slm/status
  • GET /_slm/policy/{{elasticSlmPolicy}}
  • GET /_slm/stats

ResolutionExecute the policy manually, inspect the failure reason, repair repository or policy configuration, and require a SUCCESS snapshot before acceptance.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Rehearse this step on a disposable environment with production-shaped data and topology before changing production.

Stop conditions

  • Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
  • Stop when the independent recovery path is unavailable or an unrelated production incident is active.
12

command

Create and inspect a concrete snapshot

caution

Execute SLM or create a uniquely named manual snapshot, wait for completion, and require SUCCESS with zero failed shards before using it as restore evidence.

Why this step matters

A concrete completed snapshot is the input to a restore drill. Repository verification and a scheduled policy do not prove a usable recovery point exists.

What to understand

Snapshots are incremental at the segment level but logically independent restore points; never infer completeness from repository file size.

A snapshot only includes available primary shards. PARTIAL is not equivalent to the approved complete backup.

System changes

  • Writes snapshot metadata and segment references to the repository and consumes repository I/O.

Syntax explained

wait_for_completion=true
Returns final snapshot state for this exercise instead of only an accepted task.
failed: 0
Required evidence that all selected primary shards were included.
Command
Fill variables0/6 ready

Values stay on this page and are never sent or saved.

curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} -H 'Content-Type: application/json' -X PUT 'https://{{elasticHost}}:9200/_snapshot/{{elasticRepository}}/{{elasticSnapshot}}?wait_for_completion=true' --data '{"indices":"{{elasticIndexPrefix}}*","include_global_state":true}'; curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/_snapshot/{{elasticRepository}}/{{elasticSnapshot}}?pretty'
Example output / evidence
{ "snapshot": { "snapshot": "prechange-20260728t1130z", "state": "SUCCESS", "indices": [ "search-prod-000042" ], "shards": { "total": 3, "failed": 0, "successful": 3 } } }

Checkpoint: Checkpoint: Create and inspect a concrete snapshot

Continue whenThe named snapshot state is SUCCESS and successful shards equal total shards with failed=0.

Stop whenState is PARTIAL/FAILED, selected indices are wrong, or repository monitoring reports errors.

If this step fails

A snapshot reports PARTIAL or failed shards.

Likely causeOne or more primary shards were unavailable, repository I/O failed, or the cluster changed while the snapshot was running.

Safe checks
  • GET /_snapshot/{{elasticRepository}}/{{elasticSnapshot}}
  • GET /_cluster/health?level=shards
  • GET /_cat/recovery?v

ResolutionRestore shard health and repository reliability, then create a new uniquely named snapshot. Do not treat a partial snapshot as the approved recovery point.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Rehearse this step on a disposable environment with production-shaped data and topology before changing production.

Stop conditions

  • Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
  • Stop when the independent recovery path is unavailable or an unrelated production incident is active.
13

command

Restore into an isolated namespace

danger

Restore selected production indices under a unique prefix with aliases disabled, wait for recovery, and never overwrite live names during a test.

Why this step matters

Only an actual restore exercises repository readability, snapshot metadata, shard recovery, and operator syntax. Isolated names prevent a rehearsal from replacing production data or aliases.

What to understand

Restore on a dedicated recovery cluster when capacity or security boundaries require it; a same-cluster drill still shares failure domains.

Do not include global state in a routine drill because it can change persistent settings, templates, and feature configuration.

System changes

  • Creates restored indices with the approved unique prefix and consumes cluster storage and recovery bandwidth.

Syntax explained

rename_pattern / rename_replacement
Maps every source index to an isolated test namespace.
include_aliases: false
Prevents restored aliases from redirecting application traffic.
Command
Fill variables0/7 ready

Values stay on this page and are never sent or saved.

curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} -H 'Content-Type: application/json' -X POST 'https://{{elasticHost}}:9200/_snapshot/{{elasticRepository}}/{{elasticSnapshot}}/_restore?wait_for_completion=true' --data '{"indices":"{{elasticIndexPrefix}}*","include_aliases":false,"include_global_state":false,"rename_pattern":"(.+)","rename_replacement":"{{elasticRestorePrefix}}$1"}'
Example output / evidence
{ "snapshot": { "snapshot": "prechange-20260728t1130z", "indices": [ "restore-drill-search-prod-000042" ], "shards": { "total": 3, "failed": 0, "successful": 3 } } }

Checkpoint: Checkpoint: Restore into an isolated namespace

Continue whenAll selected shards restore successfully under only the isolated prefix and cluster health returns green.

Stop whenA target name overlaps live data, disk headroom is insufficient, global state would be restored, or recovery pressures production.

If this step fails

The restored test index is green but application checks fail.

Likely causeThe wrong indices, aliases, templates, feature states, or snapshot generation were restored, or the validation query is not representative.

Safe checks
  • GET /_cat/indices/{{elasticRestorePrefix}}*?v
  • GET /_cat/aliases?v
  • Compare deterministic document counts and sampled IDs with the source manifest.

ResolutionDelete only the isolated restore namespace, correct selection and rename rules, and repeat. Never restore over live names merely to simplify validation.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Restore on a separate recovery cluster registered read-only against the same repository; this is the stronger disaster-recovery proof.

Stop conditions

  • Stop before a disruptive test when monitoring, a named incident commander, a current backup, or a tested recovery channel is missing.
  • Stop when nodes disagree about identity, time, version, topology, or trust anchors; do not use force, trust bypasses, or quorum overrides to conceal the inconsistency.
  • Stop when the application does not implement the retry, idempotency, timeout, and connection-discovery behavior required by the tested database.
  • Stop after any unexplained data divergence, failed integrity check, authentication bypass, certificate validation failure, or loss of a majority.
14

verification

Validate restored documents and application semantics

read-only

Compare deterministic counts and sampled known IDs, run representative queries, and inspect restored shard health. Green shards alone do not prove the right data or aliases were recovered.

Why this step matters

Recovery acceptance is a data and service assertion, not a shard-color assertion. Known documents and representative queries make the restore test deterministic.

What to understand

Use a manifest captured before snapshot with counts and synthetic IDs that contain no sensitive payload.

Measure search correctness and latency on restored data; plugins, analyzers, templates, and application compatibility can fail independently of shard recovery.

System changes

  • Read-only queries against restored isolated indices.

Syntax explained

_count
Provides a deterministic document count for the selected restored index.
filter_path
Limits evidence to identity/existence fields and avoids returning sensitive document content.
Command
Fill variables0/6 ready

Values stay on this page and are never sent or saved.

curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/{{elasticRestorePrefix}}{{elasticValidationIndex}}/_count?pretty'; curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/{{elasticRestorePrefix}}{{elasticValidationIndex}}/_doc/{{elasticKnownId}}?filter_path=_index,_id,found&pretty'
Example output / evidence
{ "count": 1240, "_shards": { "total": 3, "successful": 3, "failed": 0 } }
{ "_index": "restore-drill-search-prod-000042", "_id": "synthetic-20260728", "found": true }

Checkpoint: Checkpoint: Validate restored documents and application semantics

Continue whenCounts match the snapshot manifest, the known ID exists, representative queries pass, and every restored shard is healthy.

Stop whenCounts differ unexpectedly, the known ID is absent, analyzers/templates are incompatible, or queries expose unauthorized data.

If this step fails

The restored test index is green but application checks fail.

Likely causeThe wrong indices, aliases, templates, feature states, or snapshot generation were restored, or the validation query is not representative.

Safe checks
  • GET /_cat/indices/{{elasticRestorePrefix}}*?v
  • GET /_cat/aliases?v
  • Compare deterministic document counts and sampled IDs with the source manifest.

ResolutionDelete only the isolated restore namespace, correct selection and rename rules, and repeat. Never restore over live names merely to simplify validation.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Rehearse this step on a disposable environment with production-shaped data and topology before changing production.

Stop conditions

  • Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
  • Stop when the independent recovery path is unavailable or an unrelated production incident is active.
15

command

Enable actionable monitoring and clean the drill

danger

Require alerts for cluster/shard health, disk watermarks, JVM pressure, rejected work, TLS expiry, authentication failures, and SLM outcomes, then delete only the isolated restored indices after evidence capture.

Why this step matters

The cluster needs signals for the failure modes this guide exposed, and restore artifacts must not consume production capacity indefinitely.

What to understand

Resolve the wildcard to an explicit reviewed index list before deletion and preserve the evidence manifest first.

Monitor the application synthetic query separately from cluster health; green does not mean authorized clients can search correctly.

System changes

  • Deletes only isolated restore-drill indices and reads SLM statistics; does not delete snapshots or repository contents.

Syntax explained

{{elasticRestorePrefix}}*
Scopes cleanup to the unique restore namespace; enumerate exact matches before execution.
Command
Fill variables0/4 ready

Values stay on this page and are never sent or saved.

curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} -X DELETE 'https://{{elasticHost}}:9200/{{elasticRestorePrefix}}*'; curl --config {{elasticAdminCurlConfig}} --fail --silent --show-error --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/_slm/stats?pretty'
Example output / evidence
{ "acknowledged": true }
{ "retention_runs": 1, "retention_failed": 0, "policy_stats": [ { "policy": "daily-search", "snapshots_taken": 1, "snapshots_failed": 0 } ] }

Checkpoint: Checkpoint: Enable actionable monitoring and clean the drill

Continue whenOnly drill indices are removed, production indices remain, SLM reports successful snapshots, and alert tests have owners.

Stop whenThe wildcard matches a production index, evidence is not preserved, or SLM still reports failures.

If this step fails

Monitoring says green while client requests still fail.

Likely causeCluster health does not cover TLS trust, authentication, authorization, application queries, aliases, or downstream latency.

Safe checks
  • Run a least-privilege application synthetic query through the normal endpoint.
  • Check HTTP response classes, search latency, rejected thread-pool tasks, and audit logs.

ResolutionTreat the application synthetic transaction as a separate release gate and repair the failing layer before declaring the cluster healthy.

Security notes

  • Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
  • Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
  • Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
  • Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.

Alternatives

  • Apply an ILM policy with a short retention to isolated drill indices when audit requirements prohibit immediate deletion.

Stop conditions

  • Stop before a disruptive test when monitoring, a named incident commander, a current backup, or a tested recovery channel is missing.
  • Stop when nodes disagree about identity, time, version, topology, or trust anchors; do not use force, trust bypasses, or quorum overrides to conceal the inconsistency.
  • Stop when the application does not implement the retry, idempotency, timeout, and connection-discovery behavior required by the tested database.
  • Stop after any unexplained data divergence, failed integrity check, authentication bypass, certificate validation failure, or loss of a majority.

Finish line

Verification checklist

TLS identityopenssl s_client -connect {{elasticHost}}:9200 -servername {{elasticHost}} -CAfile {{elasticCaFile}} -verify_return_error </dev/null 2>/dev/null | grep 'Verify return code'Verify return code: 0 (ok).
Cluster healthcurl --config {{elasticAdminCurlConfig}} --fail --silent --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/_cluster/health?filter_path=status,number_of_nodes,unassigned_shards'status green, intended node count, unassigned_shards 0.
Snapshot policycurl --config {{elasticAdminCurlConfig}} --fail --silent --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/_slm/policy/{{elasticSlmPolicy}}'The approved schedule, repository, selected indices, and retention.
Restore cleanupcurl --config {{elasticAdminCurlConfig}} --fail --silent --cacert {{elasticCaFile}} 'https://{{elasticHost}}:9200/_cat/indices/{{elasticRestorePrefix}}*?format=json'An empty JSON array after evidence capture and cleanup.

Recovery guidance

Common problems and safe checks

A node refuses to start after transport TLS is enabled.

Likely causeA keystore path, password, certificate chain, file permission, cluster name, or peer-verification setting is wrong; a mixed encrypted/unencrypted transport cluster cannot form.

Safe checks
  • systemctl status elasticsearch --no-pager
  • journalctl -u elasticsearch -n 200 --no-pager
  • sudo -u elasticsearch test -r {{elasticTransportKeystore}}

ResolutionKeep the node out of service, correct certificate and keystore settings, and re-run local validation. Restore the reviewed configuration if the maintenance window cannot safely continue.

HTTPS works only with curl -k or a trust-server-certificate option.

Likely causeThe CA is not trusted, the node certificate lacks the requested DNS SAN, or an intermediary is presenting a different certificate.

Safe checks
  • openssl s_client -connect {{elasticHost}}:9200 -servername {{elasticHost}} -CAfile {{elasticCaFile}} -verify_return_error </dev/null
  • openssl x509 -in {{elasticHttpCertificate}} -noout -subject -issuer -dates -ext subjectAltName

ResolutionCorrect DNS, issue a matching certificate from the approved CA, and distribute the CA. Remove every validation bypass before proceeding.

The application receives 403 after the new role is applied.

Likely causeThe role omits a required cluster or index privilege, the index pattern is wrong, or the application previously depended on accidental superuser access.

Safe checks
  • Use the authenticate API with the application identity.
  • Use the has-privileges API for the exact required operations.
  • Inspect audit logs without logging credentials or document bodies.

ResolutionAdd only the missing operation and resource scope after application-owner review. Do not grant superuser or broad wildcard privileges as a shortcut.

Repository verification fails on some nodes.

Likely causeThe shared mount is absent, mounted at a different path, not listed in path.repo, not writable by Elasticsearch, or lacks linearizable filesystem semantics.

Safe checks
  • findmnt {{elasticRepoPath}}
  • sudo -u elasticsearch test -r {{elasticRepoPath}} -a -w {{elasticRepoPath}}
  • Call POST /_snapshot/{{elasticRepository}}/_verify.

ResolutionMake the same supported repository visible with correct ownership on every master and data node, then repeat repository verification before taking a snapshot.

A snapshot reports PARTIAL or failed shards.

Likely causeOne or more primary shards were unavailable, repository I/O failed, or the cluster changed while the snapshot was running.

Safe checks
  • GET /_snapshot/{{elasticRepository}}/{{elasticSnapshot}}
  • GET /_cluster/health?level=shards
  • GET /_cat/recovery?v

ResolutionRestore shard health and repository reliability, then create a new uniquely named snapshot. Do not treat a partial snapshot as the approved recovery point.

The restored test index is green but application checks fail.

Likely causeThe wrong indices, aliases, templates, feature states, or snapshot generation were restored, or the validation query is not representative.

Safe checks
  • GET /_cat/indices/{{elasticRestorePrefix}}*?v
  • GET /_cat/aliases?v
  • Compare deterministic document counts and sampled IDs with the source manifest.

ResolutionDelete only the isolated restore namespace, correct selection and rename rules, and repeat. Never restore over live names merely to simplify validation.

Heap pressure, long GC pauses, or OOM kills increase after manual sizing.

Likely causeHeap exceeds the container/host budget, Xms and Xmx differ, filesystem cache was starved, or custom JVM options conflict with automatic sizing.

Safe checks
  • GET /_nodes/stats/jvm?filter_path=nodes.*.jvm.mem,nodes.*.jvm.gc
  • GET /_nodes/_all/jvm?filter_path=nodes.*.jvm.using_compressed_ordinary_object_pointers,nodes.*.jvm.mem.heap_max_in_bytes
  • journalctl -u elasticsearch -n 200 --no-pager

ResolutionReturn to automatic sizing unless measured workload evidence requires an override. If overriding, keep Xms equal to Xmx and no more than half of available node/container memory.

Cluster health remains yellow or red after a rolling restart.

Likely causeA node did not rejoin, shard allocation is blocked, disk watermarks are exceeded, TLS identity differs, or allocation exclusions remain.

Safe checks
  • GET /_cluster/health?pretty
  • GET /_cat/nodes?v
  • GET /_cluster/allocation/explain

ResolutionStop the rollout, restore the missing node/configuration, clear only understood temporary exclusions, and wait for shard recovery before touching another node.

The SLM policy exists but no successful snapshot appears.

Likely causeThe schedule has not fired, repository access failed, the policy is disabled/mis-scoped, or master scheduling was interrupted.

Safe checks
  • GET /_slm/status
  • GET /_slm/policy/{{elasticSlmPolicy}}
  • GET /_slm/stats

ResolutionExecute the policy manually, inspect the failure reason, repair repository or policy configuration, and require a SUCCESS snapshot before acceptance.

Monitoring says green while client requests still fail.

Likely causeCluster health does not cover TLS trust, authentication, authorization, application queries, aliases, or downstream latency.

Safe checks
  • Run a least-privilege application synthetic query through the normal endpoint.
  • Check HTTP response classes, search latency, rejected thread-pool tasks, and audit logs.

ResolutionTreat the application synthetic transaction as a separate release gate and repair the failing layer before declaring the cluster healthy.

After the procedure

Alternatives and next steps

Consider these alternatives

  • Elastic Cloud manages transport security and platform snapshots; use its native controls rather than self-managed filesystem instructions.
  • Use an official object-storage repository with workload identity where shared filesystems cannot provide consistent locking and independent retention.
  • Restore onto a separate recovery cluster for stronger isolation and disaster-recovery evidence.

Operate it safely

  • Automate weekly repository verification, restore drills, certificate-expiry checks, and SLM failure alerts.
  • Review application roles and API keys whenever index aliases, data streams, or service ownership change.
  • Load-test representative search/index workloads before any manual heap override and trend GC, cache, breaker, disk, and rejection signals.
  • Protect snapshot storage with independent credentials, retention, and immutability appropriate to the threat model.

Reference

Frequently asked questions

Is a green cluster proof that backups work?

No. Green describes shard allocation. You still need repository verification, a SUCCESS snapshot, an isolated restore, and deterministic data/application checks.

Should we set heap to 50% of RAM?

Fifty percent is an upper bound, not a target. Elastic recommends automatic sizing for most production deployments; smaller heaps can leave more useful filesystem cache.

Can we use curl -k temporarily?

Not as acceptance evidence. It disables certificate validation and can hide the exact identity failure the TLS configuration is meant to prevent.

Recovery

Rollback

Restore reviewed per-node configuration and keystore material if security or repository changes prevent service, one safe boundary at a time. Snapshot data is never manually edited.

  1. Stop the rollout before modifying another node and preserve logs/configuration evidence.
  2. For a failed TLS conversion, restore the previous complete configuration on every node under the coordinated restart plan; never leave mixed transport modes.
  3. For a failed rolling path.repo change, restore the previous node configuration and wait for green health before continuing.
  4. Delete only the newly created role/policy/repository registration when no workload depends on it; keep repository files intact.
  5. Delete only isolated restore-drill indices after enumerating exact names; do not remove the source snapshot until retention policy permits.

Evidence

Sources and review

Verified 2026-07-24Review due 2026-10-22
Current Elasticsearch releaseofficialElastic transport TLS setupofficialElastic self-managed security setupofficialElasticsearch JVM settings and heap guidanceofficialElastic snapshot repository managementofficialElastic shared filesystem repositoryofficialElastic snapshot creation and monitoringofficial