Secure RabbitMQ with TLS, users, policies, reliable queues, and monitoring
Build a RabbitMQ 4.3 AMQPS boundary, least-privilege vhost identity, durable quorum-queue path, bounded dead-letter policy, observable client acknowledgements, and measured leader failover.
Operate a messaging cluster whose confidentiality, authorization, backpressure, delivery responsibility, quorum behavior, and failure signals are explicit and behaviorally tested.
- RabbitMQ 4.3.x
- Erlang/OTP RabbitMQ 4.3 supported series
- Healthy three-node cluster Three independent disk nodes run compatible RabbitMQ/Erlang versions with no partitions or alarms.
- PKI and network Stable DNS, server certificates, CA distribution, private listener policy, synchronized time, and key rotation exist.
- Client ownership Publisher and consumer owners can implement confirms, mandatory returns, manual acks, bounded prefetch, and idempotency.
- Change safety Definitions export, recovery access, monitoring, failure budget, named abort owner, and test message namespace are approved.
OneLiners never runs these steps or stores secrets. Review placeholders, versions, current state, and change-control requirements before using a command.
Full guide
What you will build
- A RabbitMQ 4.3 cluster with hostname-validated AMQPS, named least-privilege vhost identities, bounded policies, and official Prometheus monitoring.
- A durable quorum-queue message path using mandatory routing, persistent messages, publisher confirms, manual consumer acknowledgements, dead letters, and a measured leader-failover drill.
- Applications cannot connect over unintended plaintext or administer resources outside their namespace.
- A synthetic message is routed, confirmed, consumed, and acknowledged once; poison messages and queue growth have bounded behavior.
- One queue leader node can stop and rejoin while a majority keeps the queue available and monitoring captures recovery time.
Architecture
How the parts fit together
Three independent RabbitMQ disk nodes host a quorum queue replicated by Raft. Clients validate the AMQPS endpoint and authenticate to a dedicated vhost. Exchanges and bindings route persistent mandatory publishes to the quorum queue; confirms transfer broker responsibility and manual acknowledgements transfer consumer completion. Policies bound storage and dead-letter poison messages. Prometheus exposes health to a private collector.
- The TLS-validating publisher authenticates to one vhost and publishes a persistent mandatory message.
- The exchange routes to a durable quorum queue; a majority accepts it before the publisher confirm.
- A consumer receives within bounded prefetch, processes idempotently, and manually acknowledges success.
- Failures requeue unacked deliveries or dead-letter poison messages under bounded policy; monitoring exposes backlog and resource pressure.
Assumptions
- Three RabbitMQ 4.3 disk nodes already form a healthy cluster on independent supported Linux hosts.
- Stable private DNS, synchronized time, PKI, secret management, firewall policy, monitoring, and out-of-band host access exist.
- Application owners can change client settings and prove mandatory returns, publisher confirms, manual acknowledgements, and idempotency.
- The selected workload suits quorum queues; very high throughput/large fan-out workloads have been capacity-tested against streams or other alternatives.
Key concepts
- Virtual host
- A logical RabbitMQ namespace and first authorization boundary for exchanges, queues, bindings, and permissions.
- Quorum queue
- A durable replicated queue using Raft; confirmed messages remain safe while a majority of members survives.
- Publisher confirm
- Broker acknowledgement that covers publisher-to-queue responsibility, independent of consumer processing.
- Consumer acknowledgement
- Consumer signal that processing completed and the delivery may be removed.
- Prefetch
- Bound on unacknowledged deliveries in flight to a consumer/channel.
- Resource alarm
- Memory or disk safety threshold that can block publishers across the cluster.
Before you copy
Values used in this guide
{{rabbitA}}First RabbitMQ node DNS name.
Example: mq-a.example.net{{rabbitB}}Second RabbitMQ node DNS name.
Example: mq-b.example.net{{rabbitC}}Third RabbitMQ node DNS name.
Example: mq-c.example.net{{rabbitHost}}AMQPS service DNS name validated by clients.
Example: rabbit.example.net{{rabbitCaFile}}Trusted messaging CA bundle.
Example: /etc/rabbitmq/tls/ca.pem{{rabbitServerCertificate}}RabbitMQ server leaf certificate.
Example: /etc/rabbitmq/tls/server.pem{{rabbitServerKey}}secretRestricted server private key.
Example: /etc/rabbitmq/tls/server-key.pem{{rabbitVhost}}Dedicated application vhost.
Example: orders-prod{{rabbitUser}}Dedicated application principal.
Example: orders-app{{rabbitResourcePrefix}}Regex-safe naming prefix controlled by the application.
Example: orders\.{{rabbitExchange}}Durable application exchange.
Example: orders.events{{rabbitDlx}}Dead-letter exchange.
Example: orders.dlx{{rabbitQueue}}Primary quorum queue.
Example: orders.events{{rabbitDlq}}Dead-letter quorum queue.
Example: orders.dead{{rabbitRoutingKey}}Application routing key.
Example: orders.created{{rabbitDeadLetterKey}}Dead-letter routing key.
Example: orders.dead{{rabbitExerciseId}}Unique synthetic message and exercise identifier.
Example: rmq-20260728-049{{rabbitDefinitionsFile}}Protected definitions export path.
Example: /srv/secure-backups/rabbit-definitions-20260728.jsonSecurity and production boundaries
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Stop before continuing if
- Stop before a disruptive test when monitoring, a named incident commander, a current backup, or a tested recovery channel is missing.
- Stop when nodes disagree about identity, time, version, topology, or trust anchors; do not use force, trust bypasses, or quorum overrides to conceal the inconsistency.
- Stop when the application does not implement the retry, idempotency, timeout, and connection-discovery behavior required by the tested database.
- Stop after any unexplained data divergence, failed integrity check, authentication bypass, certificate validation failure, or loss of a majority.
verification
Inventory versions, cluster membership, listeners, and alarms
Record the exact RabbitMQ/Erlang release, nodes, partitions, feature flags, listeners, and resource alarms before changing security or topology.
Why this step matters
A secure queue policy cannot be evaluated on an unknown or degraded cluster. Erlang/OTP and RabbitMQ versions also determine TLS and feature behavior.
What to understand
Require three independent disk nodes for a three-member quorum queue; cluster membership alone does not prove queue replica placement.
Record active feature flags and complete any supported upgrade sequence before introducing new 4.3 behavior.
System changes
- Read-only diagnostics; no users, queues, policies, or listeners are changed.
Syntax explained
cluster_status- Shows node membership, running nodes, partitions, and versions relevant to quorum.
alarms- Reports memory and disk alarms that can block publishers cluster-wide.
sudo rabbitmq-diagnostics server_version; sudo rabbitmq-diagnostics cluster_status; sudo rabbitmq-diagnostics listeners; sudo rabbitmq-diagnostics alarmsAsking node rabbit@mq-a for its RabbitMQ version... 4.3.4 Cluster name: rabbit@mq-a Disk Nodes rabbit@mq-a rabbit@mq-b rabbit@mq-c Interface: [::], port: 5671, protocol: amqp/ssl Node rabbit@mq-a reported no alarms
Checkpoint: Checkpoint: Inventory versions, cluster membership, listeners, and alarms
Continue whenThree intended nodes run RabbitMQ 4.3, have no partition/alarm, and expose only approved listeners.
Stop whenVersions differ unexpectedly, any node is absent, a partition/alarm exists, or plaintext AMQP is exposed contrary to policy.
If this step fails
Publishers are blocked although the application and network look healthy.
Likely causeA memory or disk alarm on any cluster node has propagated a cluster-wide publisher block.
rabbitmq-diagnostics alarmsrabbitmq-diagnostics memory_breakdownrabbitmq-diagnostics check_if_node_is_quorum_critical
ResolutionReduce pressure safely by restoring consumers/capacity or adding disk, not by disabling alarms. Confirm all cluster-wide alarms clear before resuming load.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
verification
Prove host identity, time, storage, and failure domains
Resolve every node name from every peer, verify synchronized time and disk headroom, and confirm quorum replicas do not share a single host, zone, or storage failure domain.
Why this step matters
Quorum queues depend on timely consensus and durable disk. A node can be healthy while stopping it would make one or more queues lose majority.
What to understand
Use the quorum-critical diagnostic immediately before maintenance; cluster node count alone cannot describe every queue's replica set.
Measure disk latency and free space on the RabbitMQ data filesystem, not merely the root mount.
System changes
- Read-only host and RabbitMQ diagnostics.
Syntax explained
check_if_node_is_quorum_critical- Fails when stopping the local node would make a quorum queue/stream unavailable.
Values stay on this page and are never sent or saved.
getent hosts {{rabbitA}} {{rabbitB}} {{rabbitC}}; timedatectl show -p NTPSynchronized --value; df -h /var/lib/rabbitmq; sudo rabbitmq-diagnostics check_if_node_is_quorum_critical10.30.0.11 mq-a.example.net 10.30.0.12 mq-b.example.net 10.30.0.13 mq-c.example.net yes /dev/nvme0n1p2 200G 54G 146G 28% /var/lib/rabbitmq ok
Checkpoint: Checkpoint: Prove host identity, time, storage, and failure domains
Continue whenNames/time/storage are healthy and the candidate maintenance node is not quorum-critical.
Stop whenA failure domain is shared, time/DNS differs, disk is pressured, or the node is quorum-critical.
If this step fails
A quorum queue becomes unavailable after a node failure.
Likely causeA majority of queue members is unavailable, the queue has an unsuitable even or undersized membership, or placement concentrated replicas in one failure domain.
rabbitmq-queues quorum_status --vhost {{rabbitVhost}} {{rabbitQueue}}rabbitmq-diagnostics cluster_statusrabbitmqctl list_queues -p {{rabbitVhost}} name type state online
ResolutionRestore enough original members to regain a majority. Do not use destructive force operations until recovery options and acknowledged-message loss have been assessed.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
verification
Validate the server certificate and CA
Inspect chain, lifetime, key usage, and SANs for the exact AMQPS endpoint before deploying it. Use a production CA; self-signed development material is not a production trust plan.
Why this step matters
Clients must verify the identity they requested, not only encrypt traffic. Correct EKU and SANs avoid ambiguous or over-broad certificates.
What to understand
Private keys must be readable by the RabbitMQ service and no broader; CA signing keys remain offline.
If mTLS authenticates clients, issue client certificates with client-auth EKU and define rotation/revocation separately.
System changes
- Read-only certificate validation.
Syntax explained
extendedKeyUsage- Constrain server and client certificates to their intended authentication purpose.
subjectAltName- Carries DNS identities verified by clients.
Values stay on this page and are never sent or saved.
openssl verify -CAfile {{rabbitCaFile}} {{rabbitServerCertificate}}; openssl x509 -in {{rabbitServerCertificate}} -noout -subject -issuer -dates -ext subjectAltName -ext extendedKeyUsage{{rabbitServerCertificate}}: OK
subject=CN=mq-a.example.net
issuer=CN=Messaging CA
X509v3 Subject Alternative Name: DNS:mq-a.example.net,DNS:rabbit.example.net
X509v3 Extended Key Usage: TLS Web Server AuthenticationCheckpoint: Checkpoint: Validate the server certificate and CA
Continue whenVerification is OK and the certificate has the approved SAN, EKU, issuer, and rotation lifetime.
Stop whenThe key is exposed, SAN/EKU is wrong, chain fails, or expiry monitoring is absent.
If this step fails
A client reports unknown CA, hostname mismatch, or handshake failure.
Likely causeThe client trust store lacks the CA, the server SAN omits the connection name, protocol/cipher policy is incompatible, or mTLS is required without a client certificate.
openssl s_client -connect {{rabbitHost}}:5671 -servername {{rabbitHost}} -CAfile {{rabbitCaFile}} -verify_return_error </dev/nullrabbitmq-diagnostics listenersrabbitmq-diagnostics cipher_suites --format openssl
ResolutionFix the chain, SAN, client certificate, or supported TLS policy. Never solve production trust failures with verification disabled.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
config
Enable AMQPS and retire unintended plaintext
Configure the TLS listener, trust chain, peer-verification policy, and modern protocol versions. Roll one node at a time only when quorum-critical checks permit it.
Why this step matters
AMQPS protects credentials and messages in transit. Disabling unintended plaintext prevents clients from silently falling back to an unencrypted listener.
What to understand
With verify_peer and fail_if_no_peer_cert=false, server identity is protected and presented client certificates are verified, but password clients need not present one.
Set fail_if_no_peer_cert=true only after every client supports mTLS and certificate-to-user authentication is deliberately configured.
System changes
- Replaces the plaintext AMQP listener with TLS on each node and requires RabbitMQ restart/reload according to the deployment method.
Syntax explained
listeners.tcp = none- Disables the default plaintext AMQP listener.
verify_peer- Requests and verifies peer certificate chains according to the configured CA.
/etc/rabbitmq/rabbitmq.confValues stay on this page and are never sent or saved.
listeners.tcp = none
listeners.ssl.default = 5671
ssl_options.cacertfile = {{rabbitCaFile}}
ssl_options.certfile = {{rabbitServerCertificate}}
ssl_options.keyfile = {{rabbitServerKey}}
ssl_options.verify = verify_peer
ssl_options.fail_if_no_peer_cert = false
ssl_options.versions.1 = tlsv1.3
ssl_options.versions.2 = tlsv1.2Interface: [::], port: 5671, protocol: amqp/ssl, purpose: AMQP 0-9-1 and AMQP 1.0 over TLS TLS Library: OpenSSL - OpenSSL 3.0.13
Checkpoint: Checkpoint: Enable AMQPS and retire unintended plaintext
Continue whenEvery node advertises only approved AMQPS and management/metrics listeners, and clients validate the hostname and CA.
Stop whenA quorum-critical node would restart, any client lacks AMQPS support, or validation works only when disabled.
If this step fails
RabbitMQ refuses to start after TLS configuration changes.
Likely causeThe certificate/key/CA path, PEM encoding, private-key permission, Erlang TLS setting, listener collision, or configuration syntax is invalid.
rabbitmq-diagnostics check_runningrabbitmq-diagnostics log_tail --number 200openssl x509 -in {{rabbitServerCertificate}} -noout -subject -issuer -dates
ResolutionRestore the reviewed configuration, correct the certificate material offline, and validate on one non-production node before repeating the rolling change.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Terminate TLS at an approved proxy only when the plaintext backend network is an explicitly accepted trust boundary and client identity semantics are preserved.
Stop conditions
- Stop before a disruptive test when monitoring, a named incident commander, a current backup, or a tested recovery channel is missing.
- Stop when nodes disagree about identity, time, version, topology, or trust anchors; do not use force, trust bypasses, or quorum overrides to conceal the inconsistency.
- Stop when the application does not implement the retry, idempotency, timeout, and connection-discovery behavior required by the tested database.
- Stop after any unexplained data divergence, failed integrity check, authentication bypass, certificate validation failure, or loss of a majority.
verification
Prove the live AMQPS identity
Perform a hostname-validating handshake against every advertised endpoint, then use RabbitMQ diagnostics to confirm the intended listener.
Why this step matters
A configured listener is accepted only when a real client verifies the certificate and negotiates an approved protocol without trust bypasses.
What to understand
Test the service alias and each direct node name used by clients or health checks.
Protocol/cipher output is operational evidence, not a permanent hardcoded cipher recommendation; maintain it with platform crypto policy.
System changes
- Read-only TLS handshake and listener inspection.
Syntax explained
-verify_return_error- Makes chain errors fail instead of being informational.
Values stay on this page and are never sent or saved.
openssl s_client -connect {{rabbitHost}}:5671 -servername {{rabbitHost}} -CAfile {{rabbitCaFile}} -verify_return_error </dev/null 2>/dev/null | grep -E 'Protocol|Cipher|Verify return code'; sudo rabbitmq-diagnostics listenersProtocol : TLSv1.3 Cipher : TLS_AES_256_GCM_SHA384 Verify return code: 0 (ok) Interface: [::], port: 5671, protocol: amqp/ssl
Checkpoint: Checkpoint: Prove the live AMQPS identity
Continue whenTLS 1.2 or 1.3 negotiates, certificate verification is 0, and AMQPS is the only application listener.
Stop whenAny endpoint has a hostname/chain error, weak protocol, or unintended plaintext listener.
If this step fails
A client reports unknown CA, hostname mismatch, or handshake failure.
Likely causeThe client trust store lacks the CA, the server SAN omits the connection name, protocol/cipher policy is incompatible, or mTLS is required without a client certificate.
openssl s_client -connect {{rabbitHost}}:5671 -servername {{rabbitHost}} -CAfile {{rabbitCaFile}} -verify_return_error </dev/nullrabbitmq-diagnostics listenersrabbitmq-diagnostics cipher_suites --format openssl
ResolutionFix the chain, SAN, client certificate, or supported TLS policy. Never solve production trust failures with verification disabled.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
command
Create a dedicated vhost, user, and scoped permissions
Separate the application into its own vhost, create a named non-administrator user through an interactive secret prompt, and grant regexes matching only its exchanges and queues.
Why this step matters
Virtual hosts and permission regexes provide tenancy and resource boundaries. User tags such as administrator govern management UI capabilities and should not be assigned to workloads.
What to understand
Omitting the password argument prompts rather than exposing a secret in process lists and shell history; production automation should use an approved definitions/secret bootstrap.
Map configure, write, and read separately when the application does not create topology at runtime.
System changes
- Creates one vhost, one authenticated user, and a permission tuple scoped by the approved resource prefix.
Syntax explained
configure/write/read- Regexes separately control topology changes, publishing targets, and consumed resources.
-p- Scopes permissions to exactly one virtual host.
Values stay on this page and are never sent or saved.
sudo rabbitmqctl add_vhost {{rabbitVhost}}; sudo rabbitmqctl add_user {{rabbitUser}}; sudo rabbitmqctl set_permissions -p {{rabbitVhost}} {{rabbitUser}} '^{{rabbitResourcePrefix}}.*' '^{{rabbitResourcePrefix}}.*' '^{{rabbitResourcePrefix}}.*'; sudo rabbitmqctl list_user_permissions {{rabbitUser}}Adding vhost "orders-prod" ... Adding user "orders-app" ... Password: Setting permissions for user "orders-app" in vhost "orders-prod" ... vhost configure write read orders-prod ^orders\..* ^orders\..* ^orders\..*
Checkpoint: Checkpoint: Create a dedicated vhost, user, and scoped permissions
Continue whenThe user has only the approved vhost permission row and no administrator/monitoring tag.
Stop whenA wildcard is broader than the application namespace, secret entry is logged, or the application needs administrator tags.
If this step fails
The application receives ACCESS_REFUSED for its vhost or queue.
Likely causeThe user lacks vhost access, permission regexes do not match the resource, credentials target the wrong vhost, or a topic permission denies routing keys.
rabbitmqctl list_user_permissions {{rabbitUser}}rabbitmqctl list_permissions -p {{rabbitVhost}}rabbitmqctl authenticate_user {{rabbitUser}}
ResolutionCorrect only the missing configure/write/read regex after mapping the application's topology and message flow. Do not grant administrator or `.*` by default.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
decision
Remove production dependence on the guest account
Confirm no client uses guest, then delete or rotate it according to bootstrap policy. Remote guest access is refused by default, but a known default credential should not remain an operational dependency.
Why this step matters
RabbitMQ's guest account is intended for local bootstrap and cannot be the production identity model. Deleting it safely requires evidence that automation and probes no longer depend on it.
What to understand
Create and test named operator break-glass access before deleting guest.
Do not enable remote guest access as a troubleshooting shortcut; diagnose the intended user/vhost/TLS path.
System changes
- The shown commands are read-only; a separately approved `delete_user guest` removes the default user only after dependency proof.
Syntax explained
list_connections user- Shows live principal use without revealing credentials.
sudo rabbitmqctl list_users; sudo rabbitmqctl list_connections user peer_host peer_portListing users ... user tags orders-app [] ops-admin [administrator] guest [administrator] Listing connections ... user peer_host peer_port orders-app 10.30.4.25 48122
Checkpoint: Checkpoint: Remove production dependence on the guest account
Continue whenNo live or configured workload uses guest and named operator/application accounts work.
Stop whenAny bootstrap, health check, or emergency procedure still depends on guest.
If this step fails
The application receives ACCESS_REFUSED for its vhost or queue.
Likely causeThe user lacks vhost access, permission regexes do not match the resource, credentials target the wrong vhost, or a topic permission denies routing keys.
rabbitmqctl list_user_permissions {{rabbitUser}}rabbitmqctl list_permissions -p {{rabbitVhost}}rabbitmqctl authenticate_user {{rabbitUser}}
ResolutionCorrect only the missing configure/write/read regex after mapping the application's topology and message flow. Do not grant administrator or `.*` by default.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
command
Declare a durable quorum queue and dead-letter topology
Use the current rabbitmqadmin CLI to declare durable exchange, dead-letter exchange/queue, binding, and an explicitly quorum-typed application queue.
Why this step matters
Quorum queues are the default replicated data-safety choice for durable workloads in RabbitMQ 4.3. Durable topology plus persistent messages and confirms are all required; no one setting supplies end-to-end safety.
What to understand
Use an odd initial quorum group, typically three. Even membership adds no majority benefit, and very large groups reduce throughput.
Declare queue type in application/IaC; queue type is immutable and should not be imposed accidentally by a broad policy.
System changes
- Creates durable exchanges, two quorum queues, and a routing binding in the application vhost.
Syntax explained
--type quorum- Creates a replicated Raft-backed queue whose confirmed messages survive while a majority remains.
--durable true- Persists topology across broker restarts; publishers must still mark messages persistent.
Values stay on this page and are never sent or saved.
rabbitmqadmin --vhost {{rabbitVhost}} exchanges declare --name {{rabbitExchange}} --type direct --durable true; rabbitmqadmin --vhost {{rabbitVhost}} exchanges declare --name {{rabbitDlx}} --type direct --durable true; rabbitmqadmin --vhost {{rabbitVhost}} queues declare --name {{rabbitDlq}} --type quorum --durable true; rabbitmqadmin --vhost {{rabbitVhost}} queues declare --name {{rabbitQueue}} --type quorum --durable true; rabbitmqadmin --vhost {{rabbitVhost}} bindings declare --source {{rabbitExchange}} --destination-type queue --destination {{rabbitQueue}} --routing-key {{rabbitRoutingKey}}exchange declared exchange declared queue declared queue declared binding declared
Checkpoint: Checkpoint: Declare a durable quorum queue and dead-letter topology
Continue whenThe application and dead-letter queues report type quorum, durable true, and the exact binding exists.
Stop whenReplica placement lacks three eligible nodes, names collide with existing classic queues, or the routing key is unreviewed.
If this step fails
A quorum queue becomes unavailable after a node failure.
Likely causeA majority of queue members is unavailable, the queue has an unsuitable even or undersized membership, or placement concentrated replicas in one failure domain.
rabbitmq-queues quorum_status --vhost {{rabbitVhost}} {{rabbitQueue}}rabbitmq-diagnostics cluster_statusrabbitmqctl list_queues -p {{rabbitVhost}} name type state online
ResolutionRestore enough original members to regain a majority. Do not use destructive force operations until recovery options and acknowledged-message loss have been assessed.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
command
Apply bounded queue and dead-letter policy
Apply a narrowly matched policy that caps queue bytes, rejects overflow to the publisher, routes dead letters, and limits poison-message delivery for the selected quorum queue namespace.
Why this step matters
Bounded queues prevent an unavailable consumer from exhausting disk. Rejecting publishes creates explicit backpressure, while dead-lettering stops infinite poison-message requeue loops.
What to understand
Policies are easier to change than application x-arguments, but precedence and matching patterns must be reviewed to avoid silently overriding another policy.
The dead-letter exchange requires a binding to the DLQ and least-privilege write permission at declaration time.
System changes
- Creates or replaces one vhost-scoped policy affecting matching quorum queues.
Syntax explained
overflow: reject-publish- Rejects new publishes when the byte cap is reached so publishers can observe pressure.
delivery-limit: 20- Dead-letters quorum-queue messages after bounded unsuccessful deliveries.
Values stay on this page and are never sent or saved.
sudo rabbitmqctl set_policy -p {{rabbitVhost}} quorum-safety '^{{rabbitResourcePrefix}}.*' '{"max-length-bytes":10737418240,"overflow":"reject-publish","dead-letter-exchange":"{{rabbitDlx}}","dead-letter-routing-key":"{{rabbitDeadLetterKey}}","delivery-limit":20}' --priority 50 --apply-to quorum_queues; sudo rabbitmqctl list_policies -p {{rabbitVhost}}Setting policy "quorum-safety" for pattern "^orders\..*" to "{...}" with priority "50" for vhost "orders-prod" ...
vhost name pattern apply-to definition priority
orders-prod quorum-safety ^orders\..* quorum_queues {"delivery-limit":20,"max-length-bytes":10737418240,"overflow":"reject-publish"} 50Checkpoint: Checkpoint: Apply bounded queue and dead-letter policy
Continue whenThe policy matches only intended quorum queues, has the approved limits, and the DLX route exists.
Stop whenThe regex matches unrelated queues, another policy overrides it, or the publisher cannot handle rejected confirms.
If this step fails
A message is repeatedly redelivered until it consumes capacity.
Likely causeThe consumer negatively acknowledges with requeue, cannot process a poison message, and no bounded delivery/dead-letter policy terminates the loop.
Inspect redelivered rate and delivery-count headers.rabbitmqctl list_queues -p {{rabbitVhost}} name messages_ready messages_unacknowledged policyInspect the dead-letter queue without dumping sensitive payloads.
ResolutionApply a reviewed delivery limit and dead-letter path, fix consumer idempotency/validation, and replay quarantined messages only after the defect is understood.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
verification
Prove routing, persistence, confirms, and manual acknowledgements
Run the owned application integration probe over AMQPS: publish a persistent uniquely identified message with mandatory routing and publisher confirms, consume it with manual acknowledgement, and verify exactly-once business handling.
Why this step matters
Topology existence is not a delivery guarantee. The production client must use mandatory routing, publisher confirms, persistent messages, manual consumer acknowledgements, and idempotent processing.
What to understand
rabbitmqadmin proves a concrete round trip; the application client test must additionally prove asynchronous confirm and returned-message handlers.
Consumer acknowledgements and publisher confirms solve different halves of delivery and are independent.
System changes
- Publishes and consumes one non-sensitive synthetic persistent message through the application topology.
Syntax explained
delivery_mode: 2- Marks the message persistent; durable queues and confirms are still required.
ack_requeue_false- Acknowledges the fetched probe and removes it instead of returning it to the queue.
Values stay on this page and are never sent or saved.
rabbitmqadmin --vhost {{rabbitVhost}} publish exchange={{rabbitExchange}} routing_key={{rabbitRoutingKey}} properties='{"delivery_mode":2,"message_id":"{{rabbitExerciseId}}"}' payload='{"probe":"{{rabbitExerciseId}}"}'; rabbitmqadmin --vhost {{rabbitVhost}} get queue={{rabbitQueue}} ackmode=ack_requeue_false count=1Message published successfully
routing_key: orders.created
message_count: 0
payload: {"probe":"rmq-20260728-049"}
properties: {"delivery_mode":2,"message_id":"rmq-20260728-049"}Checkpoint: Checkpoint: Prove routing, persistence, confirms, and manual acknowledgements
Continue whenThe exact message ID routes once, the publisher confirms responsibility, and the consumer acknowledges only after successful processing.
Stop whenThe message is unroutable, unconfirmed, duplicated, auto-acked before processing, or requeued in a loop.
If this step fails
Messages are published but silently disappear.
Likely causeThe exchange/routing key has no binding, mandatory publishing is disabled, the queue/exchange is non-durable, or the publisher does not wait for confirms.
rabbitmqctl list_bindings -p {{rabbitVhost}} source_name destination_name routing_keyrabbitmqctl list_queues -p {{rabbitVhost}} name type durable messagesInspect publisher returned-message and confirm metrics.
ResolutionUse durable topology, mandatory publishing with a return handler, and asynchronous publisher confirms. Reconcile unconfirmed messages by an application idempotency key.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
verification
Tune bounded prefetch and observe consumer capacity
Inspect ready/unacknowledged counts, consumer capacity, channels, and prefetch under representative load. Select a bounded prefetch through measurement rather than using zero.
Why this step matters
Unbounded in-flight delivery can exhaust consumers and broker memory. Very low prefetch wastes throughput; the correct value depends on processing latency and resource budget.
What to understand
Manual acknowledgement ensures failed/closed consumers cause unacked deliveries to be requeued, so handlers must be idempotent.
RabbitMQ documents 100–300 as a common throughput range, not a universal default; start lower and measure.
System changes
- Read-only queue/channel metrics; client prefetch changes belong in reviewed application configuration.
Syntax explained
messages_unacknowledged- Deliveries in flight awaiting consumer acknowledgement.
consumer_capacity- Fraction of time the queue can immediately deliver to consumers; interpret with backlog and latency.
Values stay on this page and are never sent or saved.
sudo rabbitmqctl list_queues -p {{rabbitVhost}} name type messages_ready messages_unacknowledged consumers consumer_capacity; sudo rabbitmqctl list_channels connection number messages_unacknowledged prefetch_countname type messages_ready messages_unacknowledged consumers consumer_capacity orders.events quorum 0 24 4 1.0 connection number messages_unacknowledged prefetch_count <rabbit@mq-a.1821.0> 1 6 50
Checkpoint: Checkpoint: Tune bounded prefetch and observe consumer capacity
Continue whenUnacked deliveries remain bounded by consumers × prefetch, backlog drains, and handlers acknowledge/nack every outcome.
Stop whenPrefetch is zero/unbounded, unacked count grows without bound, or redelivery loops appear.
If this step fails
Unacked messages grow and consumer memory rises.
Likely causeConsumers use automatic acknowledgement, unbounded/oversized prefetch, slow processing, or fail to ack/nack on every outcome.
rabbitmqctl list_queues -p {{rabbitVhost}} name messages_ready messages_unacknowledged consumers consumer_capacityrabbitmqctl list_channels connection number messages_unacknowledged prefetch_count
ResolutionUse manual acknowledgements, bounded prefetch tuned to measured processing, idempotent handlers, and a dead-letter/retry strategy that avoids hot requeue loops.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
command
Enable Prometheus metrics on the monitoring boundary
Enable RabbitMQ's Prometheus plugin, verify its listener and scrape output from the approved monitoring network, and alert on resource alarms, quorum health, backlog, unacked messages, confirms, connections, and certificate expiry.
Why this step matters
Broker and queue failures need time-series signals before users report missing or delayed work. The metrics endpoint itself must remain on a trusted collection boundary.
What to understand
Use per-object metrics only where cardinality is controlled; aggregated metrics reduce overhead for large fleets.
Alert on trends and SLO impact, not only absolute queue depth; expected batch queues may legitimately retain messages.
System changes
- Enables the official Prometheus plugin and starts its metrics listener.
Syntax explained
rabbitmq_prometheus- Official low-overhead Prometheus exporter plugin.
127.0.0.1 test- Verifies locally without implying the endpoint should be public.
sudo rabbitmq-plugins enable rabbitmq_prometheus; sudo rabbitmq-diagnostics listeners; curl --fail --silent http://127.0.0.1:15692/metrics | grep -E 'rabbitmq_(alarms|queue_messages_ready|queue_messages_unacked)' | headEnabling plugins on node rabbit@mq-a:
rabbitmq_prometheus
started 1 plugins.
Interface: [::], port: 15692, protocol: http, purpose: Prometheus exporter API
rabbitmq_alarms{node="rabbit@mq-a"} 0
rabbitmq_queue_messages_ready{queue="orders.events",vhost="orders-prod"} 0Checkpoint: Checkpoint: Enable Prometheus metrics on the monitoring boundary
Continue whenThe approved collector can scrape metrics, untrusted networks cannot, and alert rules have owners/runbooks.
Stop whenThe listener is publicly reachable, scrape cardinality overloads the broker, or critical alarms have no response path.
If this step fails
Metrics endpoint is reachable from untrusted networks or exposes excessive detail.
Likely causeThe Prometheus listener is broadly bound, firewall policy is open, or a reverse proxy/auth boundary was assumed but not enforced.
rabbitmq-diagnostics listenersss -lntp | grep 15692Test reachability from an explicitly untrusted segment.
ResolutionBind and firewall the metrics endpoint to the monitoring network, or place it behind the approved authenticated collector boundary.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
command
Export topology definitions and state their limit
Export users, vhosts, permissions, policies, exchanges, queues, and bindings to protected configuration recovery storage. Explicitly record that definitions do not contain queued messages.
Why this step matters
Versioned definitions accelerate topology reconstruction and review, but they are configuration artifacts—not a backup of message bodies or a substitute for application replay.
What to understand
Definitions may contain password hashes and sensitive topology names. Encrypt, restrict, retain, and audit them as secrets.
Test import into an isolated cluster and compare topology without connecting production publishers or consumers.
System changes
- Writes a JSON definitions export to the approved protected path; broker runtime state is unchanged.
Syntax explained
export_definitions- Exports broker metadata/topology, permissions, and policies but not queued messages.
sha256sum- Creates an integrity identifier for the exact exported artifact.
Values stay on this page and are never sent or saved.
sudo rabbitmqctl export_definitions {{rabbitDefinitionsFile}}; sudo chmod 0600 {{rabbitDefinitionsFile}}; sha256sum {{rabbitDefinitionsFile}}Exporting definitions in JSON to a file at "rabbit-definitions-20260728.json" ... 8c6e2ac7d2d675141ce146027e72d1ac51e6bb61ac20d5eab44c0e8b89d866e8 rabbit-definitions-20260728.json
Checkpoint: Checkpoint: Export topology definitions and state their limit
Continue whenThe encrypted/restricted definitions artifact has a checksum and passes an isolated import/topology comparison.
Stop whenThe file is world-readable, stored with source code, assumed to contain messages, or not restore-tested.
If this step fails
Definitions import succeeds but expected messages are absent.
Likely causeDefinitions contain topology and metadata, not queued message bodies; the operator confused configuration recovery with message backup.
rabbitmqctl list_queues -p {{rabbitVhost}} name messages typeCompare definitions checksum and declared topology.
ResolutionRecover message state from the application's source of truth or a tested broker recovery design. Do not claim exported definitions are a message backup.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Rehearse this step on a disposable environment with production-shaped data and topology before changing production.
Stop conditions
- Stop when the observed state differs from the documented prerequisite or concrete expected evidence.
- Stop when the independent recovery path is unavailable or an unrelated production incident is active.
decision
Perform a bounded quorum-queue leader failover
After confirming the target node is not quorum-critical, stop the node that currently leads the synthetic queue, observe leader election, publish/consume a new confirmed probe, then restart and wait for the replica to synchronize.
Why this step matters
A quorum queue is accepted only when a real majority election and client recovery are measured. Cluster health without a message round trip can hide endpoint, retry, and confirm defects.
What to understand
Run only one failure at a time; the three-member queue is temporarily one additional failure from losing majority.
Measure time to new leader, time to confirmed publish, time to consumed/acked probe, and time to full replica synchronization.
System changes
- Stops and restarts RabbitMQ on exactly one approved node, forcing affected quorum queues to elect leaders.
Syntax explained
check_if_node_is_quorum_critical- Prevents planned maintenance when local shutdown would make a quorum resource unavailable.
sudo rabbitmq-diagnostics check_if_node_is_quorum_critical; sudo systemctl stop rabbitmq-server; sleep 10; sudo systemctl start rabbitmq-server; sudo rabbitmq-diagnostics await_startupok Stopping rabbitmq-server: rabbit@mq-a. Starting rabbitmq-server... RabbitMQ is fully booted and running
Checkpoint: Checkpoint: Perform a bounded quorum-queue leader failover
Continue whenThe queue remains available through a new leader, confirmed/acked probe succeeds within SLO, and the restarted member resynchronizes.
Stop whenThe node is quorum-critical, any other node is unhealthy, confirms fail, or the queue loses majority.
If this step fails
A quorum queue becomes unavailable after a node failure.
Likely causeA majority of queue members is unavailable, the queue has an unsuitable even or undersized membership, or placement concentrated replicas in one failure domain.
rabbitmq-queues quorum_status --vhost {{rabbitVhost}} {{rabbitQueue}}rabbitmq-diagnostics cluster_statusrabbitmqctl list_queues -p {{rabbitVhost}} name type state online
ResolutionRestore enough original members to regain a majority. Do not use destructive force operations until recovery options and acknowledged-message loss have been assessed.
Security notes
- Keep administrative endpoints on private networks, authenticate every operator, and record who approved the change window. TLS without identity validation protects confidentiality but not server authenticity.
- Supply passwords and private-key passphrases through an approved secret manager or an interactive prompt. Never place them in shell history, process arguments, tutorial variables, logs, tickets, or screenshots.
- Use a dedicated low-privilege application identity and a separate short-lived administrative identity. A successful superuser test does not prove the application permission boundary.
- Treat replication, snapshots, and backups as different controls. Replication improves availability; only a protected, retained, independently restored copy establishes recoverability.
Alternatives
- Run the same leader-failover test on a production-shaped staging cluster if the live service lacks a safe error budget.
Stop conditions
- Stop before a disruptive test when monitoring, a named incident commander, a current backup, or a tested recovery channel is missing.
- Stop when nodes disagree about identity, time, version, topology, or trust anchors; do not use force, trust bypasses, or quorum overrides to conceal the inconsistency.
- Stop when the application does not implement the retry, idempotency, timeout, and connection-discovery behavior required by the tested database.
- Stop after any unexplained data divergence, failed integrity check, authentication bypass, certificate validation failure, or loss of a majority.
Finish line
Verification checklist
openssl s_client -connect {{rabbitHost}}:5671 -servername {{rabbitHost}} -CAfile {{rabbitCaFile}} -verify_return_error </dev/null 2>/dev/null | grep 'Verify return code'Verify return code: 0 (ok).sudo rabbitmq-diagnostics cluster_status; sudo rabbitmq-diagnostics alarmsAll intended nodes running, no partitions, and no alarms.sudo rabbitmq-queues quorum_status --vhost {{rabbitVhost}} {{rabbitQueue}}One leader, three members, and quorum available.Run the owned AMQPS synthetic publish-confirm/consume-ack transaction using {{rabbitExerciseId}}.One confirmed publish, one processed delivery, one acknowledgement, and no duplicate business effect.Recovery guidance
Common problems and safe checks
RabbitMQ refuses to start after TLS configuration changes.
Likely causeThe certificate/key/CA path, PEM encoding, private-key permission, Erlang TLS setting, listener collision, or configuration syntax is invalid.
rabbitmq-diagnostics check_runningrabbitmq-diagnostics log_tail --number 200openssl x509 -in {{rabbitServerCertificate}} -noout -subject -issuer -dates
ResolutionRestore the reviewed configuration, correct the certificate material offline, and validate on one non-production node before repeating the rolling change.
A client reports unknown CA, hostname mismatch, or handshake failure.
Likely causeThe client trust store lacks the CA, the server SAN omits the connection name, protocol/cipher policy is incompatible, or mTLS is required without a client certificate.
openssl s_client -connect {{rabbitHost}}:5671 -servername {{rabbitHost}} -CAfile {{rabbitCaFile}} -verify_return_error </dev/nullrabbitmq-diagnostics listenersrabbitmq-diagnostics cipher_suites --format openssl
ResolutionFix the chain, SAN, client certificate, or supported TLS policy. Never solve production trust failures with verification disabled.
The application receives ACCESS_REFUSED for its vhost or queue.
Likely causeThe user lacks vhost access, permission regexes do not match the resource, credentials target the wrong vhost, or a topic permission denies routing keys.
rabbitmqctl list_user_permissions {{rabbitUser}}rabbitmqctl list_permissions -p {{rabbitVhost}}rabbitmqctl authenticate_user {{rabbitUser}}
ResolutionCorrect only the missing configure/write/read regex after mapping the application's topology and message flow. Do not grant administrator or `.*` by default.
Messages are published but silently disappear.
Likely causeThe exchange/routing key has no binding, mandatory publishing is disabled, the queue/exchange is non-durable, or the publisher does not wait for confirms.
rabbitmqctl list_bindings -p {{rabbitVhost}} source_name destination_name routing_keyrabbitmqctl list_queues -p {{rabbitVhost}} name type durable messagesInspect publisher returned-message and confirm metrics.
ResolutionUse durable topology, mandatory publishing with a return handler, and asynchronous publisher confirms. Reconcile unconfirmed messages by an application idempotency key.
A quorum queue becomes unavailable after a node failure.
Likely causeA majority of queue members is unavailable, the queue has an unsuitable even or undersized membership, or placement concentrated replicas in one failure domain.
rabbitmq-queues quorum_status --vhost {{rabbitVhost}} {{rabbitQueue}}rabbitmq-diagnostics cluster_statusrabbitmqctl list_queues -p {{rabbitVhost}} name type state online
ResolutionRestore enough original members to regain a majority. Do not use destructive force operations until recovery options and acknowledged-message loss have been assessed.
Publishers are blocked although the application and network look healthy.
Likely causeA memory or disk alarm on any cluster node has propagated a cluster-wide publisher block.
rabbitmq-diagnostics alarmsrabbitmq-diagnostics memory_breakdownrabbitmq-diagnostics check_if_node_is_quorum_critical
ResolutionReduce pressure safely by restoring consumers/capacity or adding disk, not by disabling alarms. Confirm all cluster-wide alarms clear before resuming load.
Unacked messages grow and consumer memory rises.
Likely causeConsumers use automatic acknowledgement, unbounded/oversized prefetch, slow processing, or fail to ack/nack on every outcome.
rabbitmqctl list_queues -p {{rabbitVhost}} name messages_ready messages_unacknowledged consumers consumer_capacityrabbitmqctl list_channels connection number messages_unacknowledged prefetch_count
ResolutionUse manual acknowledgements, bounded prefetch tuned to measured processing, idempotent handlers, and a dead-letter/retry strategy that avoids hot requeue loops.
A message is repeatedly redelivered until it consumes capacity.
Likely causeThe consumer negatively acknowledges with requeue, cannot process a poison message, and no bounded delivery/dead-letter policy terminates the loop.
Inspect redelivered rate and delivery-count headers.rabbitmqctl list_queues -p {{rabbitVhost}} name messages_ready messages_unacknowledged policyInspect the dead-letter queue without dumping sensitive payloads.
ResolutionApply a reviewed delivery limit and dead-letter path, fix consumer idempotency/validation, and replay quarantined messages only after the defect is understood.
Metrics endpoint is reachable from untrusted networks or exposes excessive detail.
Likely causeThe Prometheus listener is broadly bound, firewall policy is open, or a reverse proxy/auth boundary was assumed but not enforced.
rabbitmq-diagnostics listenersss -lntp | grep 15692Test reachability from an explicitly untrusted segment.
ResolutionBind and firewall the metrics endpoint to the monitoring network, or place it behind the approved authenticated collector boundary.
Definitions import succeeds but expected messages are absent.
Likely causeDefinitions contain topology and metadata, not queued message bodies; the operator confused configuration recovery with message backup.
rabbitmqctl list_queues -p {{rabbitVhost}} name messages typeCompare definitions checksum and declared topology.
ResolutionRecover message state from the application's source of truth or a tested broker recovery design. Do not claim exported definitions are a message backup.
Reference
Frequently asked questions
Do durable queues and persistent messages guarantee no loss?
Not alone. Reliable publishing also requires publisher confirms, a replicated queue such as quorum, and application handling for unconfirmed outcomes.
Are consumer acknowledgements the same as publisher confirms?
No. Confirms cover publisher-to-broker responsibility; consumer acknowledgements cover broker-to-consumer processing.
Does export_definitions back up queued messages?
No. It exports topology and metadata such as users, vhosts, policies, exchanges, queues, and bindings, not message bodies.
Recovery
Rollback
Rollback restores reviewed listener/security configuration one node at a time only while quorum remains. Queue type cannot be changed in place; recover topology deliberately and preserve message evidence.
- Stop further node restarts or policy changes and retain the currently healthy majority.
- Restore the previous rabbitmq.conf and certificate paths on the failed node, start it, and wait for quorum synchronization before another node.
- Restore the prior policy definition if new limits reject valid traffic; keep dead-lettered messages quarantined until reviewed.
- Remove only the newly created test vhost/user/topology when no workload uses them and definitions evidence is preserved.
- Reconcile unconfirmed or redelivered business messages by idempotency key; never purge a queue to make monitoring green.
Evidence