'rabbitmq-upgrade revive' command returns error

[alphamonkey79]

Describe the bug

When attempting a rolling upgrade of RabbitMQ from 3.13.2 to 3.13.6 'rabbitmq-upgrade revive' command errors on 1st node in cluster and upgrade stalls.

rabbitmq-upgrade revive
Will put node rabbit@rabapnph201 back into regular operating mode. The node will again serve client traffic and host primary queue replicas.
Error:
{{:case_clause, {:error, :name_not_registered}}, [{:rabbit_maintenance, :"-revive_local_quorum_queue_replicas/0-lc$^0/1-0-", 1, [file: ~c"rabbit_maintenance.erl", line: 341]}, {:rabbit_maintenance, :revive_local_quorum_queue_replicas, 0, [file: ~c"rabbit_maintenance.erl", line: 349]}, {:rabbit_maintenance, :revive, 0, [file: ~c"rabbit_maintenance.erl", line: 103]}]}

Reproduction steps

1. rabbitmq-upgrade revive
   ...
   Error:
   {{:case_clause, {:error, :name_not_registered}}, [{:rabbit_maintenance, :"-revive_local_quorum_queue_replicas/0-lc$^0/1-0-", 1, [file: ~c"rabbit_maintenance.erl", line: 341]}, {:rabbit_maintenance, :revive_local_quorum_queue_replicas, 0, [file: ~c"rabbit_maintenance.erl", line: 349]}, {:rabbit_maintenance, :revive, 0, [file: ~c"rabbit_maintenance.erl", line: 103]}]}

Expected behavior

We expect the command to not error as with previous successful rolling upgrades.

Additional context

No response

[michaelklishin]

@alphamonkey79 I cannot reproduce and this has not been reported elsewhere (that I know of). We need details about the state of the cluster.

One easy workaround would be to leave out rabbitmq-upgrade drain and rabbitmq-upgrade revive, they are not required for an upgrade.

[michaelklishin]

If you did perform rabbitmq-upgrade drain and the node was restarted, rabbitmq-upgrade revive is not necessary since the maintenance mode state is reset on boot.

If the node was not restarted, restarting it will "revive" it.

[michaelklishin]

Most likely a quorum queue was being deleted concurrently with the "revival" operation. So this list of errors handled simply needs one extra clause added.

The reason why it is so hard to hit is that rabbitmq-upgrade drain will disconnect all clients, so the operation must be performed concurrently on another node.

[michaelklishin]

Another possible scenario is that the QQ has just been created, so its record exists in the schema data store but the process isn't yet running on this node.

[michaelklishin]

#12014 will avoid this exception.

To summarize:

1. rabbitmq-diagnostics revive is only necessary if you run rabbitmq-diagnostics drain without restarting the node, since the node is revived on boot unconditionally
2. You can leave rabbitmq-diagnostics drain out for once, it is not mandateory for a rolling upgrade
3. Based on the little information there is, a quorum queue replica has just been concurrently created or removed. Therefore re-running the command might be enough

[alphamonkey79]

Hello Michael,

This is a 3 node cluster and is currently in a mixed version state.
node 1: now at 3.13.6 after failed rolling upgrade automation (re-running revive command errors)
node 2: remains at 3.13.2 (re-running revive command errors)
node 3: remains at 3.13.2 (re-running revive command DOES NOT ERROR)

This is the 3rd environment / cluster to go from 3.13.2 to 3.13.6 and the first to run into this issue which is strange.

The potential of a quorum queue replica create / remove concurrency makes sense given that we only see the error on 2 out of 3 nodes.

[alphamonkey79]

[twoodcoc_alt@rabapnph201 ~]$ sudo rabbitmqctl cluster_status
Cluster status of node rabbit@rabapnph201 ...
Basics

Cluster name: pyx-np-henrietta
Total CPU cores available cluster-wide: 6

Disk Nodes

rabbit@rabapnph201
rabbit@rabapnph202
rabbit@rabapnph203

Running Nodes

rabbit@rabapnph201
rabbit@rabapnph202
rabbit@rabapnph203

Versions

rabbit@rabapnph201: RabbitMQ 3.13.6 on Erlang 26.2.5.2
rabbit@rabapnph202: RabbitMQ 3.13.2 on Erlang 26.2.5
rabbit@rabapnph203: RabbitMQ 3.13.2 on Erlang 26.2.5

[michaelklishin]

Simply take out the revive command, a node restart will reset maintenance mode state.

[michaelklishin]

I have pushed some Upgrades guide updates to make it clear that revive is not necessary in most cases, and its only purpose is to undo (some) effects of drain rabbitmq/rabbitmq-website@bb21df7.

@kjnilsson has identified an unintentional mistake in what queues do those command attempt to operate on, which can produce this and other unnecessary exceptions. Even though #12014 will handle this and most (if not all) other errors our Raft library can return, a separate fix for that is coming in 3.13.7.

Together with an easy to adopt workaround (a node must be restarted one way or another), this hopefully can be considered addressed.

[alphamonkey79]

Thank you @michaelklishin!
We are proceeding with our upgrade based on this fix / work-a-round.

Tim