Federation links start in one environment but not the other wish seemingly identical configuration files

[tubatodd]

Describe the bug

After a new install of Rabbitmq on RHEL 9.4, with federated exchanges, the federation links for the associated upstreams do not start. federation status shows "No links"

If I swap the OS for RHEL 8.10, everything works. The only difference between the 2 setups is the OS base image. The rabbit and erlang versions remain the same and using identically configured configuration files. The underlying OS base images are configured the same.

Configuration has been accomplished through ansible automation. So generated files and config are confirmed identical.

Reproduction steps

1. Use RHEL9 minimal base image
2. Install rabbitmq and erlang
3. Configure rabbitmq including exchanges and queues
4. Configure amqps upstreams between federated clusters
5. federation status shows no links
   ...

Expected behavior

federation links started and shown under "federation status"

Additional context

NOT WORKING
OS: RHEL 9.4
Rabbit Version: 3.13.6
Erlang version: 26.2.5.2

WORKING
OS: RHEL 8.10
Rabbit Version: 3.13.6
Erlang version: 26.2.5.2

[michaelklishin]

Please use Discussion for questions. You haven't provided any evidence of an issue. Our team does use issues for discussions.

Nothing in RabbitMQ is RHEL 8 (or RHEL 9)-specific, so something else is different between the environments.

I'm afraid you haven't provided any relevant information to even form a hypothesis. We do not guess in this community.

Exchange federation links are started when

• There is a configured upstream (or a set of upstreams)
• There is a policy that matches some exchanges

And there is one fairly obvious condition for a link to be running:

• Federation can connect to the target upstream

By far the most common reasons for "federation links not starting" is not understanding the above
conditions and how policies work, in particular that only one policy is applicable at a time, and that out of N policies with equal priorities a random one will be selected since RabbitMQ cannot know which one you really intend to take precedence.

In other words, you likely have conflicting policies that match the exchanges you expect to be federated, and in one environment the "correct" one is picked while in the other a different policy applies, for example, one that does not set up federation.

A very common reason for this is policies that use --apply-to all and do not explicitly set a --priority which means that that the priority of 0 will be used.

Federation plugin logs a lot of relevant information, in particular at debug level, such as connection failures. Exchanges will be marked with their effective policy in the management UI.

[michaelklishin]

1. Use RHEL9 minimal base image
2. Install rabbitmq and erlang
3. Configure rabbitmq including exchanges and queues
4. Configure amqps upstreams between federated clusters
5. federation status shows no links

It's not enough to configure upstreams. Some policies must configure federation-related settings and must match some exchanges, or no links will be started because there will be nothing for the plugin to federate.

[tubatodd]

@michaelklishin my apologies for the gross oversimplification. Our setup is 2x 3 node clusters setup with 2 federated upstreams. With the exact same configuration files and exact same erlang and rabbitmq-server package versions we had different results. One the RHEL8 system the federated upstreams created links 100% of the time. When deploying to RHEL9, the federation links fail 100% of the time.

I'm with you. I can't image the operating system having differences enough to cause this issue, but we've eliminated every other factor. We've eliminated selinux and fapolicyd as causes as they are disabled. Firewalld rules are identical. Hypervisor level configuration is identical. Is there anything environment that COULD have an impact on federation links?

Additionally, we engaged rabbitmq support through a paid support group. They too were dumbfounded and could not explain the difference.

[tubatodd]

@michaelklishin the following comment by you provided a resolution.

By far the most common reasons for "federation links not starting" is not understanding the above
conditions and [how policies work](https://www.rabbitmq.com/docs/parameters#policies), in particular that only one policy is applicable at a time, and that out of N policies with equal priorities a random one will be selected since RabbitMQ cannot know which one you really intend to take precedence.

There were 2 policies both with priority 0. For some reason RHEL8 would consistently "randomly" select the policy we wanted while RHEL9 would not. I set the federation policy to priority 1 and immediately the links started.

Thank you for your assistance.

[michaelklishin]

Thank you for reporting back!

I will add a documentation section to the Federation guide next week.

[michaelklishin]

https://www.rabbitmq.com/docs/federation#troubleshooting