Changing rabbit connection timeout

[dtuantran]

@Didayolo
...

Changing the default worker connection timeout of RabbitMQ

Since the version 3.8, RabbitMQ added consumer_timeout as 1800 s by default. It means that if the compute worker doesn't answer RabbitMQ server after 30 minutes, RabbitMQ closes the connection. Therefore, if we have a submission running for more than 30 minutes, RabbitMQ suggests that the compute worker disappeared and distributes this submission to another compute worker. It ends up with all compute worker having submission running and blocking.

Issues this PR resolves

Render this parameter as environment variable and fix it to 100000000 milliseconds, more 27h.

Checklist

• Code review by me
• Hand tested by me
• I'm proud of my work
• Code review by reviewer
• Hand tested by reviewer
• CircleCi tests are passing
• Ready to merge

[Didayolo]

Hi @dtuantran,

I know we basically have this as local changes in the production server, but without the new timeout argument in the .env, and the code looks good to me.

How can this be tested? Have you tested it? What is required when deploying this in production?

[dtuantran]

Hi @dtuantran,

I know we basically have this as local changes in the production server, but without the new timeout argument in the .env, and the code looks good to me.

How can this be tested? Have you tested it? What is required when deploying this in production?

Yes, I tested on my local instance. RabbitMQ has new timeout value when starting. Here is the step to verify:

Connect to the rabbit container

docker compose exec rabbit bash

Execute this command

rabbitmqctl eval 'application:get_env(rabbit, consumer_timeout).'

The result should be as we configure in .env