[Testing] Create Simulations for Caboose Pool Selection

[AmeanAsad]

Description

Currently we rely heavily on our staging environment now to test and iterate on optimizations to Caboose's node selection. Staging is not an ideal environment because:

• We have limited control on the nodes that we talk to due to relying on other components and the properties of those nodes could be changing over time. We should be able to run tests in a controlled environment such that we can isolate the exact scenario we trying to optimize.
• The feedback loop for testing changes in staging is too slow. This slows down development a lot.

Proposed Changes:

The described issue points towards that we should write tests in caboose that allow to run local controlled simulations of the Caboose pool selection process.

We can start with the following changes:

• We expand the CabooseHarness to support more nodes.
• We create mocks for latency, load, reliability, etc. distributions. Those mocks define what the ideal nodes to talk to are.
• A test is run over a fixed period of time with some fixed request load where caboose talks to these nodes and the caboose pool converges on a subset of these nodes.
• Successful tests imply that the final selected caboose pool matches the expected ideal nodes based on the mock distributions provided.