Subgraph support for query batching

[garypen]

This project is an extension of the existing work to support client side batching in the router.
The current implementation is experimental and is publicly documented.
The additional work to enable batching requests to subgraphs is captured in this issue.
Currently the concept of a batch is preserved until the end of the RouterRequest processing. At this point we convert each batch request item into a separate SupergraphRequest. These are then planned and executed concurrently within the router and re-assembled into a batch when they complete. It's important to note that, with this implementation, the concept of a batch, from the perspective of an executing router, now disappears and each request is planned and executed separately.
This extension will modify the router so that the concept of a batch is preserved, at least outwardly, so that multiple subgraph requests are "batched" (in exactly the same format as a client batch request) for onward transmission to subgraphs. The goal of this work is to provide an optimisation by reducing the number of round-trips to a subgraph from the router.
Additionally, the work will address an unresolved issue from the existing experimental implementation and promote the existing implementation from experimental to fully supported.

Fixes #2002

---

Review Guidance

This is a fairly big PR, so I've written these notes to help make the review more approachable.

1. The most important files to review are (in order):

• .changesets/feat_garypen_2002_subgraph_batching.md
• docs/source/executing-operations/query-batching.mdx
• apollo-router/src/services/router/service.rs
• apollo-router/src/services/supergraph/service.rs
• apollo-router/src/query_planner/plan.rs
• apollo-router/src/services/subgraph_service.rs
• apollo-router/src/batching.rs

First read the documentation. Hopefully that will make clear how this feature works. I've picked these files as being most important (and ordered them for review) because:

router service => This is where we spot incoming batches and create context BatchQuery items to manage them through the router. We also re-assemble them on the way back and identify any batches which may need to be cancelled.

supergraph service => Here we pick up the information about how many fetches we believe each BatchQuery will need to make.

plan => The new query_hashes() does this fetch identification for us. This is the most important function in this feature.

subgraph service => Here's is where we intercept the calls to subgraphs and park threads to wait for batch execution to be performed. We do a lot of work here, so this is where most of the intrusive changes are: assembling and dis-assembling batches and managing the co-ordination between a number of parked tasks.

batching => This is the implementation of batch co-ordination. Each batch has a task which manages a variety of channels to facilitate communication between the incoming batches, waiting tasks and outgoing (to subgraph) batches. I'm suggesting reading this after reading through the service changes because it should mainly just be implementation details and you will be able to follow what is happening without knowing all this detail initially. Once you understand the changes to the services, you will need to read this code. Feel free to peek ahead though if that's how you like to review stuff.

2. There are still a couple of TODOs which will be resolved early next week. They are both related to how we handle context cloning, so a decision is still pending there.

Obviously all the files need to be reviewed, but the remaining files should be fairly mechanical/straight-forward.

---

Checklist

Complete the checklist (and note appropriate exceptions) before the PR is marked ready-for-review.

• Changes are compatible¹
• Documentation² completed
• Performance impact assessed and acceptable
• Tests added and passing³
  • Unit Tests
  • Integration Tests
  • [ ] Manual Tests

Exceptions

Note any exceptions here

Notes

Footnotes

1. It may be appropriate to bring upcoming changes to the attention of other (impacted) groups. Please endeavour to do this before seeking PR approval. The mechanism for doing this will vary considerably, so use your judgement as to how and when to do this. ↩

2. Configuration is an important part of many changes. Where applicable please try to document configuration examples. ↩

3. Tick whichever testing boxes are applicable. If you are adding Manual Tests, please document the manual testing (extensively) in the Exceptions. ↩

[router-perf]

CI performance tests

• reload - Reload test over a long period of time at a constant rate of users
• events_big_cap_high_rate_callback - Stress test for events with a lot of users, deduplication enabled and high rate event with a big queue capacity using callback mode
• events_without_dedup_callback - Stress test for events with a lot of users and deduplication DISABLED using callback mode
• large-request - Stress test with a 1 MB request payload
• const - Basic stress test that runs with a constant number of users
• no-graphos - Basic stress test, no GraphOS.
• step-jemalloc-tuning - Clone of the basic stress test for jemalloc tuning
• events - Stress test for events with a lot of users and deduplication ENABLED
• events_callback - Stress test for events with a lot of users and deduplication ENABLED in callback mode
• events_big_cap_high_rate - Stress test for events with a lot of users, deduplication enabled and high rate event with a big queue capacity
• events_without_dedup - Stress test for events with a lot of users and deduplication DISABLED
• xxlarge-request - Stress test with 100 MB request payload
• xlarge-request - Stress test with 10 MB request payload
• step - Basic stress test that steps up the number of users over time

[o0Ignition0o]

Couple of nits, nothing that prevents a merge, lgtm.