client: Support TSO RPC Parallelizing

[MyonKeminta]

Development Task

• The main tracking issue in TiDB repo: TSO Request Parallelizing pingcap/tidb#54960

We used to find that in some OLTP workloads where the QPS is high and the queries are simple, the TSO Wait duration usually become a significant portion of the total duration of queries. In TiDB, TSO loading is already made concurrent with some other works such as compiling. In cases that the queries are simple, it would be hard to further optimize it by making it concurrent with more phases of the SQL execution. But we found a practical way to optimize it is to do it from the TSO client.

Currently, a TSO client object has a goroutine that collects GetTS (and GetTSAsync) calls (tsoRequests) as a batch, send it to PD, wait for the response, and dispatch the results to these tsoRequests, serially. As a result, each GetTS calls may need to spend up to 1x TSO RPC time to wait for being collected to the next batch.

Considering the case that PD's TSO allocator is not the bottle neck and can deal with more TSO requests (so that the majority part of TSO RPC's time cost is on the network), we find that it's possible to start collecting the next batch and send it before receiving the response of the previous batch. So that each GetTS call needs to wait for less time to be batched, and gets a shorter total duration.

So this is an approach that reduces the duration of GetTS & GetTSAsync - Wait at the expense of higher TSO RPC OPS and higher pressure to PD. It's not suitable to be enabled by default, but we can provide such an option when the TSO Wait duration becomes a problem.

Subtasks

Side changes:

• Merge the two xxxTSOStream types so that the error handling and metrics reporting logic for PD server deployment and TSO service deployment can be reused. client: Merge the two tsoStream types to reuse the same error handling and metrics reporting code #8433
• Split sending and receiving part of tsoStream into separated goroutines
• Let tsoDispatcher support batching according to estimated TSO RPC duration from tsoStream