Add three-stage enqueuer/dispatcher scheme to SocketAsyncEngine, ThreadPoolWorkQueue and ThreadPoolTypedWorkItemQueue

[eduardo-vp]

Fixes #93028.

In these scenarios, we currently handle two states: Scheduled and NotScheduled.

This change introduces the "Determining" state, which avoids requesting another resource (thread or TP work item) hastily when more items are enqueued while in this state. Instead, the parallelizer takes care of that. This change ensures that only one thread can be parallelizing at any time.

The transitions are as following:

• From NotScheduled, the only transition is to Scheduled if new items are enqueued (a resource to process them is requested).
• From Scheduled, the only transition is to Determining right before trying to dequeue an item.
• From Determining, it'll transition to NotScheduled if it's determined that there are no more items to process. Otherwise, it'll transition to Scheduled if either new items were enqueued or it's determined that another resource is needed to guarantee all items are processed (in both cases a new resource is requested).

[kouvel]

CC @dotnet/ncl

[wfurt]

what do we expect here? DO we see improvements in any of the existing micro benchmarks?
cc @tmds as well.

[kouvel]

what do we expect here? DO we see improvements in any of the existing micro benchmarks? cc @tmds as well.

The relevant issue is #93028, I believe there were some CPU time improvements in a few scenarios. @eduardo-vp, could you please share some of the data? Also please add "Fixes #93028" to the description so that the issue is linked.

[eduardo-vp]

In the customer-reported scenario exposed here #72153 there was a significant improvement in the CPU usage by implementing these changes. These numbers were collected using a local WSL2 instance.

	Main branch	Changed branch
Throughput (MB/s)	1028.5	1064.3
CPU usage reported by top (%)	260	175

Also some extra data was collected using the ASP.NET perf lab, the scenarios were Plaintext, JSON and Fortunes with different number of connections. Plaintext and JSON didn't show a relevant difference but also there was some less CPU usage for Fortunes as shown below.

[kouvel]

Just a small suggestion, otherwise LGTM, thanks!

[eduardo-vp]

CI errors are known