Introduce new transports driven by java.nio

[Tim-Brooks]

# Description

We are interested in introducing alternative transport options that do not depend on external libraries. From time to time we have had some friction with using netty related to buffer pooling, oom excetions, exception handling, unsafe, etc. We want to look explore if a homegrown transport could potentially meet our needs.

Where we are now

We have a org.elasticsearch.libs.nio library that provides low-level NIO-based network constructs. Everything is built around the NioSelector. This is a single-threaded event loop that handles the operations for a collection of NioChannel. Specific channel behavior based on read, write, or accept events is determined by configuring a ChannelContext.

We have implemented a transport-nio plugin using these facilities. This plugin provides fully functional binary and http transports. Additionally there is a MockNioTransport built on the nio library in the test framework package. This transport is used for all inter-cluster communication in tests.

Work to be completed

• Byte pooling for NioTransport - we currently reuse bytes for writing. However, we also need to introduce some version for reading. There is an open PR (Implement byte array reusage in nio transport #27051) that includes this work. However, we are still hashing out the details. There is an issue tracking this work (Implement byte array reuse in nio transport #27563).
• Move base nio classes to a library jar. (Introduce elasticsearch nio library jar #27802)
• Move NioTransport to a plugin
• Finish working http implementation (HTTP implementation based on elasticsearch-nio #28898)
• Byte pooling for nio http transport - this can likely reuse some of the work introduced in Implement byte array reuse in nio transport #27563.
• Benchmarking - Add to rally benchmarks.
• Benchmarking - Look into network specific benchmarks
• Merge AcceptingSelector and SocketSelector. There is not a reason why these need to be separate classes / running threads.
• Explore using the same network selectors for processing both http and tcp channels. There is no reason we need dedicated network threads for these different transports. Theoretically we might get better resource utilization by sharing the network threads.
• Experiment with sharing byte pooling. Do we want to use the same page recyclers that we use for various requests? Or do we want to have byte pools dedicated to the network?
  • We will probably need to implement a byte array allocator specific to networking purposes. Our current allocators only support 16KB byte arrays. However, TLS support requires byte arrays at least ~16,900 bytes.
• Experiment with direct ByteBuffers. Using direct byte buffers prevents some copying when reading and writing to the network. We currently have recyclers that should support pooling direct byte buffers. We should look at performance if we use these for reading / writing opposed to on heap byte arrays.
  • Add DirectByteBuffer strategy for transport-nio #36289 implements DirectByteBuffer pooling. However, we still copy data when we do a read or write. Removing this would be beyond the scope of this issue (TLS, compression, etc all operate on heap bytes).
• Add bounds to selector queues. There are currently unbounded queues for channels to be closed, writes to be processed, and channels to be opened. Based on this issue (Improve thread pools #18613) maybe we want to bound these? At minimum we probably want major log warns if these queues get too large.

[original-brownbear]

We stopped this effort #82085