Support idle connection timeout with pending sockets

[JamesNK]

Fixes #2210

I think the issue is caused by a race when there is a long time interval between a channel connecting and a request being made:

1. When a request is made, the socket is double-checked to be healthy before it's given to SocketsHttpHandler to be turned into a connection.
2. After the health check, the server/proxy closes what it thinks is an idle socket.
3. The handler then uses the closed socket to send a request and fails.

The prospective fix is to close and recreate the socket if it exceeds the idle connection timeout. Default of 1 minute.

[JamesNK]

cc @jhyuzhijian @kalduzov

[jhyuzhijian]

cc @jhyuzhijian @kalduzov

#2210 the issue behavior
.net client request java server no problem
.net client request .net server has problem
so i think close and recreate the socket if it exceeds the idle connection timeout. Default of 1 minute can not resove this question

[JamesNK]

Default of 1 minute can not resove this question

Why not? Kestrel has a default keep-alive timeout of 130 seconds.

https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.server.kestrel.core.kestrelserverlimits.keepalivetimeout?view=aspnetcore-7.0#microsoft-aspnetcore-server-kestrel-core-kestrelserverlimits-keepalivetimeout

[jhyuzhijian]

all right,i will test it

[kalduzov]

We currently have 3 services in testing. On two of them, such an error occurs and they do not actively load the channel. Now about 1 error per hour. The third one is actively working with Grpc and there have been no such errors in the last 24 hours.

This partly confirms what you're thinking @JamesNK .

[mgravell]

nit on naming (this is a "consider", not a "please change"), but: LGTM