Multiple System.Net.Http.Functional.Tests Timeouts

[steveisok]

First found in https://dev.azure.com/dnceng/public/_build/results?buildId=876982&view=logs&j=bf603221-817a-5f42-894a-9eb35c3c7f97&t=7b6c19b2-2655-5e8f-d26a-63d3381d63ae

Kusto indicates this suite has timed out 8 times today and 15 times in the last 10 days.

Jobs
| project JobId, Name, Source, Type, Build, Started, QueueName , Properties
| where Started >= ago(10d) 
| where Source contains "dotnet/runtime"
| join kind=inner
(WorkItems | project FriendlyName, JobId, Status, Finished - Started, ExitCode, FailCount
| where FriendlyName == "System.Net.Http.Functional.Tests" and Status == "Timeout") on JobId

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[wfurt]

It seems like predominantly conformance tests like

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-master-b77bac7807a8401d8d/System.Net.Http.Functional.Tests/console.dbce9eb1.log?sv=2019-07-07&se=2020-11-24T19%3A19%3A17Z&sr=c&sp=rl&sig=wPx%2BLzcadA%2B%2BS9%2FYAXBehSf65v%2FclxXb7xhNeIWjADM%3D

   System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_DomainJoinedServerUsesKerberos_UseIpAddressAndHostHeader_Success [SKIP]
      Condition(s) not met: "IsDomainJoinedServerAvailable"
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadWriteAsync_Canceled_ThrowsOperationCanceledException', Elapsed: 00:02:15
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadWriteAsync_Canceled_ThrowsOperationCanceledException', Elapsed: 00:04:15
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadWriteAsync_Canceled_ThrowsOperationCanceledException', Elapsed: 00:06:15
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadWriteAsync_Canceled_ThrowsOperationCanceledException', Elapsed: 00:08:15
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadWriteAsync_Canceled_ThrowsOperationCanceledException', Elapsed: 00:10:15
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadWriteAsync_Canceled_ThrowsOperationCanceledException', Elapsed: 00:12:15
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.Http1CloseResponseStreamConformanceTests.ReadWriteAsync_Canceled_ThrowsOperationCanceledException', Elapsed: 00:14:15

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-43535-merge-5f9bf3af761f4246b9/System.Net.Http.Functional.Tests/console.fac75075.log?sv=2019-07-07&se=2020-11-24T15%3A09%3A29Z&sr=c&sp=rl&sig=1VG8jDDnNWTXJRdLWfPnMISSrNtSQ1bJfftStkFNcVo%3D

    System.Net.Http.Functional.Tests.SocketsHttpHandler_HttpClientHandlerTest.SendAsync_RequestVersion20_ResponseVersion20 [SKIP]
      Condition(s) not met: "IsWindows10Version1607OrGreater"
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.HttpClientTest.Send_TimeoutResponseContent_Throws', Elapsed: 00:03:47
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.HttpClientTest.Send_TimeoutResponseContent_Throws', Elapsed: 00:05:47
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.HttpClientTest.Send_TimeoutResponseContent_Throws', Elapsed: 00:07:47
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.HttpClientTest.Send_TimeoutResponseContent_Throws', Elapsed: 00:09:47
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.HttpClientTest.Send_TimeoutResponseContent_Throws', Elapsed: 00:11:47
   System.Net.Http.Functional.Tests: [Long Running Test] 'System.Net.Http.Functional.Tests.HttpClientTest.Send_TimeoutResponseContent_Throws', Elapsed: 00:13:47

cc: @stephentoub @alnikola @antonfirsov

[stephentoub]

It seems like predominantly conformance tests like

I checked in #44342 to hopefully help narrow down the ReadWriteAsync_Canceled_ThrowsOperationCanceledException issue. That could very well be a real product problem.

The other Send_TimeoutResponseContent_Throws one cited isn't in the stream tests. It looks to have one of the same problems I fixed recently in another test: if the client times out so fast that it never initiates a connection to the server, the server side and thus the whole test will hang.

[wfurt]

Thanks for quick fix @stephentoub. I'll keep watching and we discussed to update our CI monitoring reports to see this sooner.

[wfurt]

closing as it did not fail in master for 2 weeks.