Iperf3: connection is getting closed from the client side unexpectedly

[Ankita13-code]

NOTE: The iperf3 issue tracker is for registering bugs, enhancement
requests, or submissions of code. It is not a means for asking
questions about building or using iperf3. Those are best directed
towards the Discussions section for this project at
https://github.com/esnet/iperf/discussions
or to the iperf3 mailing list at iperf-dev@googlegroups.com.
A list of frequently-asked questions
regarding iperf3 can be found at http://software.es.net/iperf/faq.html.

Context

iperf3 commands are failing with the following error: the client has unexpectedly closed the connection.

• Version of iperf3: 3.17.1

• Hardware:

• Operating system (and distribution, if any): Azure Linux/CBL-Mariner and SUSE Linux

Please note: iperf3 is supported on Linux, FreeBSD, and macOS.
Support may be provided on a best-effort basis to other UNIX-like
platforms. We cannot provide support for building and/or running
iperf3 on Windows, iOS, or Android.

• Other relevant information (for example, non-default compilers,
  libraries, cross-compiling, etc.):

Please fill out one of the "Bug Report" or "Enhancement Request"
sections, as appropriate. Note that submissions of bug fixes, new
features, etc. should be done as a pull request at
https://github.com/esnet/iperf/pulls

Bug Report

• Expected Behavior
  The lisa test case perf_udp_iperf_sriov uses iperf3 to test the network throughput between 2 Azure Linux VMs. This test should pass for Azure Linux VMs.
• Actual Behavior
  The test is intermittently failing with the following error - error: the client has unexpectedly closed the connection.
• Steps to Reproduce
  -- Spin up 8 iperf servers at the same time in parallel (we are using port 750-757 here) - sudo iperf3 -s -1 -J -i10 -f g -p <port>
  -- Now run 8 clients in parallel with the following parameters - sudo iperf3 -t 10 -c <server_ip> -u -J -i1 -f g -p <port> -P 64 -l 8192 -4

When these commands run for the last port (757 in this case) the error occurs.

• Possible Solution

Enhancement Request

• Current behavior

• Desired behavior

• Implementation notes

[davidBar-On]

The test is intermittently failing with the following error - error: the client has unexpectedly closed the connection

This error means that the client terminated before the normal end of the test, or that for some reason the network interface with the client was disconnected, etc. Therefore, the Client's output messages should be given to try to better understand the cause of the problem.

[Ankita13-code]

Hi @davidBar-On,
here are the logs for both server as well as client - logs.zip

[davidBar-On]

It seems that the problem in is that the server "received" the client's close of the control channel, before the "done" message (IPERF_DONE) sent by the client arrived to the server. Since this is not really an error, I submitted PR #1765 that does not issue an error in this case (only prints a warning message).

@Ankita13-code, can you try building using the PR's code to make sure it solves the problem?

[Ankita13-code]

Hi @davidBar-On , I tried building using the PR's code, but it did not solve the problem. Rather the race condition was hit more frequently in this case. Here are the logs - logs - iperf.zip

[davidBar-On]

Hi @Ankita13-code, I now see that PR #1765 is probably not relevant, as it handles failure at the end of the test, while it seems that the problem is during the connection of the streams at the beginning of the test.

Comparing the cookies of the two failed servers to the cookies sent by the clients, I don't find the relevant clients. You can see that the last client's cookie is uce67qp2fbr7tey366hbbs3mmvum7ncqjb6k, and both the client and server for this test completed successfully. However, the client log does not include the log for the two failed server tests - cookies fj2p5c6ouayhve66zknl24wcw6mwupik7yhg and cfqzkwvp62awbs4cvosxs4exvmckcqumhab4.

Can you try to understand which clients performed the failed tests and send server and client logs for failed test(s)?