Ton of ServerDisconnectedErrors against HTTPS AWS S3 REST API

[thehesiod]

I've tried both master + the latest official release.

I have a script with 100 worker coroutines in one process which continuously do S3 operations against a shared aiohttp session (mostly copies). And the main process has about ~10 workers doing S3 operations (just listing).

Each process uses its own shared session created as so:

connector = aiohttp.TCPConnector(force_close=False, keepalive_timeout=10)
session = aiohttp.ClientSession(connector=connector)

With this setup I'm getting errors like the following pretty consistently. If I retry the command it succeeds. Any ideas?

2015-11-11T13:44:29.332Z ERROR PID:4352 s3op::bucket(444) Retrying 1/5 on s3 request: https://test-for-amohr.s3.amazonaws.com/weather/prod/backup/legacy/dataNCDCMonthly/201104.csv?uploadId=__gBSTjtp_Dnz7ik2PtFfkBh0EjyPNpwfhFKvQ.JkA3adkIW84Jb2.aDtYYB5gAr68_klUoIGDQVP_uQ.XBBbAl6BC4Fi9qxSn2kIVLNQ_s1nqSil6Xe5RfQYpLl9ogi
Traceback (most recent call last):
  File "/Users/amohr/dev/aiohttp/aiohttp/client.py", line 178, in _request
    yield from resp.start(conn, read_until_eof)
  File "/Users/amohr/dev/aiohttp/aiohttp/client_reqrep.py", line 597, in start
    message = yield from httpstream.read()
  File "/Users/amohr/dev/aiohttp/aiohttp/streams.py", line 578, in read
    result = yield from super().read()
  File "/Users/amohr/dev/aiohttp/aiohttp/streams.py", line 433, in read
    yield from self._waiter
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 386, in __iter__
    yield self  # This tells Task to wait for completion.
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/tasks.py", line 288, in _wakeup
    value = future.result()
  File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
aiohttp.errors.ServerDisconnectedError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/amohr/dev/fbn-aio-s3/aios3/bucket.py", line 436, in _request
    response = yield from self._session.request(verb, url, chunked=chunked, headers=headers, data=payload)
  File "/Users/amohr/dev/aiohttp/aiohttp/client.py", line 478, in __iter__
    resp = yield from self._coro
  File "/Users/amohr/dev/aiohttp/aiohttp/client.py", line 185, in _request
    raise aiohttp.ClientResponseError() from exc
aiohttp.errors.ClientResponseError

[jettify]

What library are you using for s3 calls? or just simple presigned urls?

[thehesiod]

simple presigned. So I think I may have a handle on the underlying issue. I noticed I was getting the same issue in my main thread, and I was erroneously using time.sleep instead of asyncio.sleep, so I think this is tied to keep-alive. I just need to ensure there's enough cycles to serve the keep alives. Sorry for the disruption. Perhaps though this would be good to document.

[thehesiod]

doh, I've modified my other process, and even when there's low cpu usage I still get server disconnects. Re-opening

[asvetlov]

Server may close keep-alive connection on own reasons (inactivity timeout, high resource consumption etc.)
That's totally correct.

I've added an explicit test for described behavior 77bb567.

[fafhrd91]

i think we have right behavior. for this case, aws actually closes connection, so aiohttp raises right exception. i think it is should be aios3 concern how to handle this.

at KeepSafe we have high load application that uses aws extensively, we do not have any problems.

[fafhrd91]

i noticed that was closes connection if you do not read/write any data to s3 connection. so question @thehesiod do you actually consume what you read from s3. aiohttp has flow control logic.

[thehesiod]

as long as its not a 204 I consume the output. Feel free to close, I basically had to add retry logic when I catch these errors. I agree there's not enough information here to proceed. I'll open again if my retry logic starts failing as well.

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.