SocketIO server refuses to shutdown with Ctrl+C (sometimes)

[lukeyeager]

I sometimes run into problems when I want to shut down Flask-SocketIO. The server intermittently takes several seconds to respond to Ctrl+c. I stumbled across the magic incantation for one application, but now I'm working on a little demo app and am running into the same type of issue I've seen before.

socketio = flask_socketio.SocketIO(app)

@manager.command
def run_socketio():
    socketio.run(app, host='0.0.0.0')

Full source code - https://github.com/lukeyeager/flask-sqlalchemy-socketio-demo/blob/6cc45a2fb7/main.py#L64

Issue with more details - lukeyeager/flask-sqlalchemy-socketio-demo#2

I'm not using Gevent or Eventlet - so I shouldn't need monkey-patching, right?

Thanks for the help!

[miguelgrinberg]

No need to monkey patch.

There is a bug in Werkzeug's handling of the Ctrl-C signal when in multithreaded mode that I've also noticed, but haven't had time to characterize and submit as a bug to that project yet. The bug is present only when using multiple threads, which is a requirement of this extension.

If you install eventlet or gevent you will bypass the issue, as Werkzeug is not used in those cases. I'll leave this bug open as a reminder to investigate the problem.

[lukeyeager]

Thanks for the quick response @miguelgrinberg

I tried it with eventlet and the server doesn't shut down on the first try:

^Cwsgi exiting
^CTraceback (most recent call last):
  File "./main.py", line 68, in <module>
    manager.run()
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/flask_script/__init__.py", line 412, in run
    result = self.handle(sys.argv[0], sys.argv[1:])
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/flask_script/__init__.py", line 383, in handle
    res = handle(*args, **config)
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "./main.py", line 64, in run_socketio
    socketio.run(app, host='0.0.0.0')
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/flask_socketio/__init__.py", line 373, in run
    run_server()
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/flask_socketio/__init__.py", line 368, in run_server
    log_output=log_output, **kwargs)
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/eventlet/wsgi.py", line 842, in server
    pool.waitall()
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/eventlet/greenpool.py", line 120, in waitall
    self.no_coros_running.wait()
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
    return hubs.get_hub().switch()
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run
    self.wait(sleep_time)
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 85, in wait
    presult = self.do_poll(seconds)
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll
    return self.poll.poll(seconds)
KeyboardInterrupt

Tried it with gevent and it seems to work great:

^CKeyboardInterrupt
Traceback (most recent call last):
  File "./main.py", line 68, in <module>
    manager.run()
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/flask_script/__init__.py", line 412, in run
    result = self.handle(sys.argv[0], sys.argv[1:])
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/flask_script/__init__.py", line 383, in handle
    res = handle(*args, **config)
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "./main.py", line 64, in run_socketio
    socketio.run(app, host='0.0.0.0')
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/flask_socketio/__init__.py", line 402, in run
    self.wsgi_server.serve_forever()
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/gevent/baseserver.py", line 284, in serve_forever
    self._stop_event.wait()
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/gevent/event.py", line 77, in wait
    result = self.hub.switch()
  File "/home/ubuntu/code/flask-sqlalchemy-socketio-demo/venv/local/lib/python2.7/site-packages/gevent/hub.py", line 338, in switch
    return greenlet.switch(self)
KeyboardInterrupt

[miguelgrinberg]

@lukeyeager I think I see the eventlet problem in their code. Take a look here: https://github.com/eventlet/eventlet/blob/503c584a7d30d7ced1509df2c0c9262be1157589/eventlet/wsgi.py#L864. They catch the first Ctrl-C, print "wsgi exiting" and then keep going, without allowing the interrupt exception to bubble up the stack. That's why you have to hit Ctrl-C again to really break the app.

I'm not sure why these frameworks appear a bit unpolished in many ways. I also have a lot of issues with clients closing their browsers without properly closing the socket. That also generates all sorts of errors that IMHO these frameworks should suppress.

[lukeyeager]

Two Ctrl+c's isn't the end of the world. This is only for the development server anyway (I'll use gunicorn for production).

But I can't get either gevent or eventlet to send messages properly now. #203.

[lukeyeager]

I resolved #203 with your help - thanks! Monkey patching the std libs didn't change the shutdown behavior in any way (which is probably to be expected).

[robsdedude]

I also came across this issue. What's the state of it? Is there a fix or possible work-around in sight?

[miguelgrinberg]

The workaround is to hit Ctrl-C twice, and if that doesn't work, to kill the web server. Unfortunately this isn't a bug on this side. Handing these signals is tricky, so many of the web servers I support have this problem.

[robsdedude]

M'Key. Hitting Ctrl-C twice is no solution as my server runs inside a docker container and docker stop <container> will send a SIGTERM. It also sends a SIGKILL after a few seconds if nothing happens. But that's no nice solution. Anyway, thanks for the reply.

[miguelgrinberg]

@roba91 if you have a specific use case that is preventing you from doing what you need, I recommend that you take the issue to the web server project you are using. I've seen this problem with eventlet, gevent, Werkzeug and more recently asyncio based servers, so it is widespread.

[robsdedude]

Reading https://github.com/eventlet/eventlet/blob/503c584a7d30d7ced1509df2c0c9262be1157589/eventlet/wsgi.py#L867 it seems that eventlet waits for all green threads to finish. Thus, I assume that Flask-SocketIO or python-socketio or so has some threads that stay alive.

UPDATE:
Not so sure anymore. Using gevent with gevent-websocket is a" fix" in my case...

[miguelgrinberg]

@roba91 the mistake in eventlet's case lies (I think) in assuming that all greenlets are going to exit promply. That works more or less okay for HTTP, but when you have a WebSocket connection you have a greenlet that is not going to end unless it's told to, or the other side closes the connection. If I wanted to properly handle Ctrl-C from my side, I would need to catch the KeyboardInterrupt myself, which isn't trivial, given that the server is controlled by eventlet. Assuming I override eventlet's server to give Flask-SocketIO the option to exit cleanly, then I could signal all running WebSocket handlers to exit, and only then re-raise the exception so that eventlet can do its thing.

I still think there should be better handling from eventlet's side though. At least there should be an official mechanism to give applications the chance to clean up before exiting.

[temoto]

Hello guys.
If you use eventlet, please try this patch.
pip install https://github.com/eventlet/eventlet/archive/84cc764afdf83370ec236163e9ab05f0dad9c6e7.zip

It should close idle HTTP connections promptly.

Please suggest how can I help with websockets. AFAIK there is no idle/busy state for them.

[miguelgrinberg]

@temoto When I implemented websocket support in sanic I had the same problem. The way I solved it, is by keeping track of all the active websocket tasks, and then cancel them as part of the exit signal handler. Here is the specific commit that does this: miguelgrinberg/sanic@fd823c6.

Would the same approach work for eventlet? This is, I think, a good approach, because if the websocket handler function does not care, then the task cancelled exception can be caught by the framework, but when the application needs to do some cleanup, it can catch the exception to do so before exiting.

[temoto]

@miguelgrinberg if I understood correctly, that's raising custom cancel exception in every running task. Thank you for sharing. Seems quite applicable here too.

[miguelgrinberg]

Yes, that's correct. Asyncio will raise CancelledError on the cancelled task. The task can catch it if it wants to do cleanup, or else let it bubble up. Sounds like eventlet does the same.

[miguelgrinberg]

While this is a problem, it is often an issue in the web servers and not in this package. I'm going to close as won't fix for now and then look at individual issues that may be caused in this package if they reappear.