Logging does not fully work in pytest

[olevski]

If we use the python logging module or the sanic logger (which is just using the python logging module too), then when an error occurs during the running of a test the logs from it are not shown at all by pytest.

We saw this when schemathesis caused a failure that resulted in a 500 but when the test finishes the logs that pytest captured does not show anything and you cannot see or troubleshoot the problem.

To reproduce:

• pick an endpoint
• raise a random error (i.e. ValueError or KeyError)
• run schemathesis to test this endpoint
• schemathesis will fail because in all cases the endpoint will return 500 but the logs from pytest will not show anything even though the error handler should log the unhandled/unexpected exception

To remediate this we added a conditional print statement that prints the exception - and this shows up in pytest. But it should be that the logging module should simply work in this case.

What I tried to do to fix this is try to flush all logs before the server shuts down, or even try to flush all logs right after the exception handler printed the exception. But I could never get the error to show up in the logs when running pytest.