revisit error handling for getting batch job logs

[bossie]

This ERROR log has some peculiarities/bugs:

1. the (ref: ...) part is how the openeo-python-client typically renders the id (= request ID) in an error response (where e.g. OpenEO editor shows it in a dialog box IIRC): here it is already rendered in the back-end as part of the message:
2. the request ID is empty, for some reason (= defaults to "no-request");
3. the underlying error (a connection timeout towards ES) is returned in-band: it becomes a log entry with the exception rendered in the message, an empty (log) id in a 200 OK response.

3 is a consequence of choosing to stream the log entries from ES to the client: a 200 OK is already sent to the client and then log entries are appended to the response body. Maybe this is a case of premature optimization on my part and simply collecting the logs in memory before returning them to the client is better. The underlying error then becomes an actual 504 HTTP response and the client can retry the request, should he want to.

[soxofaan]

Found some answers/input while doing some digging for https://github.com/eu-cdse/openeo-cdse-infra/issues/29:

the ref: ... is explicitly added in the exception thrown from the ES query loop:

openeo-geopyspark-driver/openeogeotrellis/logs.py

Lines 152 to 156 in 9183af4

	message = (
	"Temporary failure while retrieving logs: ConnectionTimeout. "
	+ f"Please try again and report this error if it persists. (ref: {req_id})"
	)
	raise OpenEOApiException(status_code=504, message=message) from exc

the id=... is indeed part of the default rendering of OpenEOApiException but that is defined in python-driver, not python client:
https://github.com/Open-EO/openeo-python-driver/blob/3f49f2777959a4845afbb891ef697d68e1bcfdeb/openeo_driver/errors.py#L83-L84

the request ID is indeed unset (rendered as "no-request") because of this:
the get_job_logs view handler builds the response in a streaming fashion (response body is yielded in chunks):
https://github.com/Open-EO/openeo-python-driver/blob/3f49f2777959a4845afbb891ef697d68e1bcfdeb/openeo_driver/views.py#L1432-L1453
because of WSGI implementation details the flask request context is closed before the first response chunk is produced.
This practically means that the code that tries to get the request id

openeo-geopyspark-driver/openeogeotrellis/logs.py

Line 97 in 9183af4

req_id = FlaskRequestCorrelationIdLogging.get_request_id()

is executed when the request context is already gone, resulting in an unset request id. Conceptually, the code looks like it is inside the request context, but in practice that is not the case.

[soxofaan]

Did a first attempt in Open-EO/openeo-python-driver@e546e3d to get the actual request id in both the error logs in ES and the "Log collection for job ... failed" message that the end users gets