weird TypeError DryRunDataCube is not JSON serializable

[soxofaan]

reported by @jdries earlier today: sometimes GET https://openeo.dataspace.copernicus.eu/openeo/1.1/processes fails with

TypeError('Object of type DryRunDataCube is not JSON serializable')

[soxofaan]

from logs in kibana:

Traceback (most recent call last):
  File "/opt/openeo/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/openeo/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/opt/openeo/lib/python3.8/site-packages/openeo_driver/util/view_helpers.py", line 42, in wrapped
    return func(*args, **kwargs)
  File "/opt/openeo/lib/python3.8/site-packages/openeo_driver/views.py", line 672, in processes
    return jsonify({'processes': processes, 'links': []})
  File "/opt/openeo/lib/python3.8/site-packages/flask/json/__init__.py", line 361, in jsonify
    f"{dumps(data, indent=indent, separators=separators)}\n",
  File "/opt/openeo/lib/python3.8/site-packages/flask/json/__init__.py", line 139, in dumps
    rv = _json.dumps(obj, **kwargs)
  File "/usr/lib64/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/usr/lib64/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/opt/openeo/lib/python3.8/site-packages/flask/json/__init__.py", line 57, in default
    return super().default(o)
  File "/usr/lib64/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type DryRunDataCube is not JSON serializable

[soxofaan]

At the moment it also happens op openeo.vito.be, e.g.

• consistently on https://openeo.vito.be/openeo/1.1/processes/vito/BIOMASS and https://openeo.vito.be/openeo/1.1/processes/vito/CropSAR_px
• occasionally on https://openeo.vito.be/openeo/1.1/processes/vito/NDII

[soxofaan]

This one is hard to figure out, but I'm working on this hypothesis: we have this process graph node caching feature, which manipulates the process graph in-place (to cache values) and this manipulated process graph leaks into places where we assume the process graph is immutable.
Did first attempt to workaround this with Open-EO/openeo-python-driver@bf93bff