Possible Threading Issue

[BananaHemic]

I understand, as outlined in the README.usage, that Gstreamer and mfx are heavily threaded frameworks that don't share synchronization mechanisms. That being said, I have come across a threading issue that I believe should be fixable.

The pipeline:
gst-launch-1.0 videotestsrc is-live=1 ! video/x-raw,width=4096,height=1536,framerate=60/1,format=NV12 ! queue ! gamma gamma=0.5 ! queue ! mfxh264enc ! video/x-h264,profile=high ! h264parse ! matroskamux name=mux streamable=false ! queue ! filesink location="out.mkv" sync=1

Creates a file with very visible artifacts. It appears, at least to me, that mfxh264enc isn't locking the frame throughout the entire coding process. As a result, parts of the frame have the gamma applied, and parts do not. I tested this on an i7-6770HQ.

[ishmael1985]

Tried the pipeline you provided, and yes I can see the intermittent corruption.
I then applied the async-depth=16 option for mfxh264enc and I don't see the artifacts anymore, at least from my side.
Indeed, the problem is that there is no way afaik that downstream knows whether an MFX frame surface is locked or unlocked by MSDK, hence the real solution to this fix would be to devise an explicit synchronization mechanism that notifies downstream that a surface is free for use. In the same way, MSDK has no idea whether downstream is currently working on the locked surface, so it proceeds to unlock the surface regardless of whether GStreamer is still using it or not.
The only workaround so far around this would be to ensure that there are enough surfaces available to compensate for the pipeline latency, so increasing the async-depth is one of those options.

[BananaHemic]

Would it be possible for the encoder to detect if there are no other mfx elements? If there are no other mfx elements then, presuming I correctly understand the issue, we would then know when the surface is in use. Is that correct?

[ishmael1985]

The MFX encoder task detects any upstream peer MFX elements such an MFX decoder or MFX VPP task for MFX session sharing, as well as to compute the optimal number of surfaces to be shared between the upstream MFX task and the MFX encoder task.
To clarify the issue here, it's when the downstream GStreamer element (some other sink, or a postprocessing element such videocrop or gamma for example) is still processing the MFX surface, but the MSDK framework currently has no mechanism that can be tightly coupled with the GStreamer framework to know if the downstream element is still processing the MFX surface or not. The MSDK framework internally releases locked MFX surfaces when it is done processing them, and we can detect the status of those MFX surfaces via a 'locked' flag which is set / unset by the MSDK framework itself and not the application. But this does not necessarily mean that the MFX surface itself has been completely processed by the downstream element, and so there is a possibility of read-write hazards on the MFX surface as the MSDK framework then proceeds to write new contents on the MFX surface but the GStreamer downstream is still reading / processing the same MFX surface that is fixed on some GstVideoCodecFrame, hence you see that corruption.
In other words, we'd definitely be able to know if the surface is being used by MSDK or not, but we can't guarantee it is still being used by some downstream GStreamer element. That's why it's important to devise an explicit synchronization mechanism that tightly couples both MSDK and GStreamer framework to resolve this kind of issue.