ExoPlayer.setVideoEffects seek stucked Occasionally

[captain81]

Version

Media3 main branch

More version details

When using ExoPlayer.setVideoEffects 。Normal codec.releaseOutputBuffer, the codec will send the frame buffer to the surface, and then the surfacetexture will have an onFrameAvaliable callback, indicating that the current frame drawing is completed.After seeking several times, codec will flush, codec.releaseOutputBuffer 5 times, but not receive any surfacetexture onFrameAvaliable callback. Why surfacetexture onFrameAvaliable callback not invoke?

Devices that reproduce the issue

Android14

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Yes

Reproduction steps

1.use ExoPlayer.setVideoEffects
2. seek many times

Expected result

seek anytimes play normal

Actual result

seek many times, play stuck

Media

NA

Bug Report

• You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

[captain81]

I have added this patch bef3d51. Seek stucked Occasionally.codec.releaseOutputBuffer 5 times, but not receive any surfacetexture onFrameAvaliable callback

[claincly]

This is curious, can you check a few things, given that you've already looked at ExternalTextureManager? We have trouble reproducing it

• Do you see Interrupted when waiting for MediaCodec frames to arrive being logged?
• Can you check the method releaseAllFramesFromMediaCodec() being called a few times?
• Can you check that availableFrameCount and pendingFrames are updated as expected?

I think by your description, it sounded like registerInputFrame() is called 5 times, after you seek, but SurfaceTexture.OnFrameAvailableListener is not called?

[captain81]

I have resolved this issue. As seek invoke flush, codec.releaseOutputBuffer queueBuffer to BufferQueue, but the consumer surfaceTexture.setOnFrameAvailableListener.OnFrameAvailable is interrupted by flush, so the queued Buffer can't consumed, codec.releaseOutputBuffer queueBuffer again surfaceTexture OnFrameAvailable can't be invoked forever. so we should invoke surfaceTexture.updateTexImage() to consumer the queued buffer here.

private void maybeQueueFrameToExternalShaderProgram() {
    if (externalShaderProgramInputCapacity.get() == 0
        || availableFrameCount == 0
        || currentFrame != null) {
      **_surfaceTexture.updateTexImage();_**
      return;
    }

Is this fix OK?

[claincly]

Thanks for the investigation, I think the find is valid. As for the fix I'm not sure it's safe to call surfaceTexture.updateTexImage() when the available count is zero.

We should address the issue from the point that the runnable (surfaceTexture.setOnFrameAvailableListener.OnFrameAvailable) is not cancelled during flushing.

OTOH, I wonder how do you manage to reliably trigger this case when testing?

[captain81]

Use ExoPlayer.setVideoEffects,just seek many times, when flush is between codec.releaseOutputBuffer and surfaceTexture.OnFrameAvailable , surfaceTexture.OnFrameAvailable will not invoke. If flush is after codec.releaseOutputBuffer and surfaceTexture.OnFrameAvailable, this issue cannot be reproduced. As OnFrameAvailable should use acquireBuffer, or OnFrameAvailable will not be invoked later. You can run seek or fastseek automatic

[captain81]

DefaultVideoFrameProcessor.flush() videoFrameProcessingTaskExecutor.flush(); will remove ExternalTextureManager's videoFrameProcessingTaskExecutor, so surfaceTexture.OnFrameAvailable can't invoke surfaceTexture.updateTexImage(). I Just move videoFrameProcessingTaskExecutor.flush() after textureManager.releaseAllRegisteredFrames(), as textureManager.releaseAllRegisteredFrames() has invoked surfaceTexture.updateTexImage()

      CountDownLatch latch = new CountDownLatch(1);
      TextureManager textureManager = inputSwitcher.activeTextureManager();
      textureManager.releaseAllRegisteredFrames();
      videoFrameProcessingTaskExecutor.flush();(move to here)

is this fix OK?

[claincly]

No not really - that's because when flushing, we want to stop VideoFrameProcessing doing any further work, including presenting newly processed frames, and flushing the task executor (sort of) guarantees that.

I think a proper fix would be making the task we post in surfaceTexture.OnFrameAvailable non-canceallable. I'll try push a fix

Thanks for providing the repro hint!