HLS Widevine VOD playback stops with "Playback failed" at the second (and every subsequent) key rotation

[kapvode]

Issue description

When I play the video, it stops at the point where it needs to load the third key. The message on the screen is "Playback failed".

The exception message on an Android 7.0 device is:

com.google.android.exoplayer2.ExoPlaybackException: android.media.MediaCodec$CryptoException: Error decrypting data: requested key has not been loaded

On an Android 9.0 device, the exception message is:

com.google.android.exoplayer2.ExoPlaybackException: android.media.MediaCodec$CryptoException: Crypto key not available

If I press play, the video resumes until the next key rotation, when it stops with the same error. I can keep repeating this.

Reproduction steps

I downloaded the 720p version of Tears of Steel from YouTube, using youtube-dl, and renamed it to tears.mp4.

youtube-dl -F 'https://www.youtube.com/watch?v=41hv2tW5Lc4'

22           mp4        1280x534   720p 1382k , avc1.64001F, mp4a.40.2@192k (44100Hz) (best)

youtube-dl -f 22 'https://www.youtube.com/watch?v=41hv2tW5Lc4'

I used the pre-compiled Shaka Packager v2.4.1 Linux binary.

I prepared two versions of the video: one with a clear lead, and one without (remove the --clear_lead 10 line). The content ID is test.

packager \
    --enable_widevine_encryption \
    --clear_lead 10 \
    --key_server_url https://license.uat.widevine.com/cenc/getcontentkey/widevine_test \
    --signer widevine_test \
    --aes_signing_key 1ae8ccd0e7985cc0b6203a55855a1034afc252980e970ca90e5202689f947ab9 \
    --aes_signing_iv d58ce954203b7c9a9a9d467f59839249 \
    --content_id 74657374 \
    --crypto_period_duration 10 \
    --protection_scheme cenc \
    --segment_duration 10 \
    --hls_master_playlist_output index.m3u8 \
    --mpd_output index.mpd \
    --generate_static_live_mpd \
    input=../tears.mp4,stream=audio,segment_template='audio-$Number%03d$.mp4',playlist_name='audio.m3u8',init_segment='audio-init.mp4' \
    input=../tears.mp4,stream=video,segment_template='video-$Number%03d$.mp4',playlist_name='video.m3u8',init_segment='video-init.mp4'

I tried playing the stream in the demo app:

{
    "name": "Test stream",
    "uri": "http://web-server/stream/index.{m3u8|mpd}",
    "drm_scheme": "widevine",
    "drm_license_url": "http://cwip-shaka-proxy.appspot.com/no_auth",
    "drm_multi_session": true
}

Playing the HLS version results in the described behavior. If it has the clear lead, the error happens a bit later. It always happens when the third key is about to be loaded.

The MPD version works fine.

Link to test content

Already explained.

A full bug report captured from the device

I hope the reproduction steps are enough. If not, I will capture one later.

Version of ExoPlayer being used

I used the demo app from r2.11.1.

Device(s) and version(s) of Android being used

Huawei P10 Lite (WAS-LX1), Android 7.0, API 24.

Wileyfox Swift, Android 9.0, API 28.

[kapvode]

For example, these are the first few segments in the example without a clear lead, and slightly abbreviated.

video.m3u8

#EXT-X-KEY
#EXTINF:13.458,
video-001.mp4
#EXT-X-KEY
#EXTINF:9.958,
video-002.mp4
#EXT-X-KEY
#EXTINF:10.000,
video-003.mp4
#EXT-X-KEY
#EXTINF:6.833,
video-004.mp4

audio.m3u8

#EXT-X-KEY
#EXTINF:10.008,
audio-001.mp4
#EXT-X-KEY
#EXTINF:10.008,
audio-002.mp4
#EXT-X-KEY
#EXTINF:9.985,
audio-003.mp4
#EXT-X-KEY
#EXTINF:10.008,
audio-004.mp4

Playback stops at 00:20, which is when the second audio segment ends, and we need the third key.

When I press play again, it resumes and stops at 00:30, which is when the third audio segment ends, and we need the fourth key.

This is an example with a clear lead.

video.m3u8

#EXTINF:13.458,
video-001.mp4
#EXT-X-DISCONTINUITY
#EXT-X-KEY
#EXTINF:9.958,
video-002.mp4
#EXT-X-KEY
#EXTINF:10.000,
video-003.mp4
#EXT-X-KEY
#EXTINF:6.833,
video-004.mp4

audio.m3u8

#EXTINF:10.008,
audio-001.mp4
#EXT-X-DISCONTINUITY
#EXT-X-KEY
#EXTINF:10.008,
audio-002.mp4
#EXT-X-KEY
#EXTINF:9.985,
audio-003.mp4
#EXT-X-KEY
#EXTINF:10.008,
audio-004.mp4

Playback stops at 00:30. When I resume, it stops at 00:40.

[kapvode]

I made a mistake in the reproduction steps.

To create an example without a clear lead, use --clear_lead 0, instead of removing the parameter completely.

[icbaker]

I'm not able to repro this by following your instructions.

Can you either host the stream somewhere and send a link to the manifest, or upload the whole directory somewhere so I can ensure I'm using exactly the same media.

[kapvode]

Thank you for the quick response.

This is the original video:
https://devel.uniqcast.com/test/exoplayer-6309/tears.mp4

HLS without a clear lead:
https://devel.uniqcast.com/test/exoplayer-6309/av/index.m3u8

MPD without a clear lead:
https://devel.uniqcast.com/test/exoplayer-6309/av/index.mpd

HLS with a clear lead:
https://devel.uniqcast.com/test/exoplayer-6309/av-clear-lead/index.m3u8

MPD with a clear lead:
https://devel.uniqcast.com/test/exoplayer-6309/av-clear-lead/index.mpd

[icbaker]

Thanks for the links, I can repro exactly how you describe.

I added some logging to the OnKeyStatusChangeListener and noticed the following when playing the no-clearlead version.

At the beginning of playback, during the initial load, these 6 key IDs are returned from the DRM server:

0793A30D8FBF5053B1277F0D2ADA0DDF   // audio-002.mp4
11CF13015A8B50628A978892C2533DB0   // Doesn't appear in either audio.m3u8 or video.m3u8
361A86CEECC35CD6A586A92B678F7437   // video-001.mp4
42226A85883D52539C0CF019B861FCA3   // audio-001.mp4
42C9EFBC6ADE54B0905BDBF297D2F2BF   // Doesn't appear in either audio.m3u8 or video.m3u8
EAE03BBB46835E09993165474604CEA5   // video-002.mp4

i.e. we have enough info at that point to decrypt the first 2 audio & video segments, plus a couple of key IDs that I don't know what they're for.

Then when the playback fails, and I press 'resume', the following Key IDs are returned from the DRM server:

0793A30D8FBF5053B1277F0D2ADA0DDF   // audio-002.mp4
11CF13015A8B50628A978892C2533DB0   // Doesn't appear in either audio.m3u8 or video.m3u8
361A86CEECC35CD6A586A92B678F7437   // video-001.mp4
42226A85883D52539C0CF019B861FCA3   // audio-001.mp4
42C9EFBC6ADE54B0905BDBF297D2F2BF   // Doesn't appear in either audio.m3u8 or video.m3u8
4C0A7A769A135F9FB1D76409FD8E91FB   // video-003.mp4
6B91D0A25CB3580F86655B2BAD6B6BFC   // audio-003.mp4
EAE03BBB46835E09993165474604CEA5   // video-002.mp4
F7613BE149765079820B7EB28DDD4723   // Doesn't appear in either audio.m3u8 or video.m3u8

So it seems at the moment like we're not issuing new license requests during HLS playback for upcoming segments with different keys.

[kapvode]

The keys that you don't see in the media playlists are for SD video. The license proxy seems to be configured to return audio, SD and HD keys, even though the example only uses the HD video key.

If it helps, I have also created a Dockerfile that should allow you to reproduce the streams locally.

FROM nginx:1.17

RUN apt update \
    && apt install -y python curl

RUN cd /usr/local/bin \
    && curl -LO https://github.com/ytdl-org/youtube-dl/releases/download/2020.01.24/youtube-dl \
    && chmod +x youtube-dl \
    && curl -L -o packager https://github.com/google/shaka-packager/releases/download/v2.4.1/packager-linux \
    && chmod +x packager

RUN cd /usr/share/nginx/html \
    && youtube-dl -f 22 -o tears.mp4 'https://www.youtube.com/watch?v=41hv2tW5Lc4' \
    && mkdir no-clear-lead \
    && cd no-clear-lead \
    && packager \
        --enable_widevine_encryption \
        --key_server_url https://license.uat.widevine.com/cenc/getcontentkey/widevine_test \
        --signer widevine_test \
        --aes_signing_key 1ae8ccd0e7985cc0b6203a55855a1034afc252980e970ca90e5202689f947ab9 \
        --aes_signing_iv d58ce954203b7c9a9a9d467f59839249 \
        --content_id 74657374 \
        --protection_scheme cenc \
        --clear_lead 0 \
        --crypto_period_duration 10 \
        --segment_duration 10 \
        --hls_master_playlist_output index.m3u8 \
        --mpd_output index.mpd \
        --generate_static_live_mpd \
        input=../tears.mp4,stream=audio,segment_template='audio-$Number%03d$.mp4',playlist_name='audio.m3u8',init_segment='audio-init.mp4' \
        input=../tears.mp4,stream=video,segment_template='video-$Number%03d$.mp4',playlist_name='video.m3u8',init_segment='video-init.mp4' \
    && cd .. \
    && mkdir clear-lead \
    && cd clear-lead \
    && packager \
        --enable_widevine_encryption \
        --key_server_url https://license.uat.widevine.com/cenc/getcontentkey/widevine_test \
        --signer widevine_test \
        --aes_signing_key 1ae8ccd0e7985cc0b6203a55855a1034afc252980e970ca90e5202689f947ab9 \
        --aes_signing_iv d58ce954203b7c9a9a9d467f59839249 \
        --content_id 74657374 \
        --protection_scheme cenc \
        --clear_lead 10 \
        --crypto_period_duration 10 \
        --segment_duration 10 \
        --hls_master_playlist_output index.m3u8 \
        --mpd_output index.mpd \
        --generate_static_live_mpd \
        input=../tears.mp4,stream=audio,segment_template='audio-$Number%03d$.mp4',playlist_name='audio.m3u8',init_segment='audio-init.mp4' \
        input=../tears.mp4,stream=video,segment_template='video-$Number%03d$.mp4',playlist_name='video.m3u8',init_segment='video-init.mp4'

docker build -t example .
docker run --rm -it -p 8000:80 example

The URLs should now be:

http://ip:8000/tears.mp4
http://ip:8000/no-clear-lead/index.{m3u8,mpd}
http://ip:8000/clear-lead/index.{m3u8,mpd}

[ojw28]

Key rotation appears to be broken for HLS due to the way the DrmInitData is (or isn't :)) propagated.

[ojw28]

This should be fixed in dev-v2.