HLS Widevine key rotation does not work in case media playlist has multiple init segments (EXT-X-MAP HLS tags)

[ipichkov]

In case HLS media playlist contains multiple init segments (when Widevine key rotation is enabled), keyId from the first segment is being used for all segments, even for those coming after subsequent EXT-X-KEY/EXT-X-MAP HLS tags:

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-MEDIA-SEQUENCE:270157573
#EXT-X-TARGETDURATION:7
#EXT-X-DISCONTINUITY-SEQUENCE:794 
#EXT-X-PROGRAM-DATE-TIME:2021-06-01T16:53:03.000Z
#EXT-X-MAP:URI="CCURStream_MultiPortMulticast1-1_1606256636_init.cmfv"
#EXT-X-KEY:METHOD=SAMPLE-AES,KEYFORMAT="urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed",URI="data:text/plain;base64,AAAAW3Bzc2gAAAAA7e+LqXnWSs6jyCfc1R0h7QAAADsSECQFYZtCfBytQw1n5b0Bc9gaCnZlcmltYXRyaXgiDnI9bmFzYTRrJnM9ODY2KgVTRF9IREjzxombBg==",KEYID=0x2405619B427C1CAD430D67E5BD0173D8,IV=0x7cf7a20609d067665f5a2730983c6f41,KEYFORMATVERSIONS="1"
#EXT-X-MAP:URI="CCURStream_MultiPortMulticast1-1_1606256636_init.cmfv?ccur_keyrot_t=1622565000"
#EXTINF:6.006,
CCURStream_MultiPortMulticast1-1_T1622566383438000~D6006000.cmfv
...
#EXTINF:6.006,
CCURStream_MultiPortMulticast1-1_T1622566797852000~D6006000.cmfv
#EXT-X-KEY:METHOD=SAMPLE-AES,KEYFORMAT="urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed",URI="data:text/plain;base64,AAAAW3Bzc2gAAAAA7e+LqXnWSs6jyCfc1R0h7QAAADsSEJP3QFza93RoHe0BgjXsyxMaCnZlcmltYXRyaXgiDnI9bmFzYTRrJnM9ODY2KgVTRF9IREjzxombBg==",KEYID=0x93F7405CDAF774681DED018235ECCB13,IV=0x29f70e94c18a73f6f3d93cead30e736a,KEYFORMATVERSIONS="1"
#EXT-X-MAP:URI="CCURStream_MultiPortMulticast1-1_1606256636_init.cmfv?ccur_keyrot_t=1622566800"
#EXTINF:6.006,
CCURStream_MultiPortMulticast1-1_T1622566803858000~D6006000.cmfv
...
#EXTINF:6.006,
CCURStream_MultiPortMulticast1-1_T1622568599652000~D6006000.cmfv
#EXT-X-KEY:METHOD=SAMPLE-AES,KEYFORMAT="urn:uuid:edef8ba9-79d6-4ace-a3c8-27dcd51d21ed",URI="data:text/plain;base64,AAAAW3Bzc2gAAAAA7e+LqXnWSs6jyCfc1R0h7QAAADsSEMQDLGy9XZwClrMz0ri86nsaCnZlcmltYXRyaXgiDnI9bmFzYTRrJnM9ODY2KgVTRF9IREjzxombBg==",KEYID=0xC4032C6CBD5D9C0296B333D2B8BCEA7B,IV=0x651d8ad62eea3bfd5d539f53c05ab32f,KEYFORMATVERSIONS="1"
#EXT-X-MAP:URI="CCURStream_MultiPortMulticast1-1_1606256636_init.cmfv?ccur_keyrot_t=1622568600"
#EXTINF:6.006,
CCURStream_MultiPortMulticast1-1_T1622568605658000~D6006000.cmfv
...
#EXTINF:6.006,
CCURStream_MultiPortMulticast1-1_T1622569975026000~D6006000.cmfv

This triggers Widevine CDM decrypt error after a new init segment gets fetched and attempted to be used as player continues reusing FragmentedMp4Extractor instance:

05-19 15:48:10.503  3135  3135 E WVCdm   : [policy_engine.cpp(55):CanDecryptContent] PolicyEngine::CanDecryptContent Key 'EEA09A0642061FC28DB43E34CE7E05B4' not in license.
05-19 15:48:10.503  3135  3135 E WVCdm   : Decrypt error result in session sid46 during encrypted block: 5
05-19 15:48:10.516 16030 16030 D EventLogger: drmSessionReleased [eventTime=504.12, mediaPos=2408.05, window=0, period=0]
05-19 15:48:10.536 16030 16030 D EventLogger: drmSessionReleased [eventTime=504.14, mediaPos=2408.05, window=0, period=0]
05-19 15:48:10.556 16030 16030 D EventLogger: drmSessionReleased [eventTime=504.16, mediaPos=2408.05, window=0, period=0]
05-19 15:48:10.575 16030 16030 D EventLogger: drmSessionReleased [eventTime=504.18, mediaPos=2408.05, window=0, period=0]
05-19 15:48:10.595 16030 16030 D EventLogger: drmSessionReleased [eventTime=504.20, mediaPos=2408.05, window=0, period=0]
05-19 15:48:10.614 16030 16030 D EventLogger: drmSessionReleased [eventTime=504.22, mediaPos=2408.05, window=0, period=0]
05-19 15:48:10.634 16030 16030 D EventLogger: drmSessionReleased [eventTime=504.24, mediaPos=2408.05, window=0, period=0]
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal: Playback error
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:   com.google.android.exoplayer2.ExoPlaybackException: MediaCodecVideoRenderer error, index=0, format=Format(2, null, null, video/avc, avc1.64001E, 291776, null, [720, 480, 29.97], [-1, -1]), format_supported=NO_UNSUPPORTED_DRM
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at com.google.android.exoplayer2.BaseRenderer.createRendererException(BaseRenderer.java:333)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.feedInputBuffer(MediaCodecRenderer.java:1345)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.render(MediaCodecRenderer.java:800)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at com.google.android.exoplayer2.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:812)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at com.google.android.exoplayer2.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:401)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at android.os.Handler.dispatchMessage(Handler.java:102)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at android.os.Looper.loop(Looper.java:193)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at android.os.HandlerThread.run(HandlerThread.java:65)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:   Caused by: android.media.MediaCodec$CryptoException: Crypto key not available
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at android.media.MediaCodec.native_queueSecureInputBuffer(Native Method)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at android.media.MediaCodec.queueSecureInputBuffer(MediaCodec.java:2609)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at com.google.android.exoplayer2.mediacodec.SynchronousMediaCodecAdapter.queueSecureInputBuffer(SynchronousMediaCodecAdapter.java:63)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       at com.google.android.exoplayer2.mediacodec.MediaCodecRenderer.feedInputBuffer(MediaCodecRenderer.java:1336)
05-19 15:48:10.642 16030 16256 E ExoPlayerImplInternal:       ... 6 more

The issue was originally observed in 2.11.4 code base. It appears that HlsMediaChunk.createInstance API would need to check if initDataSpec is changed (e.g. different URI) when making a decision to reuse Extractor instance.

Hopefully the following diff makes the issue clear:

--- a/library/hls/src/main/java/com/google/android/exoplayer2/source/hls/HlsMediaChunk.java
+++ b/library/hls/src/main/java/com/google/android/exoplayer2/source/hls/HlsMediaChunk.java

@@ -143,6 +143,8 @@ import org.checkerframework.checker.nullness.qual.RequiresNonNull;
           previousChunk.isExtractorReusable
                   && previousChunk.discontinuitySequenceNumber == discontinuitySequenceNumber
                   && !shouldSpliceIn
+                  && (previousChunk.initDataSpec == initDataSpec || previousChunk.initDataSpec != null && initDataSpec != null && previousChunk.initDataSpec.uri == initDataSpec.uri)
               ? previousChunk.extractor
               : null;
       if (previousChunk.discontinuitySequenceNumber != discontinuitySequenceNumber) {

Note, the same issue is reproducible in release-v2 as well. It can be addressed with a similar change:

--- a/library/hls/src/main/java/com/google/android/exoplayer2/source/hls/HlsMediaChunk.java
+++ b/library/hls/src/main/java/com/google/android/exoplayer2/source/hls/HlsMediaChunk.java

@@ -146,10 +146,13 @@ import org.checkerframework.checker.nullness.qual.RequiresNonNull;
           isFollowingChunk
               || (isIndependent && segmentStartTimeInPeriodUs >= previousChunk.endTimeUs);
       shouldSpliceIn = !canContinueWithoutSplice;
+      boolean sameInitDataSpec = (previousChunk.initDataSpec == initDataSpec || previousChunk.initDataSpec != null && initDataSpec != null && previousChunk.initDataSpec.uri == initDataSpec.uri);
       previousExtractor =
           isFollowingChunk
                   && !previousChunk.extractorInvalidated
                   && previousChunk.discontinuitySequenceNumber == discontinuitySequenceNumber
+                  && sameInitDataSpec
               ? previousChunk.extractor
               : null;
     } else {

The issue was reproduced on Android TV device SEI500TV, Pyxis, SEI Robotics, API level 28.

[icbaker]

Have you configured multi-session DRM support as described here? https://exoplayer.dev/drm.html#key-rotation You shouldn't need a whole new Extractor when the keys change, only the DrmSession needs to be swapped out (while keeping the same SampleQueue, Renderer and MediaCodec etc). Without multi-session support enabled the same DrmSession is used even when the drmInitData changes - so you would expect the decryption errors you're seeing.

If that's not the problem and something else is going on, please provide content we can use to reproduce the problem. Please either upload it here or send to dev.exoplayer@gmail.com using a subject in the format "Issue #1234" (where "#1234" should be replaced with this issue number.) Please also update this issue to indicate you’ve done this.

[ipichkov]

Yes, multi-session DRM support was enabled when reproducing the issue.

I can see that on key rotation (when subsequent EXT-X-KEY HLS tag gets handled) new DRM session is created with new KID as a result of adjusting upstream format in HlsSampleQueue.getAdjustedUpstreamFormat API, while the queued samples continue to use the CryptoData/KID from the previous EXT-X-KEY tag as HlsMediaChunk skips loading init segment as a result of reusing previous extractor:

  public void load() throws IOException {
    ...
    if (extractor == null && previousExtractor != null) {
      ...
      initDataLoadRequired = false;
    }
    maybeLoadInitData();
    ...
}

  private void maybeLoadInitData() throws IOException {
    if (!initDataLoadRequired) {
      return;
    }
    ...
  }

Decrypting sample triggers PolicyEngine::CanDecryptContent Key 'xxx' not in license error as the sample is assigned encryption metadata (KID) from the previous key, but current DRM session contains key for a new KID.

I will look into providing access / corresponding DRM info for an asset to reproduce the issue.

[ipichkov]

The asset info to reproduce the problem has been sent to dev.exoplayer@gmail.com

[icbaker]

Thanks for the link to the media, I was able to reproduce the problem.

I will investigate further.

[ipichkov]

Thanks for confirming.

[ipichkov]

Folks, please let me know if you need more info or there is any update you would like to share.

[ojw28]

There are a few things happening at the point of key rotation:

• The DRM init data (i.e., the PSSH box) changes. This occurs in both the media playlist (with a new data URI specified by an EXT-X-KEY tag) and in the FMP4 stream (embedded in the new initialization segment). ExoPlayer gives the DRM init data in the media playlist priority over that in the FMP4 stream. This is desirable, since it allows DRM init data to be changed without having to re-encode the underlying FMP4 assets. As a result, even though we're not loading the new initialization segment, ExoPlayer still processes the change in DRM init data at the correct point and attaches a new DRM session.
• The problem is that the initialization segment that we're not loading also contains other information that changes as a result of the key rotation. In particular, the default key ID in the SCHI box. As a result of not loading the initialization segment, samples after the key rotation are written to the sample queue with a key ID as though the key rotation hasn't occurred. Playback then fails because the new session does not have the old key loaded into it (although even if it did, decryption would fail because it's not the correct one to be using).

The fix here will be to ensure that the initialization segment is always loaded if changed, as alluded to in the original issue description. Thanks!

[ojw28]

We'll get a fixed merged for this shortly.

@ipichkov - The way this media is setup seems sub-optimal, and possibly (although I'm not certain) in violation of the CMAF specification. In particular, I don't think you're supposed to put PSSH boxes in initialization segments (this document relates to DASH, but heavily implies that this is required by the CMAF specification). I'm pretty sure you can indicate default key IDs somewhere in the moof at the start of each segment. If you were to follow this approach, then I don't think you'd ever need to change the initialization segment, since the changing PSSH box would be communicated in the media playlist only, and the changing default key ID would be communicated in the individual segments. If this media is under your control, you may wish to investigate switching to this kind of approach.

[ipichkov]

Thank you @ojw28 for the update. Unfortunately the asset packaging is outside of our control. I'm going to discuss this with Origin software vendor though to see if this is something the can address in the nearest future.