Wrong frame rate in output MPD/M3U8 with v1 CTTS boxes in input

[joeyparrish]

System info

Operating System: gLinux / Debian testing
Shaka Packager Version: v2.4.2-c60e988-release

Issue and steps to reproduce the problem

Packager Command:

wget https://storage.googleapis.com/shaka-streamer-assets/test-assets/BigBuckBunny.1080p.mp4

mkdir -p output

# With -movflags +negative_cts_offsets, we get v1 CTTS boxes with negative offsets
ffmpeg -y \
  -i BigBuckBunny.1080p.mp4 -map 0:v:0 \
  -an -c:v h264 -b:v 108k \
  -f mp4 -movflags +frag_keyframe -frag_duration 4000000.0 \
  -movflags +negative_cts_offsets \
  -keyint_min 120 -g 120 \
  intermediate_ctts_v1.mp4

packager 'in=intermediate_ctts_v1.mp4,stream=video,init_segment=output/ctts_v1_init.mp4,segment_template=output/ctts_v1_$Number$.mp4' \
  --segment_duration 4 \
  --generate_static_live_mpd --mpd_output output/dash_ctts_v1.mpd \
  --hls_playlist_type VOD --hls_master_playlist_output output/hls_ctts_v1.m3u8

# Without -movflags +negative_cts_offsets, we get v0 CTTS boxes
# and warnings from Packager about a missing edit list
ffmpeg -y \
  -i BigBuckBunny.1080p.mp4 -map 0:v:0 \
  -an -c:v h264 -b:v 108k \
  -f mp4 -movflags +frag_keyframe -frag_duration 4000000.0 \
  -keyint_min 120 -g 120 \
  intermediate_ctts_v0.mp4

packager 'in=intermediate_ctts_v0.mp4,stream=video,init_segment=output/ctts_v0_init.mp4,segment_template=output/ctts_v0_$Number$.mp4' \
  --segment_duration 4 \
  --generate_static_live_mpd --mpd_output output/dash_ctts_v0.mpd \
  --hls_playlist_type VOD --hls_master_playlist_output output/hls_ctts_v0.m3u8

Both sets of outputs should have the same frame rate, and according to ffprobe, they basically do:

# This shows 30fps for the version with v0 CTTS boxes
cat output/ctts_v0_{init,1}.mp4 | ffprobe -i -
# This shows 29.51fps for the version with v1 CTTS boxes
cat output/ctts_v0_{init,1}.mp4 | ffprobe -i -

But Shaka Packager puts very different numbers into the DASH manifests & HLS master playlists. Both DASH & HLS outputs say 30 fps for the version with v0 CTTS boxes, and 10 fps for the version with v1 CTTS boxes:

diff -u output/hls_ctts_v{0,1}.m3u8

--- output/hls_ctts_v0.m3u8     2020-04-15 11:11:54.357658216 -0700
+++ output/hls_ctts_v1.m3u8     2020-04-15 11:11:50.805648879 -0700
@@ -1,5 +1,5 @@
 #EXTM3U
 ## Generated with https://github.com/google/shaka-packager version v2.4.2-c60e988-release
 
-#EXT-X-STREAM-INF:BANDWIDTH=203692,AVERAGE-BANDWIDTH=123592,CODECS="avc1.640028",RESOLUTION=1920x1080,FRAME-RATE=30.000
+#EXT-X-STREAM-INF:BANDWIDTH=203692,AVERAGE-BANDWIDTH=123152,CODECS="avc1.640028",RESOLUTION=1920x1080,FRAME-RATE=10.000
 stream_0.m3u8

[joeyparrish]

It seems that the frame rate in the output manifests/playlists is based on the duration of the first sample. With v1 CTTS boxes, this sample duration is not calculated correctly.

I found a cheap solution: prefer the second sample duration over the first. :-) That seems to work in my tests, but there is probably a better solution. I'm still getting familiar with the source code.

[joeyparrish]

The correct duration is 512. The first segment has an offset of 1024, and Packager sees the duration as 1536. This is why it calculates a frame rate of 10 instead of 30.

[joeyparrish]

The output of a verbose log with some interesting info:

Pushing frame: , key=1, dur=1536, dts=0, cts=0, size=763
Pushing frame: , key=0, dur=512, dts=1536, cts=2560, size=265
Pushing frame: , key=0, dur=512, dts=2048, cts=1536, size=33
Pushing frame: , key=0, dur=512, dts=2560, cts=2048, size=15
Pushing frame: , key=0, dur=512, dts=3072, cts=3072, size=1001
Pushing frame: , key=0, dur=512, dts=3584, cts=3584, size=541
Pushing frame: , key=0, dur=512, dts=4096, cts=4096, size=1155

I'm not sure the first frame duration is actually "wrong". It might just be that it's the wrong thing for the frame-rate calculation.

So I see two relatively easy options to fix it:

1. Use the second frame duration instead of the first
2. Use the average frame rate over the whole presentation

In my example, I have 30 frames, 29 of which have the duration 512 and the first of which has the duration of 1536. The average duration is 546.

The final frame's CTS is 15360, but the max frame CTS is 15872. Since I'm encoding a 1-second clip, the difference between those two is significant, and makes the difference between a calculated frame rate of 29.07 fps and 28.13 fps.

The average seems less error-prone, especially for reasonably-sized content.

[kqyang]

@joeyparrish Thanks for looking into it.

For this particular stream, looks like the actual frame rate of the stream is 30 fps (15360 / 512). The first frame should have a timestamp of dts=1024 and cts=1024, but they are normalized to 0 by FFmpeg, which results in a duration of 1536 for the first frame.

So I see two relatively easy options to fix it:

1. Use the second frame duration instead of the first
2. Use the average frame rate over the whole presentation

I think taking the average is the right choice.

I am a bit concerned on the live packaging scenario, as there could be gaps and overlaps. There may be temporary fluctuation, but it should stablize over time with the average method.

[cosmin]

@joeyparrish are we good with the second frame duration approach? If not I have a patch to compute this based on average as well, and perhaps we could turn on the average for VOD mode (while leaving second sample for live?).

[joeyparrish]

Yeah, that sounds reasonable. Use second for live, use average for VOD. Average may be too difficult for live.