Not able to package WebVTT

[kaushikkunal99]

System info

Operating System: macOS Sierra
Shaka Packager Version: packager version v2.0.3-ef93a1d-release

Issue and steps to reproduce the problem

P2021944:packager kkaushik$ ./package.sh
[0516/122411:ERROR:webvtt_parser.cc(102)] Failed to read WEBVTT header - should be WEBVTT but was WEBVTT
[0516/122411:INFO:demuxer.cc(89)] Demuxer::Run() on file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/A/ToS_1080p_3M/video.mp4'.
[0516/122411:INFO:demuxer.cc(161)] Initialize Demuxer for file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/A/ToS_1080p_3M/video.mp4'.
[0516/122411:INFO:demuxer.cc(89)] Demuxer::Run() on file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/A/ToS_720p_2M/video.mp4'.
[0516/122411:INFO:demuxer.cc(161)] Initialize Demuxer for file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/A/ToS_720p_2M/video.mp4'.
[0516/122411:INFO:demuxer.cc(89)] Demuxer::Run() on file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/B/ToS_1080p_3M/video.mp4'.
[0516/122411:INFO:demuxer.cc(161)] Initialize Demuxer for file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/B/ToS_1080p_3M/video.mp4'.
[0516/122411:INFO:demuxer.cc(89)] Demuxer::Run() on file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/B/ToS_720p_2M/video.mp4'.
[0516/122411:INFO:demuxer.cc(161)] Initialize Demuxer for file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/B/ToS_720p_2M/video.mp4'.
[0516/122411:ERROR:packager_main.cc(477)] Packaging Error: 14 (INTERNAL_ERROR): Failed to parse WebVTT source. See log for details.

Packager Command:

/Users/kkaushik/Documents/development/packager/packager
'in=/Users/kkaushik/Documents/development/packager/content/paulblart/wm/A/ToS_1080p_3M/video.mp4,stream=audio,init_segment=paulblart_enc/wm/audio/und/mp4a/init.mp4,segment_template=paulblart_enc/wm/audio/und/mp4a/$Number$.m4s'
'in=/Users/kkaushik/Documents/development/packager/content/paulblart/sub.vtt,stream=text,init_segment=paulblart_enc/wm/text/init.mp4,segment_template=paulblart_enc/wm/text/$Number$.m4s'
'in=/Users/kkaushik/Documents/development/packager/content/paulblart/wm/A/ToS_1080p_3M/video.mp4,stream=video,init_segment=paulblart_enc/wm/video/avc1/ToS_1080p_3M/A/seg_init_0.mp4,segment_template=paulblart_enc/wm/video/avc1/ToS_1080p_3M/A/seg_$Time$.m4s'
'in=/Users/kkaushik/Documents/development/packager/content/paulblart/wm/B/ToS_1080p_3M/video.mp4,stream=video,init_segment=paulblart_enc/wm/video/avc1/ToS_1080p_3M/B/seg_init_0.mp4,segment_template=paulblart_enc/wm/video/avc1/ToS_1080p_3M/B/seg_$Time$.m4s'
'in=/Users/kkaushik/Documents/development/packager/content/paulblart/wm/A/ToS_720p_2M/video.mp4,stream=video,init_segment=paulblart_enc/wm/video/avc1/ToS_720p_2M/A/seg_init_0.mp4,segment_template=paulblart_enc/wm/video/avc1/ToS_720p_2M/A/seg_$Time$.m4s'
'in=/Users/kkaushik/Documents/development/packager/content/paulblart/wm/B/ToS_720p_2M/video.mp4,stream=video,init_segment=paulblart_enc/wm/video/avc1/ToS_720p_2M/B/seg_init_0.mp4,segment_template=paulblart_enc/wm/video/avc1/ToS_720p_2M/B/seg_$Time$.m4s'
--generate_static_mpd --mpd_output paulblart_enc/wm/pb.mpd
--enable_widevine_encryption
--key_server_url http://license.uat.widevine.com/cenc/getcontentkey/widevine_test
--content_id 2c5889b73007
--signer widevine_test
--aes_signing_key 1ae8ccd0e7985cc0b6203a55855a1034afc252980e970ca90e5202689f947ab9
--aes_signing_iv d58ce954203b7c9a9a9d467f59839249

Extra steps to reproduce the problem?
(1)Run the packaging script
(2)immediately throws the error

What is the expected result?
Should create the dash distribution

What happens instead?
P2021944:packager kkaushik$ ./package.sh
[0516/122411:ERROR:webvtt_parser.cc(102)] Failed to read WEBVTT header - should be WEBVTT but was WEBVTT
[0516/122411:INFO:demuxer.cc(89)] Demuxer::Run() on file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/A/ToS_1080p_3M/video.mp4'.
[0516/122411:INFO:demuxer.cc(161)] Initialize Demuxer for file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/A/ToS_1080p_3M/video.mp4'.
[0516/122411:INFO:demuxer.cc(89)] Demuxer::Run() on file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/A/ToS_720p_2M/video.mp4'.
[0516/122411:INFO:demuxer.cc(161)] Initialize Demuxer for file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/A/ToS_720p_2M/video.mp4'.
[0516/122411:INFO:demuxer.cc(89)] Demuxer::Run() on file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/B/ToS_1080p_3M/video.mp4'.
[0516/122411:INFO:demuxer.cc(161)] Initialize Demuxer for file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/B/ToS_1080p_3M/video.mp4'.
[0516/122411:INFO:demuxer.cc(89)] Demuxer::Run() on file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/B/ToS_720p_2M/video.mp4'.
[0516/122411:INFO:demuxer.cc(161)] Initialize Demuxer for file '/Users/kkaushik/Documents/development/packager/content/paulblart/wm/B/ToS_720p_2M/video.mp4'.
[0516/122411:ERROR:packager_main.cc(477)] Packaging Error: 14 (INTERNAL_ERROR): Failed to parse WebVTT source. See log for details.

[vaage]

@kaushikkunal99 Could you provide us with the webvtt file you were trying to package with. The packager is saying that there is a problem with the header of the file. Without the file, we won't be able to properly identify what is wrong.

[kaushikkunal99]

sub.vtt.zip

Please find the attached file

[vaage]

@kaushikkunal99 Thanks. It looks like we may be handling the whitespaces in a less than robust way. I will take a closer look at it.

[kaushikkunal99]

Thanks Vaage, I will wait for the fix.

[vaage]

So after looking at this a little bit deeper. Technically the content you provided is not to spec. WebVTT can only start with WEBVTT or \xFE\xFFWEBVTT. Yours start withs \xEF\xBB\xBFWEBVTT. The first bytes are an optional byte order mark and may only be FEFF.

So, this will fall to @kqyang. Do we want to be hard-set on the spec and require all content to start with WEBVTT or \xFE\xFFWEBVTT, or do we want to support some flexibility and allow it to start with WEBVTT with the leading and trailing non-displayable characters stripped away?

[kqyang]

\xFE\xFF is the BOM (byte order mark) for UTF-16 and \xEF\xBB\xBF is the BOM for UTF-8: https://en.wikipedia.org/wiki/Byte_order_mark.

@vaage

It is a bit misleading but I think U+FEFF BYTE ORDER MARK (BOM) in WebVTT specification actually means \xEF\xBB\xBF. U+FEFF seems to be a common synonym for BOM characters, which is encoded differently in different encodings: http://www.fileformat.info/info/unicode/char/FEFF/index.htm.

A WebVTT file must be encoded in UTF-8, so the BOM characters are \xEF\xBB\xBF. See Magic number(s) section in https://www.w3.org/TR/webvtt1/#text-vtt for the list of allowed starting byte sequences.