Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add rawResponse field to BaseChunkSampleSourceEventListener.onLoadCompleted #1414

Closed
Skrewtape opened this issue Apr 4, 2016 · 11 comments
Closed

Comments

@Skrewtape
Copy link

Our backend can insert several custom fields into the manifest of a video stream, which we need to interpret at playback time. It doesn't make sense to have ExoPlayer read, parse and understand these fields, but it would be nice if we could read them as they're downloaded by the player.

In order to solve this problem, we'd like to add a field called "rawResponse" to the method signature for the onLoadCompleted event callback, which would contains the raw bytes downloaded for that request, and would be populated for any files that have the type set to Chunk.TYPE_MANIFEST; i.e. DASH or HLS manifest files.

Alternately, if you can think of an alternate way to accomplish this same goal, we'd be more than willing to explore that path.

@ojw28
Copy link
Contributor

ojw28 commented Apr 4, 2016

Please provide more detail. Are you using DASH, SmoothStreaming or HLS? Is this for live streaming or for on-demand? I don't think the callback to which you refer is invoked for manifest load or refresh for DASH or SmoothStreaming.

@Skrewtape
Copy link
Author

We need this functionality for live HLS playback, but I imagine it'd also make sense for DASH or SS. Is there a different callback that could be used to handle all the cases?

@ojw28
Copy link
Contributor

ojw28 commented Apr 4, 2016

For DASH and SmoothStreaming custom data can be embedded directly into the media in EMSG boxes, which we'll be adding support for in 2.x.

I'm not really sure what the best approach would be. Can you provide an example of what these fields look like and do, how other players provide you with access, and when you expect to receive them?

@Skrewtape
Copy link
Author

The fields are custom tags in the HLS source -- they're proprietary, and give us playback-time metadata about the video as it's playing.

For example, on iOS, we use the Resource Loader Delegate to get access to the raw HLS variant data after it's downloaded. We see them as each HLS variant is downloaded and can update our internal state as playback proceeds.

@ojw28
Copy link
Contributor

ojw28 commented Apr 11, 2016

Question - Is there a reason why you don't burn your metadata into the stream as ID3, which I think would be the normal way to do something like this in the context of HLS. See this, for example. ExoPlayer already supports metadata delivery via ID3.

@Skrewtape
Copy link
Author

We're adding this extra metadata to the stream in real time; adding ID3 tags would require us to rewrite individual fragments at delivery time. This allows us to accomplish the same result with simple text manipulation instead, and also allows us to include fragments that might not originate on our servers.

Another solution we'd considered, but which seemed more complicated, was to emulate / co-opt the onMetadata interface to emit events for any unrecognized tags in the master or variant M3U8 files. Would something like that make more sense to you?

@Skrewtape
Copy link
Author

Would you be more amenable to a more targeted approach? What if, instead of adding on to the OnDownloadComplete event, we added a new event called like OnHlsVariantDownload or something like that? Or, we could add a new event that could be dispatched when HLS tags are parsed.

Or if you want to go further up the stack, we could introduce something like the AVPlayer's Resource Loader Delegate -- a class that could act as a proxy for data downloads, which we could implement to do whatever we liked with the data as it's being downloaded.

@alanra
Copy link

alanra commented Apr 26, 2016

Hi folks. I'm Alan, and I work with, umm, "Skrewtape", and I thought I'd add some clarification. What we're working on is a server-side ad insertion solution where we take content and interleave it with ad chunks. E.g., you might have a sample stitched result like this:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:11
#EXT-X-MEDIA-SEQUENCE:1
#EXTINF:10,
http://pmd205470tn.download.theplatform.com.edgesuite.net/Tech_Summit_2014_-_UX/164/731/citizen_401750083977_m3u8_video_1280x0_2432000_primary_audio_6_x4ac72b8f84f244eca9508012dfdbde58_00001.ts
#EXTINF:5.6,
http://pmd205470tn.download.theplatform.com.edgesuite.net/Tech_Summit_2014_-_UX/164/731/citizen_401750083977_m3u8_video_1280x0_2432000_primary_audio_6_x4ac72b8f84f244eca9508012dfdbde58_00002.ts
#EXT-X-DISCONTINUITY
#EXTINF:10,
http://pmd205470tn.download.theplatform.com.edgesuite.net/Tech_Summit_2014_-_UX/579/347/LA_loop_789_hls_video_1280x0_2528000_primary_audio_x65aa639e369943dc84c02316be8d50d7_00001.ts
#EXTINF:10,
http://pmd205470tn.download.theplatform.com.edgesuite.net/Tech_Summit_2014_-_UX/579/347/LA_loop_789_hls_video_1280x0_2528000_primary_audio_x65aa639e369943dc84c02316be8d50d7_00002.ts
#EXT-X-ENDLIST

So this shows a pre-roll ad, then some content, and then end scene.

But what's missing here is metadata around the impressions and beacons to fire while the ad is playing. What's also missing is the ability to include click-throughs and companions.

There are solutions to this where you would stamp an ID3 tag on the first ad segment, which would give a client an ad ID, and then you would make a HTTP call from the client to fetch additional ad information.

But we want to avoid this. First, we can't presume that these ads have been pre-processed with ID3 tags, especially if you're dealing with third-party ad systems or programmatic inventory. Second, we want to minimize traffic from the client, so we want all the necessary metadata to be in the M3U8 itself.

So when we return this playlist, we actually return this:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:11
#EXT-X-MEDIA-SEQUENCE:1
#EXT-X-VAST-IMPRESSION:http://demo.theplatform.com/summit/2015/ads/tracking/citizen-drive/impression
#EXT-X-VAST-COMPANION:WIDTH=468,HEIGHT=60
#EXT-X-VAST-COMPANION-STATIC-RESOURCE:CREATIVETYPE=image/jpeg,URL=http://demo.theplatform.com/summit/2015/ads/banners/citizen_468x60.jpg
#EXT-X-VAST-COMPANION-CLICK-THROUGH:http://www.citizenwatch.com/
#EXT-X-VAST-COMPANION-CLICK-TRACKING:http://demo.theplatform.com/summit/2015/ads/tracking/citizen-drive/bannerclick
#EXT-X-VAST-VIDEO-CLICK-THROUGH:http://www.citizenwatch.com/
#EXT-X-VAST-VIDEO-CLICK-TRACKING:http://demo.theplatform.com/summit/2015/ads/tracking/citizen-drive/videoclick
#EXT-X-VAST-VIDEO-TRACKING:EVENT=start,URL=http://demo.theplatform.com/summit/2015/ads/tracking/citizen-drive/start
#EXT-X-VAST-VIDEO-TRACKING:EVENT=firstQuartile,URL=http://demo.theplatform.com/summit/2015/ads/tracking/citizen-drive/firstQuartile
#EXT-X-VAST-VIDEO-TRACKING:EVENT=midPoint,URL=http://demo.theplatform.com/summit/2015/ads/tracking/citizen-drive/midPoint
#EXT-X-VAST-VIDEO-TRACKING:EVENT=thirdQuartile,URL=http://demo.theplatform.com/summit/2015/ads/tracking/citizen-drive/thirdQuartile
#EXT-X-VAST-VIDEO-TRACKING:EVENT=complete,URL=http://demo.theplatform.com/summit/2015/ads/tracking/citizen-drive/complete
#EXTINF:10,
http://pmd205470tn.download.theplatform.com.edgesuite.net/Tech_Summit_2014_-_UX/164/731/citizen_401750083977_m3u8_video_1280x0_2432000_primary_audio_6_x4ac72b8f84f244eca9508012dfdbde58_00001.ts
#EXTINF:5.6,
http://pmd205470tn.download.theplatform.com.edgesuite.net/Tech_Summit_2014_-_UX/164/731/citizen_401750083977_m3u8_video_1280x0_2432000_primary_audio_6_x4ac72b8f84f244eca9508012dfdbde58_00002.ts
#EXT-X-DISCONTINUITY
#EXTINF:10,
http://pmd205470tn.download.theplatform.com.edgesuite.net/Tech_Summit_2014_-_UX/579/347/LA_loop_789_hls_video_1280x0_2528000_primary_audio_x65aa639e369943dc84c02316be8d50d7_00001.ts
#EXTINF:10,
http://pmd205470tn.download.theplatform.com.edgesuite.net/Tech_Summit_2014_-_UX/579/347/LA_loop_789_hls_video_1280x0_2528000_primary_audio_x65aa639e369943dc84c02316be8d50d7_00002.ts
#EXT-X-ENDLIST

Note the "EXT-X-VAST" extensions. So now everything we need is in the M3U8 itself.

The proposal above is to expose this extra metadata in ExoPlayer, so we can parse it for a richer client experience while doing server-side ad insertion. We can have a pull request ready in a few days...

@jlacivita
Copy link

hi @ojw28. I work with @alanra & @Skrewtape as well. Do you have any thoughts on the pull request?

We're going to be using Exo for lots of high profile projects, and it'd be great if we can all stay on the same codebase, to benefit from each other's work. I think this feature will actually enable a number of other HLS manipulators in the market, not just ours.

@ojw28
Copy link
Contributor

ojw28 commented May 9, 2016

I replied to the pull request. No particular objection to providing a way to do this, but not in the way the PR is currently structured.

@ojw28
Copy link
Contributor

ojw28 commented Jun 9, 2016

I believe your requirements were addressed by the patch you sent; so closing this one. Thanks!

@ojw28 ojw28 closed this as completed Jun 9, 2016
@google google locked and limited conversation to collaborators Jun 28, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants