suspending and notifying resumption of a download for a media element fetching from a MediaSource

[wolenetz]

Migrated from w3c bugzilla tracker. For history prior to migration, please see:
https://www.w3.org/Bugs/Public/show_bug.cgi?id=28573

[wolenetz]

We need to understand the original issue better: is this something that becomes solved if we ignore the preload attribute for MSE (see issue 11).

[wolenetz]

Let's get #11 figured out first before we tackle this. For instance, #11's fix might fix this. Marking blocked accordingly.

[paulbrucecotton]

@wolenetz: Are you planning to produce a PR for this issue or should one of the other editors help here?

[wolenetz]

I'm trying to understand still (now that #11 is figured out) if any work is necessary in V1 here.
Below is a copy of the original bugzilla issue, with my comments inline (in the context of #11 now causing MSE to ignore any preload attribute on the parent media element):

"User agents may decide to not download more content at any time" by
suspending the fetch, setting networkState to NETWORK_IDLE. The decision re
when the user agent stops buffering may depend on the value of the preload
attribute. Also "User agents may allow users to selectively block or slow
media data downloads."

In MSE, the web app, not the UA, determines when appendBuffer()/appendStream() are called. 'Text in the resource fetch algorithm that refers to "the download" or "bytes received" refer to data passed in via appendBuffer() and appendStream().'

We could add some clarifying MSE spec text that references to transitions to NETWORK_IDLE and firing 'suspend' events, along with the entirety of the paragraph in the normatively referenced HTML5 spec for media-elements that begins "User agents may decide to not download more content at any time" is not involved in a media element that has a MediaSource attached, since the web app has control over this decision (and indeed can explicitly remove() buffered media or delay a repeated append if it receives QuotaExceededError on attempted appends (or an 'error' event fired at the SourceBuffer in the case of eviction failure in the stream append loop used by appendStream()) that fail to evict enough coded frames to make room for the new append).

A media element without a MediaSource receives media data only when the user
agent is looking for media data, but when the fetch involves a MediaSource,
the media element receives data whenever appendBuffer() or appendStream() is
called on a SourceBuffer. The only specified mechanism for rejecting an append is
if the buffer full flag is true. Setting the buffer full flag to true is
explicitly specified only after parsing a complete media segment header in the
segment parser loop algorithm when append state equals PARSING_MEDIA_SEGMENT
(i.e. while parsing an appended array that was not rejected because of the
buffer full flag).

Though buffer full flag might be set true during the segment parser loop, it can be cleared if coded frame eviction on the next appendBuffer or iteration of the stream append loop succeeds in freeing enough room in the buffer.

Is the expected behavior of the user agent one of these?:

(i) continues to accept media data after suspending the download, but does not
dispatch progress events for this data.
(ii) accepts appended media data after suspending the download until it has a
chance to set the buffer full flag to true, and then rejects subsequent
arrays.
(iii) sets the buffer full flag immediately when suspending the download and
rejects future appends.
(iv) does not suspend a download which has begun until either all the
SourceBuffers are all full or it has all of the media data
(i.e. endOfStream has been called). a "metadata" value for the
preload attribute is ignored. A stalled event may be queued even
if the user agent does not require more media data.

None of the above, IMO. 'suspending the download' is a 'may', and in the case of MSE, is not done by the UA. Rather, the MSE append*() operations define clearly when an append is either rejected or completed.

The NETWORK_IDLE/NETWORK_LOADING distinction seems useful for a user agent to
indicate to a player when it wants more media data or not. However, what is
missing is a specific event to notify on transition to NETWORK_LOADING. (The
"suspend" event exists for transition to NETWORK_IDLE.) "playing" and
"timeupdate" may provide some hints.

In the prepare append algorithm, if the readyState is "ended", and so is
changed to "open", and the buffer is not full, I assume the media element
networkState should be changed from NETWORK_IDLE to NETWORK_LOADING?
In that situation, there is a sourceopen event to notify of the state change
from "ended", but MediaSource readyState doesn't have a distinct value for a
suspended incomplete download.

Similarly, there is no specific event to notify a player when a SourceBuffer
transitions from a state where eviction will not unset the buffer full flag to
one where an append can succeed.

Initial transition to NETWORK_LOADING is 'loadstart'. And in MSE attachment case, 'sourceopen' is the "attachment success" signal for the web app to proceed with any addSourceBuffer() and appends as it desires.

If I understand correctly, what is really being asked for here is an event and associated web-app-visible state that it can query which gives the web app the signal that "buffer full flag" is either not currently set or it might be clearable if the app wants to append some more. Note that this seems similar to #40, which is also currently VNext, not V1.

Net: In V1, (from above): We could add some clarifying MSE spec text that references to transitions to NETWORK_IDLE and firing 'suspend' events, along with the entirety of the paragraph in the normatively referenced HTML5 spec for media-elements that begins "User agents may decide to not download more content at any time" is not involved in a media element that has a MediaSource attached, since the web app has control over this decision (and indeed can explicitly remove() buffered media or delay a repeated append if it receives QuotaExceededError on attempted appends (or an 'error' event fired at the SourceBuffer in the case of eviction failure in the stream append loop used by appendStream()) that fail to evict enough coded frames to make room for the new append).

However, this is already a may in HTML spec, so this bug seems at most V1NonBlocking (for the MSE spec text clarification) and definitely VNext for the definition of some new API for signalling when an app might expect success for an append (with #40 also in VNext).

@jdsmith3000 do you agree with this analysis and re-triage?

[wolenetz]

Also, @karlt: are you the original bug filer for https://www.w3.org/Bugs/Public/show_bug.cgi?id=28573 ?

[karlt]

Replaced with #24 (comment) so that quotes are not elided.

[karlt]

One consequence of the removal of the optional steps of the resource fetch algorithm that are sometimes associated with preload=none (Issue #11), and proposed removal of the UA's option to suspend would be that the load event is always blocked until metadata has been appended to the MediaSource. I don't think that would be desirable, and many of the web platform tests are not expecting this.

[karlt]

Thanks, wolentz for CC'ing me on this. I am the original bug filer.

I think the key issue to here for V1 is specifying when suspend
events are fired and when the NETWORK_IDLE state is used.
However, consideration of the buffer full flag may influence
decisions.

karlt wrote:

"User agents may decide to not download more content at any time" by
suspending the fetch, setting networkState to NETWORK_IDLE.

Also "User agents may allow users to selectively block or slow
media data downloads."

wolentz writes:

In MSE, the web app, not the UA, determines when
appendBuffer()/appendStream() are called.

Yes, but the UA decides whether the append succeeds.

We could add some clarifying MSE spec text that references to
transitions to NETWORK_IDLE and firing 'suspend' events, along
with the entirety of the paragraph in the normatively referenced
HTML5 spec for media-elements that begins "User agents may
decide to not download more content at any time" is not involved
in a media element that has a MediaSource attached, since the
web app has control over this decision (and indeed can
explicitly remove() buffered media or delay a repeated append if
it receives QuotaExceededError on attempted appends (or an
'error' event fired at the SourceBuffer in the case of eviction
failure in the stream append loop used by appendStream()) that
fail to evict enough coded frames to make room for the new
append).

If the intention is that NETWORK_IDLE and suspend events are never
used when the URL identifies a MediaSource, then I think that does
need clarifying.

The option of QuotaExceedError on attempted append does allow the
UA to block download of more content, but the "at any time" part
is less clear.

Are "stalled" events also suppressed?
What about "progress"?
Are any other events affected?

My impression is that the intention is that playing UI can
interact only with the media element, ignoring the nature of the
source. That would mean that the UI still needs events about
changes in download progress.

Is the expected behavior of the user agent one of these?:

(iv) does not suspend a download which has begun until either all the
SourceBuffers are all full or it has all of the media data
(i.e. endOfStream has been called). a "metadata" value for the
preload attribute is ignored. A stalled event may be queued even
if the user agent does not require more media data.

None of the above, IMO. 'suspending the download' is a 'may',
and in the case of MSE, is not done by the UA.

A "may" implies that the possibility is still there.
Clarification is needed if this cannot happen when the URL
identifies a MediaSource.

When you exclude option (iv), which parts are not correct?

After "Notify the media element that it now has all of the media
data", the NETWORK_IDLE state and suspend event are not optional.

"Once the entire media resource has been fetched (but potentially
before any of it has been decoded)

Fire a simple event named progress at the media element.

Set the networkState to NETWORK_IDLE and fire a simple event
named suspend at the media element."

Rather, the MSE append*() operations define clearly when an
append is either rejected or completed.

"SourceBuffer is full" is determined by the UA, I assume.

The NETWORK_IDLE/NETWORK_LOADING distinction seems useful
for a user agent to indicate to a player when it wants more
media data or not. However, what is missing is a specific
event to notify on transition to NETWORK_LOADING. (The
"suspend" event exists for transition to NETWORK_IDLE.)
"playing" and "timeupdate" may provide some hints.

In the prepare append algorithm, if the readyState is
"ended", and so is changed to "open", and the buffer is not
full, I assume the media element networkState should be
changed from NETWORK_IDLE to NETWORK_LOADING?  In that
situation, there is a sourceopen event to notify of the
state change from "ended", but MediaSource readyState
doesn't have a distinct value for a suspended incomplete
download.

Similarly, there is no specific event to notify a player
when a SourceBuffer transitions from a state where eviction
will not unset the buffer full flag to one where an append
can succeed.

Initial transition to NETWORK_LOADING is 'loadstart'. And in MSE
attachment case, 'sourceopen' is the "attachment success" signal
for the web app to proceed with any addSourceBuffer() and
appends as it desires.

If I understand correctly, what is really being asked for here
is an event and associated web-app-visible state that it can
query which gives the web app the signal that "buffer full flag"
is either not currently set or it might be clearable if the app
wants to append some more.

Yes, I think that is required. The immediate issue is
specification of when to change networkState and dispatch suspend
events, but consideration of this requirement may influence
decisions re networkState and suspend.

Note that this seems similar to #40,
which is also currently VNext, not V1.

Yes, "buffer full flag" change notification is similar. Issue #40
is about when more data is /needed/. "buffer full flag" is more
about when space for more data is /available/. I'm not clear which
of those corresponds to networkState.

[wolenetz]

Marking V1 until and unless we can determine if this VNext (or fix it in V1).

[wolenetz]

@jdsmith3000 I'm looking for other UA input here. Please reference the previous discussion and provide input on this issue.

[wolenetz]

@karit, in #24 (comment):

One consequence of the removal of the optional steps of the
resource fetch algorithm that are sometimes associated with
preload=none (Issue #11), and proposed removal of the UA's option
to suspend would be that the load event is always blocked until
metadata has been appended to the MediaSource. I don't think that
would be desirable, and many of the web platform tests are not
expecting this.

Which 'load' event are you referring to? Even with MSE attachment, the 'loadstart' event should be queued without any delay in the resource selection algoirthm. The 'loadedmetadata' event should be queued once reaching HAVE_METADATA. And the 'loadeddata' should be queued the first time once reaching at least HAVE_CURRENT_DATA.

[karlt]

On Fri, 20 May 2016 12:17:11 -0700, wolenetz wrote:

@karlt, in #24 (comment):

One consequence of the removal of the optional steps of the
resource fetch algorithm that are sometimes associated with
preload=none (Issue #11), and proposed removal of the UA's option
to suspend would be that the load event is always blocked until
metadata has been appended to the MediaSource. I don't think that
would be desirable, and many of the web platform tests are not
expecting this.

Which 'load' event are you referring to? Even with MSE attachment, the
loadstart' event should be queued without any delay in the resource selection
algoirthm. The 'loadedmetadata' event should be queued once reaching
HAVE_METADATA. And the 'loadeddata' should be queued the first time once
reaching at least HAVE_CURRENT_DATA.

Here, I'm referring to the document load event, referenced at
https://html.spec.whatwg.org/multipage/embedded-content.html#delaying-the-load-event-flag

The delaying-the-load-event flag is set to true during the
resource selection algorithm
https://html.spec.whatwg.org/multipage/embedded-content.html#concept-media-load-algorithm

If it cannot be set to false "if the user agent intends to not
attempt to fetch the resource until the user requests it
explicitly" nor when "a user agent decides to completely suspend a
download", then it would only be set to false "Once the readyState
attribute reaches HAVE_CURRENT_DATA".

i.e. the document load event would be delayed until the metadata
(and perhaps also the first video frame) has been appended to the source
buffer.

Many web platform tests attach a MediaSource to a media element
before the load event is queued and then expect to receive the
load event even when metadata has not been appended to a source
buffer. The implication is that user agent implementations are
not delaying the load event. I don't know whether or not that is
because "the user agent intends to not attempt to
fetch the resource until the user requests it explicitly".

[wolenetz]

@karlt thanks for the detailed explanation around your wpt-related concern that, with MSE ignoring the optional 'preload=none' steps around 'suspend' or otherwise provide some 'suspend' logic in MSE context, that the delaying-the-load-event flag could remain true until the media element reaches HAVE_CURRENT_DATA.

I see in Chromium at least, that the delaying-the-load-event-flag is set false when the 'stalled' event is queued. This accomodates MSE apps as well as regular src= whose loading has become stalled (which in both the MSE and src= cases can be detected by either lack of buffered range changes or bytes received over a period of time since the previous check for the 'progress' event when networkState is NETWORK_LOADING.)

Notably, this Chromium logic for 'stalled' is somewhat close to the normatively-referenced w3c HTML5's spec text (and informatively-referenced w3c HTML5.1 spec text), with a relaxation of 'decides to completely stall a download' to something more like "detects a stall and queues a 'stalled' event":

When a user agent decides to completely stall a download, e.g. if it is waiting until the user starts playback before downloading any further content, the element's delaying-the-load-event-flag must be set to false. This stops delaying the load event.

Interestingly, this spec text doesn't say "When a user agent decides to suspend a download".

Would it help to clarify in the MSE text that user agents must set the delaying-the-load-event-flag false when an HTMLMediaElement with MediaSource attached queues a 'stalled' event? This would align at least with Chromium's current implementation.

[karlt]

wolenetz writes:

I see in
Chromium
at least, that the delaying-the-load-event-flag is set false
when the stalled' event is queued. This accomodates MSE apps as
well as regular src= whose loading has become stalled (which in
both the MSE and src= cases can be detected by lack of buffered
range changes over a period of time since the previous check for
the 'progress' event when networkState is NETWORK_LOADING.)

Thanks. This detail is helpful.

Notably, this Chromium logic for 'stalled' is somewhat close to
the normatively-referenced w3c HTML5's spec text (and
informatively-referenced w3c HTML5.1 spec text), with a
relaxation of 'decides to completely stall a download' to
something more like "detects a stall and queues a 'stalled'
event":

On my first investigation, I thought that relaxation might be
appropriate, and copied Blink behaviour for
https:/https://bugzilla.mozilla.org/show_bug.cgi?id=975782

That change was later reverted for regular sources,
based on https://bugzilla.mozilla.org/show_bug.cgi?id=868866#c28

I agree with that comment and I don't think "stall" was
intentional wording in the spec because what it was describing was
distinct from the description of the stalled event. I filed
https://www.w3.org/Bugs/Public/show_bug.cgi?id=28392

We didn't get around to finding an appropriate solution for the
MSE case, but I filed this issue in the hope that it might start
to clarify this and other aspects.

When a user agent decides to completely stall a download,
e.g. if it is waiting until the user starts playback before
downloading any further content, the element's
delaying-the-load-event-flag must be set to false. This stops
delaying the load event.

Interestingly, this spec text doesn't say "When a user agent decides to
suspend a download".

Yes, that changed due to
https://www.w3.org/Bugs/Public/show_bug.cgi?id=28392

Would it help to clarify in the MSE text that user agents must set the
delaying-the-load-event-flag false when an HTMLMediaElement with MediaSource
attached queues a 'stalled' event? This would align at least with Chromium's
current implementation.

IMHO it seems a bit pointless to wait for the stalled event before
resetting the delaying-the-load-event flag, but I don't feel
familiar enough with all aspects of the specs to make a
recommendation.

However, delaying the load event is not desirable (IMO) and this
is the way that Blink and the Gecko copy allow the load event to
proceed, and so it would clarify things to spell this out in one
spec or another.

In the HTML spec, suspend events correspond to the media element
no longer wanting more data and stalled events to the element no
longer getting more data when it wants the data. MSE adds more
levels because there may be no attempt to provide the data when
the element wants it and an attempt to append data may occur when
the media element doesn't want more data. The latter is what
leads to the original question about suspend and buffer-full with
the four conceivable possibilities listed.

[wolenetz]

Proposed solution from today's editors' sync:

1. add some clarifying MSE spec text that 'suspend' media elements do not apply to MediaSource-attached HTMLMediaElements. References in HTML5 media element to transitions to NETWORK_IDLE and firing 'suspend' events, and the HTML5 media element paragraph that begins "User agents may decide to not download more content at any time" is not involved in a media element that has a MediaSource attached. This is because the web app has control over fetching decisions (and indeed can explicitly remove() buffered media or delay a repeated append if it receives QuotaExceededError on attempted appends (or an 'error' event fired at the SourceBuffer in the case of eviction failure in the stream append loop used by appendStream()) that fail to evict enough coded frames to make room for the new append).

2. add some clarifying MSE spec text that the 'stalled' event enqueueing by HTMLMediaElement, if MSE is attached, also sets the delaying-the-load-event-flag to false.

We even considered perhaps setting the delaying-the-load-event-flag false immediately upon MediaSource attachment to the HTMLMediaElement, but I'm not entirely certain that is correct. Chrome (and I think FF) already implement (2).

Considering the scope of the agreed-upon spec changes, we retriaged this to V1NonBlocking today.

[wolenetz]

We discussed further during today's editors' sync, with @travisleithead looped in. We don't need (2) from #24 (comment) and instead it is better to (3) just set the delaying-the-load-event-flag to false immediately upon MediaSource attachment to the HTMLMediaElement. Marking V1 to implement (1) and (3) accordingly.