Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing interface for uncompressed Content-Length when compression is used #1358

Open
MPurscheUnity opened this issue Nov 17, 2021 · 7 comments
Labels
addition/proposal New features or enhancements needs implementer interest Moving the issue forward requires implementers to express interest topic: api

Comments

@MPurscheUnity
Copy link

In order to track progress of a fetch GET request(in a similar way to onprogress events of XMLHttpRequest) it is possible to read the body of a request with a ReadableStream and retrieve the total size of the request body from Content-Length header.
However, if the request is compressed using the Content-Enconding header the Content-Length does not give the total uncompressed size of the request body but the size of the compressed request body.
It would be nice to have an interface to get the total size of the uncompressed body in order to properly track the progress of a request. There is the FetchObserver(#607) proposal but it doesn't seem to have gained any traction.

@annevk annevk added addition/proposal New features or enhancements needs implementer interest Moving the issue forward requires implementers to express interest topic: api labels Nov 17, 2021
@annevk
Copy link
Member

annevk commented Nov 17, 2021

We do keep track of this in "decoded body size" (which is only exposed through Resource Timing at the moment) but that's incremented whenever we decode a chunk. I don't think we actually know the full size upfront as you would need for a progress indicator.

@nornagon
Copy link

Would it be reasonable to expose the total number of pre-decompression bytes received so far? Something like response.body.receivedBytes, which would give the number of bytes received over the wire, which would be directly comparable to the Content-Length value.

As far as I know, it is not always possible to compute the expected total uncompressed payload size. With Content-Encoding: gzip it might be possible after receiving the gzip header, but it is not possible with Content-Encoding: deflate.

@annevk
Copy link
Member

annevk commented Aug 29, 2022

It seems reasonable in the abstract, but we'd need to figure out where to put that information. We cannot just stuff it on body I think, that's a ReadableStream and I'm not sure we'd want to create a subclass for this.

cc @domenic

@ricea
Copy link
Collaborator

ricea commented Aug 29, 2022

With Content-Encoding: gzip it might be possible after receiving the gzip header, but it is not possible with Content-Encoding: deflate.

Sadly the gzip header doesn't contain size information. The gzip footer does, but that's not useful for calculating progress.

@magcius
Copy link

magcius commented Sep 4, 2022

Putting the information on ReadableStream means it's not possible to use the convenience methods (e.g. json(), arrayBuffer(), blob()), as the body is locked to one reader. I'm not sure where the progress information should go, but ideally it would go somewhere that would work together with those convenience methods.

@jimmywarting
Copy link

it could be handy to get ahold of the resource timing when making a request.
performance.getEntries() only returns completed requests.

if we could get an incomplete one then we could figure this out by ourself and it would be easier to know which one belongs to the actual request that you have made.

fetch(url, {
  onTiming (resourceTiming) {
    // do things with it.
  }
})

Putting the information on ReadableStream means it's not possible to use the convenience methods (e.g. json(), arrayBuffer(), blob()), as the body is locked to one reader

you could always clone the response.

var clone = response.clone()
response.json().then(console.log)
clone.body.pipeTo(progressStream)

@annevk
Copy link
Member

annevk commented Sep 29, 2024

FWIW, I still think #607 is the way to go for exposing this and other information about an ongoing fetch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
addition/proposal New features or enhancements needs implementer interest Moving the issue forward requires implementers to express interest topic: api
Development

No branches or pull requests

6 participants