Skip to content
This repository has been archived by the owner on Feb 12, 2024. It is now read-only.

Gateway Improvements: Streaming, Conditional and Range Requests #1989

Merged
merged 3 commits into from
May 8, 2019

Conversation

lidel
Copy link
Member

@lidel lidel commented Apr 11, 2019

Part of an effort to run embedded js-ipfs in Brave 🦁 ipfs/ipfs-companion#716

This PR:

Added 2019-04-29:

  • Switched from deprecated hapi and joi to @hapi/hapi and @hapi/joi
  • Added support for Conditional Requests (RFC7232)
    • Returning 304 Not Modified if If-None-Match is a CID matching Etag
    • Added Last-Modified to /ipfs/ responses (improves client-side caching)
    • Always returning 304 Not Modified If-Modified-Since for immutable /ipfs/
  • Added support for Byte Range Requests (rfc7233#section-2.1)
  • Added support for ?filename= parameter (improves downloads of raw )

@ghost ghost assigned lidel Apr 11, 2019
@ghost ghost added the status/in-progress In progress label Apr 11, 2019
lidel added a commit to ipfs/ipfs-companion that referenced this pull request Apr 12, 2019
This applies cherry-picked patches from:
ipfs/js-ipfs#1989
ipfs/js-ipfs#1950
and solves stream issues on page refresh.

Content-type sniffing is now done over a meaningful amount of bytes
instead of arbitrary number.
Copy link
Member

@hugomrdias hugomrdias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lidel
Copy link
Member Author

lidel commented Apr 18, 2019

@alanshaw rebased this PR against latest master and CI no longer fails due to libp2p

@lidel lidel changed the title fix(gateway): streaming compressed payload Gateway Improvements: Streaming, Conditional and Range Requests Apr 26, 2019
@lidel lidel requested a review from hugomrdias April 26, 2019 11:01
@lidel
Copy link
Member Author

lidel commented Apr 26, 2019

As this PR was not merged and I had more improvements built on top of it, I've just pushed those additional improvements here and updated description to reflect what changed.

Note: We should review & merge this before we add support for /ipns/ to the Gateway, as there are caching nuances around mutable-vs-immutable addressed by this PR.

Question: If I wanted to add interop (go-vs-js) tests for relevant HTTP headers, which repo should I use?

lidel added a commit to ipfs/ipfs-companion that referenced this pull request Apr 26, 2019
Switching to js-ipfs from ipfs/js-ipfs#1989
for next beta release
lidel added 2 commits May 6, 2019 10:28
This change simplifies code responsible for streaming response
and makes the streaming actually work by telling the payload compression
stream to flush its content on every read().
(previous version was buffering entire thing in Hapi's compressor memory)

We also do content-type detection based on the beginning of the stream
by peeking at first `fileType.minimumBytes` bytes.

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
- Switched from deprecated `hapi` and `joi` to `@hapi/hapi` and `@hapi/joi`
- Added support for Conditional Requests (RFC7232)
  - Returning `304 Not Modified` if `If-None-Match` is a CID matching `Etag`
  - Added `Last-Modified` to `/ipfs/` responses (improves client-side caching)
  - Always returning `304 Not Modified`
    when `If-Modified-Since` is present for immutable `/ipfs/`
- Added support for Byte Range requests (RFC7233, Section-2.1)
- Added support for `?filename=` parameter (improves downloads of raw cids)

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
@lidel lidel force-pushed the fix/streaming-compressed-payload branch from 17712a4 to 373eedc Compare May 6, 2019 08:39
@lidel
Copy link
Member Author

lidel commented May 6, 2019

Rebased on top of #2015, decreasing the surface of this PR.
@alanshaw should be easier to review – mind taking a look ?

License: MIT
Signed-off-by: Alan Shaw <alan.shaw@protocol.ai>
@ghost ghost assigned alanshaw May 8, 2019
lidel added a commit to ipfs/ipfs-companion that referenced this pull request May 8, 2019
tl;dr opening /ipns/tr.wikipedia-on-ipfs.org/wiki/Mars.html works

Switched to js-ipfs with cherry-picked changes from:
ipfs/js-ipfs#2020
ipfs/js-ipfs#1989
ipfs/js-ipfs#2002
ipfs/js-ipfs-http-response#22
ipfs-inactive/js-ipfs-mfs#48
@alanshaw alanshaw merged commit 48a8e75 into master May 8, 2019
@alanshaw alanshaw deleted the fix/streaming-compressed-payload branch May 8, 2019 18:30
@ghost ghost removed the status/in-progress In progress label May 8, 2019
@alanshaw alanshaw mentioned this pull request May 9, 2019
44 tasks
alanshaw pushed a commit that referenced this pull request Jul 4, 2019
> Part of an effort to run embedded js-ipfs in Brave 🦁 ipfs/ipfs-companion#716
> Fixes #1918 

This PR will add support for `/ipns/` paths at HTTP Gateway.
Smoke test: [/ipns/tr.wikipedia-on-ipfs.org](http://127.0.0.1:9090/ipns/tr.wikipedia-on-ipfs.org/wiki/Anasayfa.html) (IPNS+DNSLink+HAMT-sharded website)
 
This PR depends on the following merged PRs:

  - Gateway Improvements from #1989
    (after merging #1989 I will rebase this PR, which will remove first two commits)
  - PeerID 
    eg. `/ipns/<PeerId-as-multihash-b58>`
    - requires #2002 to land first
  - `/ipns/<libp2p-key-in-cidv1>` 
    - requires multiformats/js-multicodec#45
  - DNSLink 
    eg. `/ipns/<fqdn>/path/file` like `/ipns/docs.ipfs.io/assets/logo.svg`
    - requires #2002 to land first
  - HAMT shard support 
    eg. `/ipns/tr.wikipedia-on-ipfs.org/wiki/Anasayfa.html` (`wiki` is a sharded directory)
    - requires ipfs/js-ipfs-http-response#22 and ipfs-inactive/js-ipfs-mfs#48  to land first
  - Tests for `/ipns/` 

License: MIT
Signed-off-by: Marcin Rataj <lidel@lidel.org>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants