feat: add support for deserialized response #1625

alvin-reyes · 2023-08-15T09:09:08Z

Changes

support for deserialized response based on the requirement #1531

Verification

Added the --serve-gateway on the `entrypoint.sh` and deployed to local devnet

Set value to `all`

echo Starting booster-http...
exec booster-http run --serve-gateway=all --api-lid=$LID_API_INFO  --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO --tracing

2023-08-20 19:15:01 2023-08-20T10:15:01.190Z    INFO    booster lib/api.go:38   Using full node API at ws://lotus:1234/rpc/v1
2023-08-20 19:15:01 2023-08-20T10:15:01.193Z    INFO    booster booster-http/run.go:156 Tracing exporter enabled
2023-08-20 19:15:01 2023-08-20T10:15:01.193Z    INFO    booster lib/api.go:64   Using storage API at ws://lotus-miner:2345/rpc/v0
2023-08-20 19:15:01 2023-08-20T10:15:01.194Z    INFO    booster lib/api.go:106  Miner address: t01000
2023-08-20 19:15:18 2023-08-20T10:15:18.354Z    INFO    booster booster-http/run.go:217 Starting booster-http node on listen address 0.0.0.0 and port 7777 with base path ''
2023-08-20 19:15:18 2023-08-20T10:15:18.355Z    INFO    booster booster-http/run.go:224 serving IPFS gateway at /ipfs/ (serving files)
2023-08-20 19:15:18 2023-08-20T10:15:18.355Z    INFO    booster booster-http/run.go:226 serving raw pieces at /piece/

Set value to `verifiable`

echo Starting booster-http...
exec booster-http run --serve-gateway=verifiable --api-lid=$LID_API_INFO  --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO --tracing

2023-08-21 19:58:36 http://boost:8042
2023-08-21 19:58:36 Starting booster-http...
2023-08-21 19:58:36 2023-08-21T10:58:36.942Z    INFO    booster lib/api.go:38   Using full node API at ws://lotus:1234/rpc/v1
2023-08-21 19:58:36 2023-08-21T10:58:36.950Z    INFO    booster booster-http/run.go:155 Tracing exporter enabled
2023-08-21 19:58:36 2023-08-21T10:58:36.950Z    INFO    booster lib/api.go:64   Using storage API at ws://lotus-miner:2345/rpc/v0
2023-08-21 19:58:36 2023-08-21T10:58:36.951Z    INFO    booster lib/api.go:106  Miner address: t01000
2023-08-21 19:58:46 2023-08-21T10:58:46.378Z    INFO    booster booster-http/run.go:216 Starting booster-http node on listen address 0.0.0.0 and port 7777 with base path ''
2023-08-21 19:58:46 2023-08-21T10:58:46.379Z    INFO    booster booster-http/run.go:223 serving IPFS gateway at /ipfs/ (serving blocks, CARs)
2023-08-21 19:58:46 2023-08-21T10:58:46.379Z    INFO    booster booster-http/run.go:225 serving raw pieces at /piece/

set value to `none`

echo Starting booster-http...
exec booster-http run --serve-gateway=none --api-lid=$LID_API_INFO  --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO --tracing

2023-08-21 19:58:36 2023-08-21T10:58:36.942Z    INFO    booster lib/api.go:38   Using full node API at ws://lotus:1234/rpc/v1
2023-08-21 19:58:36 2023-08-21T10:58:36.950Z    INFO    booster booster-http/run.go:155 Tracing exporter enabled
2023-08-21 19:58:36 2023-08-21T10:58:36.950Z    INFO    booster lib/api.go:64   Using storage API at ws://lotus-miner:2345/rpc/v0
2023-08-21 19:58:36 2023-08-21T10:58:36.951Z    INFO    booster lib/api.go:106  Miner address: t01000
2023-08-21 19:58:46 2023-08-21T10:58:46.378Z    INFO    booster booster-http/run.go:216 Starting booster-http node on listen address 0.0.0.0 and port 7777 with base path ''
2023-08-21 19:58:46 2023-08-21T10:58:46.379Z    INFO    booster booster-http/run.go:223 serving IPFS gateway at /ipfs/ (serving blocks, CARs)
2023-08-21 19:58:46 2023-08-21T10:58:46.379Z    INFO    booster booster-http/run.go:225 serving raw pieces at /piece/

Running booster-http after making a deal

made a online unverified deal for a simple car file

boost -vv deal --verified=false --provider=t01000 --commp=baga6ea4seaqdcoodqrjm65fkx5bstoh5aqe57zx7rbulpnfyp477b6d4vtyespq --car-size=104 --piece-size=128 --http-url=https://radiant.edge.estuary.tech/gw/bafybeieo26tuorkzzh5p347ucintyxgaqichmga3pv4euinokvyvemj5ya --payload-cid=bafkqagcineqee33pon2cyid3gexc4njxpuqhi2lnmvzqu --storage-price=1000000000

2023-08-22T08:19:14.983Z        DEBUG   boost   boost/deal_cmd.go:279   about to submit deal proposal   {"uuid": "d011c703-7bca-4a43-8629-0f40c737a283"}
sent deal proposal
  deal uuid: d011c703-7bca-4a43-8629-0f40c737a283
  storage provider: t01000
  client wallet: t3quyq27bdclsakjo5usrhejle3wch46fbcysvtk2vy5c2gwkqmtgco6jgeerxbjqot7v5o5rnnoan57b75pxa
  payload cid: bafkqagcineqee33pon2cyid3gexc4njxpuqhi2lnmvzqu
  url: https://radiant.edge.estuary.tech/gw/bafybeieo26tuorkzzh5p347ucintyxgaqichmga3pv4euinokvyvemj5ya
  commp: baga6ea4seaqdcoodqrjm65fkx5bstoh5aqe57zx7rbulpnfyp477b6d4vtyespq
  start epoch: 6096
  end epoch: 524496
  provider collateral: 0

Ready to publish deal

Configure booster-http to set serve-gateway=all

Look by CID

Configure booster-http to set serve-gateway=verifiable

Configure booster-http to set serve-gateway=none

into deserialized-resp

dirkmc

Looks good so far 👍
I left a couple of suggestions for code changes and descriptions.
Please run a few tests with these new parameters and paste the results in the PR comments.

cmd/booster-http/run.go

cmd/booster-http/gateway_handler.go

dirkmc · 2023-08-17T10:51:03Z

One nit: I'd also suggest using a code formatting tool for consistent whitespace

cmd/booster-http/run.go

cmd/booster-http/gateway_handler.go

dirkmc · 2023-08-18T09:07:26Z

Before the next review, please run a few tests with these new parameters and paste the results in the PR comments.

Co-authored-by: dirkmc <dirkmdev@gmail.com>

cmd/booster-http/run.go

cmd/booster-http/gateway_handler.go

cmd/booster-http/run.go

dirkmc · 2023-08-21T07:22:09Z

Before the next review, please manually run booster-http with the serve-gateway parameter set to each possible value, and paste the results in the PR comments.

alvin-reyes · 2023-08-21T11:08:32Z

@dirkmc I added the verification on the PR description above - link here #1625 (comment)

dirkmc · 2023-08-21T13:40:46Z

Thanks @alvin-reyes

Could you please also test out the functionality itself. So for each of all / verifiable / none try to retrieve a cid and verify that booster-http gives the appropriate response. So for all it should allow you to retrieve any cid. For verifiable it should only allow retrieval of verifiable data (it should return an error for unverifiable retrieval requests). For none it should return an error message when you try to retrieve any block.

You can read more about the exact behaviour in the issue description and in the Enable serving of files of the booster-http documentation.

alvin-reyes · 2023-08-22T18:13:05Z

I have added the verification on the PR description (sent you a message as well @dirkmc).

LexLuthr · 2023-12-22T08:14:49Z

Frisbii takes care of this in booster-http.

alvin-reyes and others added 4 commits August 15, 2023 18:07

feat: add support for deserialized response

8d77705

Merge branch 'filecoin-project:main' into deserialized-resp

921c0e5

chore: change the condition based to serve car/raw

ff617ed

Merge branch 'deserialized-resp' of https://github.com/alvin-reyes/boost

0a2abe4

into deserialized-resp

alvin-reyes marked this pull request as ready for review August 17, 2023 05:49

use http opts to indicate serve gateway

6ccc527

alvin-reyes requested review from dirkmc and nonsense and removed request for dirkmc August 17, 2023 06:59

dirkmc reviewed Aug 17, 2023

View reviewed changes

cmd/booster-http/run.go Outdated Show resolved Hide resolved

cmd/booster-http/gateway_handler.go Outdated Show resolved Hide resolved

modified based on review / feedback

7788269

alvin-reyes requested a review from dirkmc August 18, 2023 08:33

dirkmc reviewed Aug 18, 2023

View reviewed changes

cmd/booster-http/run.go Outdated Show resolved Hide resolved

cmd/booster-http/gateway_handler.go Outdated Show resolved Hide resolved

cmd/booster-http/gateway_handler.go Show resolved Hide resolved

Alvin Reyes and others added 5 commits August 18, 2023 05:46

Update cmd/booster-http/gateway_handler.go

c5a558d

Co-authored-by: dirkmc <dirkmdev@gmail.com>

remove serve-files on the docker booster-http entrypoint.sh

31bb7ad

change the flag to check if the ipfs gateway should be enabled

23f80aa

resolve lint issues

a6ee008

resolve lint issues

b39569a

alvin-reyes requested a review from dirkmc August 20, 2023 10:26

dirkmc reviewed Aug 21, 2023

View reviewed changes

cmd/booster-http/run.go Outdated Show resolved Hide resolved

cmd/booster-http/gateway_handler.go Show resolved Hide resolved

cmd/booster-http/run.go Outdated Show resolved Hide resolved

return error message when invalid serve-gateway is set

f310877

alvin-reyes requested a review from dirkmc August 21, 2023 11:08

LexLuthr closed this Dec 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for deserialized response #1625

feat: add support for deserialized response #1625

alvin-reyes commented Aug 15, 2023 •

edited

Loading

dirkmc left a comment

dirkmc commented Aug 17, 2023

dirkmc commented Aug 18, 2023

dirkmc commented Aug 21, 2023

alvin-reyes commented Aug 21, 2023

dirkmc commented Aug 21, 2023 •

edited

Loading

alvin-reyes commented Aug 22, 2023

LexLuthr commented Dec 22, 2023

feat: add support for deserialized response #1625

feat: add support for deserialized response #1625

Conversation

alvin-reyes commented Aug 15, 2023 • edited Loading

Changes

Verification

Added the --serve-gateway on the entrypoint.sh and deployed to local devnet

Set value to all

Set value to verifiable

set value to none

Running booster-http after making a deal

made a online unverified deal for a simple car file

Ready to publish deal

Configure booster-http to set serve-gateway=all

Look by CID

Configure booster-http to set serve-gateway=verifiable

Configure booster-http to set serve-gateway=none

dirkmc left a comment

Choose a reason for hiding this comment

dirkmc commented Aug 17, 2023

dirkmc commented Aug 18, 2023

dirkmc commented Aug 21, 2023

alvin-reyes commented Aug 21, 2023

dirkmc commented Aug 21, 2023 • edited Loading

alvin-reyes commented Aug 22, 2023

LexLuthr commented Dec 22, 2023

alvin-reyes commented Aug 15, 2023 •

edited

Loading

Added the --serve-gateway on the `entrypoint.sh` and deployed to local devnet

Set value to `all`

Set value to `verifiable`

set value to `none`

dirkmc commented Aug 21, 2023 •

edited

Loading