IPFS as a shared block store for federated indexer nodes

[gammazero]

To avoid having multiple indexer nodes all downloading the same content from content publishers, one indexer can download the content and the other indexers can get that content from a shared block store. Would IPFS make a useful shared block store?

Also, if index content is generally available via IPFS then any indexers can retrieve the content without necessarily downloading it directly from the publisher each time.

[willscott]

what do you mean by 'IPFS' here :) i don't know if we want to run a separate go-ipfs process, but if we have a blockstore and bitswap or graphsync running on the existing libp2p host we'll have the same effect

[gammazero]

By IPFS, I mean using a separate go-ipfs process. The purpose is that the index content becomes discoverable over IPFS and is a more general way to distribute index content, whether to individual indexers or to sharded/replicated federations.

It is not a design plan, but an idea to discuss at this point here in the discussions area. We should compare the two approaches and see what makes most sense short-term and long-term. I would prefer whatever is the lightest weight for publishers, and most flexible for indexer implementers.

[aschmahmann]

How well suited is your data for parallel fetching from multiple peers? Are your graphs very deep with individually small blocks or are there good branching points?

Without work like lightning storm (i.e. untrusted manifest + bitswap) or an application specific downloading protocol downloading deep DAGs from multiple peers == 😢.

[willscott]

it should be okay - we first pull the chain of metadata advertisements. this is deep, but small. something like graphsync from one peer may be fine for it.

then in parallel we can get all of the actual advertisment data.

[coryschwartz]

So this would mean writing a go-bs-ipfs or something similar I suppose? I could imagine that might be useful in other ways as well.

[willscott]

I think we could potentially use the IPFS api (ipfs add/pin, etc) of an IFPS daemon, rather than needing a blockstore abstraction, but yeah. We probably could build a blockstore over a remote IPFS node.

[gammazero]

When processing advertisements, then advertisements and their multihash entry blocks are saved to IPFS in-line with the processing of each advertisement. This is not done asynchronously to avoid building up a large amount of pending data waiting on IPFS during ingestion spikes, and it is OK for ingestion to lag significantly behind the announcement of a new advertisement.

IPFS storage cannot grow indefinitely and older index content needs to be removed to maintain storage limits. The indexer will maintain timestamps for the advertisement CIDs that are saved to IPFS. When it is necessary to reduce storage size, the oldest advertisement and all entries for that advertisement will be unpinned, and IPFS GC will be run.

Need to understand:

• Expected IPFS throughput
• IPFS storage requirements
• Use of IPFS cluster