Scope default indexing to only FIL+ deals with FastRetrieval enabled

[honghaoq]

Today network indexer has default configuration set to default announcement of index, while allowing SP to change config to false to not index. This was designed by the underlying assumption that most data are public unless specified otherwise.

As we embrace more and more enterprise use case and private data onboarding to the network, we are considering updating indexing announcement mechanism to better serve the network need. Instead of announcing index by default, we plan to instead ONLY announce index by default for FIL+ deals that has FastRetrieval enabled, i.e. the contents that are expected to be retrievable, and thus indexing these data by default is a requirement for retrieval. As for other contents, we plan to change to default NOT index, which respect private enterprise data better, and more inline with industry practice.

Specifically, we plan to check on the following two boolean fields in storage deal, and index by default when both of them are true. If either is false, it would be set to not index unless specified otherwise.

• FastRetrieval bool
• VerifiedDeal bool

Specifically, we plan to introduce an isPublic state for a deal in the next version, and do the following:

1. Setting isPublic state for existing deal: when existing deals with both of these flags above are set as true, then they are by default considered isPublic. Other deals are not.
2. SPs would be able to update and view the isPublic state of deals.
3. Going forward, isPublic state would also default to true if isVerified and FastRetrieval are both true.
4. For future new deals, we would allow clients to specify isPublic state in new deals, and in the interface.

Additional Notes:

1. When a deal state shows isPublic = false, it does NOT imply private access, to achieve full privacy handling, there needs to be evolution of the API to support a more complex ACL mechanism on top, e.g the ones that cid.gravity and lighthouse.storage are working on. Integrating with the ACL tools for data access is a separate work item that are in planning and would happen after this.
2. We plan to initially support isPublic in the deal proposal, which is not on chain. We will need to subsequently decide if we want to propose a FIP to include this state on chain as well for accountability to FIL+ notaries.

[honghaoq]

@dkkapur FYI

[dkkapur]

@honghaoq just to confirm - since this is a change in the storage market defaults/implementation, a FIP is not required right? Just an upgrade to markets/boost could support this change?

[honghaoq]

Probably dont need a FIP at this point, we can revisit later if we find out this data is needed onchain (when there is use cases justify accessing it onchain).

[flyworker]

It will be great if the ACL can support 3 method:

• IP based
• Wallet based
• username/password (optional)

[DLTX-Github]

Hasan from DLTx
From the way I understand how this should work, there has to be a flag on CID level that tells the indexer whether is should be pick up the deal and publish it or not. This can be turned on or off based on SP/client need.
The default value for this flag is debatable. For the good of the network and for the sake of not losing files stored, it should be defaulted to "yes". There is some level of privacy to the data as it's the CID that's being published and not additional info on the data being stored, and in addition, private data is expected to be encrypted before stored, so this further supports the default value of yes.
Im curious whether changing this value to no would eventually make it disappear from the indexer or not, as it is propagating through the network

[honghaoq]

Thanks Hasan, we are also hearing strong push from some SP storing private deals to make data not default published due to the ask of clients. If we default to no, it wont impact the historical stored index since we plan to only update it for future deals.

[TorfinnOlsen]

⚠️ This discussion is being relegated in favor of a committed plan to move forward represented by this github issue in the boost repo:

• Capability to skip deals for indexing: need config to mark content as not retrievable in market boost#689 (comment)

We perceive this as the simplest change we can make in order to enable SP's to set the announce status, without impacting retrieval processes. Future efforts will address the broader goal of an access list service.

⚡ tl;dr Request to add a flag to boost to allow storage clients to elect to not announce deal data to IPNI. Default behavior will be to announce just as it is today.

🕐 When is a deal announced to the indexer?

• When a deal is made, if AnnounceToIPNI=True or not set at all, CID’s announced to indexer.

• If AnnounceToIPNI=False, no announcement.

🔧 How will this be implemented?

• Currently, we are on Storage Deal Protocol v1.2 (/fil/storage/mk/1.2.0) where every deal is announced.

• This new proposal is for Storage Deal Protocol v1.2.1 (/fil/storage/mk/1.2.1)

• Boost will be updated to recognize and store as metadata this new flag.

🥡 How does this impact whether the data may be retrieved?

• It doesn’t. No impact. ACL’s must be set separately. This just relates to IPNI.

🛃 How can a client change the announce status?

• No change currently permitted. The decision is made at the time the deal is made.

👴 What about legacy deals?

• They stay announced as they are today.

🍴 Related Github Discussion

• Ability for SP/client to mark deal as private (i.e. not publicly retrievable) boost#728

👋 Relegated github issues/discussions being closed as a result of this update

• Scope default indexing to only FIL+ deals with FastRetrieval enabled #666

• Capability to skip deals for Indexing ipni/storetheindex#635

⌛ Timeline for implementation

• Current scope of effort is in the process of being analyzed and assessed by the boost team. Work not yet scheduled, but expected outcome is work completed early Q123.