The one true API, the ipfsx upgrade path

[achingbrain]

Land #1670 🤯 🎊 😅
#2506 (comment)

But what does that mean though?

Something that bothers me is that we already support promises & callbacks in the API, so removing callbacks doesn't really give our users anything except a whole bunch of refactoring for those who do used the callbackified API. That can't be the end goal of #1670.

I think we should get on and implement ipfsx.

We've seen the API grow organically over time and it doesn't present the most streamlined interface. The key point is that IPFS is all about files, yet loads of the file operations are not top-level operations. Plus we have confusing caveat-filled weirdness with things like ipfs.ls and ipfs.files.ls.

The ipfsx approach is to finally unify the files API and move them all to the top level front and centre and have the low level stuff grouped by domain (e.g. block, dag, etc). It also removes readable streams, pull streams, etc giving us a smaller, more focused API surface area and (hopefully) smaller bundle size. I think we're all agreed that this is where we're heading.

As the internals follow the path of ipfs._addAsyncIterator we can start to implement ipfsx on top of them, then eventually split out the current API as a separate module to ease upgrading for people.

Step 1

Allow the user to expose the new API as an experimental flag. The old API will not be exposed - this may mean that some operations are impossible until they are implemented. Welcome to the bleeding edge.

const IPFS = require('ipfs')

const ipfs = await IPFS.create({
  EXPERIMENTAL: {
    ipfsx: true
  }
  // ... other options
})

// call the new API
await ipfs.mkdir('/somedir')

Step 2

Once the API is implemented, remove the old API entirely, deps and all. This should give us a leaner bundle and small surface area for those who have kept up to date.

Then create a wrapper for the new API that implements the old by calling new API methods and is callbackified/promisified/promise-nodified. This gives existing users a smooth upgrade path.

We should be able to verify it using today's interface-ipfs-core suite, though we should make tomorrow's version will only test the ipfsx interface.

const IPFS = require('ipfs')
const legacy = require('ipfs-legacy')

const ipfs = await legacy(IPFS).create({
  // ... other options
}))

// call the old API
await ipfs.files.mkdir('/somedir')

[achingbrain]

Or, more succinctly:

Am just finishing off async iterator functions for the remaining (non-mfs*) files api calls, then can start implementing the above if people agree with the approach.

* = these are async iterable already

[mikeal]

The current implementation of ipfsx is pulling in the raw codecs from IPLD. You should probably just move to the new Block API instead https://github.com/ipld/js-block

You may also want to consider returning/yielding Block instances instead of CID’s and encoded data. In both cases, the data and CID are computed lazily in the new Block API and cached indefinitely. This will avoid double-hashing and double-encoding problems, and also avoids these activities entirely if they are never requested.

[mikeal]

If you plan to continue to use the ipfsx code base I can send a PR that migrates to the Block API so you can see what it all looks like, I just won’t bother if this is actually going to be re-implemented when you integrate into js-ipfs.

[achingbrain]

It would probably be reimplemented using the API in ipfs-shipyard/ipfsx as a template.

[daviddias]

I'm not onboard with this. The hypothesis that giving yet another API as an option in addition to all the previous ones will benefit the users should not be assumed. I see it missing to recognize the amount of understanding necessary to even grok why these exist, why one is better than the other and why would a user need to care.

Creating a great API is like creating a great product, there needs to be a lot of "no" involved otherwise it is just a compromise soup. We've already agreed both in person (Berlin Dev Meetings and IPFS Camp + other hack weeks) and async (https://github.com/ipfs/interface-js-ipfs-core/issues/284) that the Files API needs to be refactored and that APIs such as Block & Object can either be deprecated or hidden.

If you want to explore a migrate path such as this (#2509), consider integrating in that migrate path something that clearly routes the users into a more happy place, rather than giving even more options and expecting them to differentiate well (specially when there are almost no docs).

My vote goes for having the new API in beta and call the old one legacy to make it evident that users should move away from it (or something that clearly points to the user what is new and good and what is old and should change).

[alanshaw]

My vote goes for having the new API in beta and call the old one legacy to make it evident that users should move away from it (or something that clearly points to the user what is new and good and what is old and should change).

IMHO, releasing under the EXPERIMENTAL flag would give us the licence to release an incomplete core API, that may have to change before it becomes the blessed core API.

I take your point on adding another API though, and on reflection I think you're correct that we're not clearly moving users towards a more happy place in this way. At least, it would require a significant effort on our part to comminucate that EXPERIMENTAL.ipfsx will become the core API and it will be easier to use, more performant and a bunch of other reasons.

The problem I'm trying to solve is that adding ipfsx APIs is going to take time - we can't send a single PR with all the changes, it has to be done incrementally. I also don't think it's really an option for us to have a main release that actually removes functionality without an upgrade path (unless it's a planned removal of an API).

We could say 0.50.0 will have all the new core APIs, which would allow us to make releases as 0.50.0-beta.x as we land ipfsx APIs. That would give us space to use 0.40 - 0.49 for landing features and releasing with the old APIs - that's like 15 weeks - think we could do it.

We'd still use master when working on ipfsx but we'd have a temporary branch for the 0.50.0-beta.x releases that would change the default export to the new APIs not the old ones.

On a separate note I think "beta" is a loaded term and ideally I'd like the HTTP and CLI APIs to also be revamped before we move js-ipfs to beta.

Also, one final assertion - I'm not going to release anything related to these API changes that doesn't have great documentation and useful examples to accompany it 😁

[mikeal]

My vote goes for having the new API in beta and call the old one legacy to make it evident that users should move away from it (or something that clearly points to the user what is new and good and what is old and should change).

It’s a little odd as a consumer to have a legacy API, a Beta API, and no “current” or “stable” API. I also think it’s just generally a good practice to put out new things early, gather feedback, iterate, and then once they stabilize consider replacing the existing API. No matter how good an idea is, and I think this new API is a great idea, you still want to have a discussion about replacing the current API based on the full implementation and feedback it has received rather than just replacing with a beta API that hasn’t been used in the wild much yet.

[achingbrain]

The point of having ipfsx behind an EXPERIMENTAL flag is that it would be just that, experimental, so we can add, remove and change things and people would have to explicitly opt-in to that wild ride.

It’s a little odd as a consumer to have a legacy API, a Beta API, and no “current” or “stable” API

Agreed - the current API would continue to be the current API, until ipfsx is ready to become the current API (some time in February/early March) by @alanshaw's reckoning), at which point it would become the legacy API (and preferably would not be in the core codebase any more).

We can't really tell people to move off the current API until there's something for them to move to, and it has good answers for the majority of use cases.

we'd have a temporary branch for the 0.50.0-beta.x releases

Rather than it having different code, it might just have the EXPERIMENTAL.ipfsx flag on by default, then for 0.51.0 we could remove the legacy API code and release that as a separate wrapper module.

I'm not going to release anything related to these API changes that doesn't have great documentation and useful examples to accompany it

This should be standard practise for everything we do 🚀