Open to adding an optional "batched get" capability?

[andymatuschak]

Many implementations of abstract-leveldown involve FFI, which often incurs transaction overhead. There are good ways to mitigate this overhead for writes and for iterators (the batch API and chunked reads, respectively), but there's no good way to avoid significant FFI overhead when making large numbers of random-access get calls. Existing implementors can batch reads at every tick, but that approach introduces delay for common-case simple reads, as well as extra bookkeeping to track information the client may already have.

How would you all feel about a proposal to make get accept an array of keys? This would be an optional feature, with support indicated by a new supports manifest key (e.g. batchedGet). If there's support for this feature, I'm willing to write a more formal proposal describing its semantics in more detail, implement demonstration support in memdown, and add appropriate tests.

[vweevers]

Thank you for the detailed problem description. For prior discussion, see Level/levelup#491 and Level/levelup#491 (comment).

I'm open to the idea.

The current API does support getting multiple keys from a single iterator, with iterator.seek(), which not only enables getting multiple non-contiguous keys but also getting multiple ranges of keys. Which is to say, it's flexible, but too cumbersome for the simpler use cases. And I previously believed (or hoped) that "batch get" could be implemented in userland on top of iterators, but this is difficult to optimize.

How would you all feel about a proposal to make get accept an array of keys?

We can't reuse get([]) because array keys are already valid input (depending on the keyEncoding). It must be a new method. And it should have snapshot guarantees.

[andymatuschak]

Ah, I see! This is a pretty clear dupe of Level/levelup#491 and Level/levelup#13 (sorry I missed those!), so let's go ahead and close this.

And I previously believed (or hoped) that "batch get" could be implemented in userland on top of iterators, but this is difficult to optimize.

Yes, I see how this could be done, but in the case of random-access read, it still requires 2N FFI round-trips per requested key; I'd like a constant-overhead interface.

Having read the discussion and some of the considerations involved, I'll probably go ahead and just implement this in my own library (https://github.com/andymatuschak/react-native-leveldown) and link it in the relevant issues here in case others want to implement a similar API.

[wholebuzz]

Did you ever implement this @andymatuschak?

[andymatuschak]

Nope, and my plans have changed; I'm no longer likely to!

[vweevers]

See Level/community#101