Performance issues

[JAndritsch]

Hello,

I am using LinvoDB with Levelup as the backend in an Electron application and noticing some pretty slow query performance with a moderately sized data set.

The documents I'm retrieving are pretty small and indexes exist for every field that I need to query on, however LinvoDB documentation suggests that all fields are indexed by default unless you opt out of it.

For reference, here is my "table" in question:

let files = new LinvoDB('files', {
  fingerprint: { type: String, index: true, unique: true },
  status: { type: String, index: true },
  eventId: { type: Number, index: true },
  hidden: { type: Boolean, index: true }
}, {});

Note: this is not the complete schema of this database.

What I've observed so far is that queries seem to take a very long time when the number of documents returned is large, however the queries speed up if the number of returned documents is small.

For example:

Given a database of 6000 documents, performing a find(criteria) that returns all of those records takes about 1.8 seconds. (Note that the initial query to load the data takes roughly 4-5 seconds, but subsequent queries average out to about 1.8). Running a query against that same data set of 6000 items but with criteria that returns only 25 documents takes 0.25 seconds.

Based on these numbers, it seems that the slowness isn't on the search via index but rather the operation(s) involved with loading the documents. The more documents that need to be loaded, the slower the query gets.

I thought perhaps I could increase query performance by bumping the cacheSize of Levelup, but that had no impact. In fact, my database size on disk doesn't even exceed the default cache size of 8mb.

As these number increase, query performance gets much worse. When my database grows to 11,000 documents, queries that return all of those documents take about 4.3 seconds whereas queries that return only 25 results take 0.5 seconds.

Is there something I can do to help narrow down what might be causing this behavior?

Thanks,

Joel

[Ivshti]

This is a known issue, although unfortunately I have not investigated. Maybe you can look into the cursor.js code and check how long the index takes, and then how long the retrieval takes.

My experience is that the index is OK in terms of speed, but then taking the data out of the store can take a while for various reasons - my suspicion is that it's time lost in levelup/leveldown abstractions

Maybe you can try with another store (try with memdown) to see what will happen

[JAndritsch]

I noticed that several of the methods in Cursor use Underscore. I swapped that lib for lodash, which is known to be faster and provides the same utilities. That sped up the code a bit (300-400ms on a DB containing ~4000 documents). I'm sure there are still more optimizations that can be done, but swapping to lodash might be a quick win.

[jx-c]

Have similar query performance issue. Swap the Underscore for lodash did make big performance improvement.