Globally sorted data (hbase like range sharding)

[ddorian]

I read in the docs it would be available in roadmap I think, but I made this issue to be notified when.

Am I correct ?

[rkarthik007]

Thanks for your question @ddorian. It is certainly in our roadmap.

Would like to understand your use-case a bit better to see what features you would need - especially since you reference HBase.

• What column types did you want to use in the range sorted primary key? Is it primarily strings/bytes or other types like INT?
• Do you know your "tablet splits" (region splits for a table in HBase) ahead of time or do you depend on dynamic splitting? If you know your split points and just need a way to create a table with those specified, that is much simpler and immediate (since we're doing this for secondary indexes anyway).

[ddorian]

I meant more for dynamically splitting (& later merging) like hbase.
Best case would be to split on partition columns so you have a way of guaranteeing some rows be in 1 range.

I don't see how you can make secondary indexes scale on cases when there are many same values (which globally sorted helps by splitting on (value+primary-key)) with pre-known split poitns?

[rkarthik007]

Got it. We are planning on implementing dynamic splitting - which you correctly pointed out are needed for low cardinality indexes (with few unique values). We will definitely inform you once that is closer.

[kmuthukk]

@ddorian: Thanks. We do have dynamic splitting/merging in the plans:

-- to support tables where there is no natural "hash" partition in the primary key, but you want to do range queries (> , < , etc.) on the primary key.

-- to support secondary indexes where you want do range queries on those indexed columns, but based on the data set in question pre-splitting may not yield a good balanced distribution.

As you suggested, in both cases, what you maybe indexing on (be it the primary key or the secondary index), could have an uneven distribution in the key space if the space is pre-split. For example, if the column is customer_name, and the table has lot more names starting with "a", we would need the ability to dynamically split the corresponding tablets as they grow.

[ddorian]

When/if you add it, you'll have the foundation to build anything on top (hbase/phoenix, cockroachdb etc). Hopefully a scripts/modules system after that too (redis modules, hbase coprocessors), cant build everything yourself.

[ddorian]

Done in #1134