Memoize dask datashape

[philippjfr]

As suggested in #797

[philippjfr]

Okay, the previous approach still wasn't saved. This new approach loads all the categories into memory in a way that is probably less efficient but guaranteed to be correct, but then adds memoization so you only pay the cost once.

[jbednar]

Does that mean that we have to forgo the speedups when reading fastparquet-written files with fastparquet, in order to make the other cases safe? Or can we have both the speedup in the best case and the correctness in the other cases?

[philippjfr]

Does that mean that we have to forego the speedups when reading fastparquet-written files with fastparquet, in order to make the other cases safe?

I would assume that fastparquet categorical columns are known at the time of load so we would get the benefit but I'll have to test that assumption.

[philippjfr]

I would assume that fastparquet categorical columns are known at the time of load so we would get the benefit but I'll have to test that assumption.

This assumption was wrong but loading the categories using fastparquet is significantly faster than with pyarrow (although pyarrow seems to be faster than fastparquet in general). Therefore I do think it must be using the metadata in some form.