Fix empty/columns check on HDFStore [Resolves #589] #592
Conversation
Pandas' read_hdf does not implement a method for only loading the first n rows that works reliably on different datasets, particularly with multi-indices. This is problematic because head_of_matrix is how the MatrixStore implements empty() and columns() methods. Instead of implementing head_of_matrix, we implement empty() and columns() on HDFStore directly, obviating the need for a head_of_matrix method that only reads the first row.
| except ValueError: | ||
| # There is no known way to make the start/stop operations work all the time | ||
| # , there is often a ValueError when trying to load just the first row | ||
| # However, if we do get a ValueError that means there is data so it can't be empty |
Codecov Report
@@ Coverage Diff @@
## master #592 +/- ##
==========================================
+ Coverage 82.49% 82.49% +<.01%
==========================================
Files 83 83
Lines 4845 4851 +6
==========================================
+ Hits 3997 4002 +5
- Misses 848 849 +1
Continue to review full report at Codecov.
|
|
As for whether we implement HDF-on-S3 here or elsewhere, I'm not totally sure, but implementing it here kind of makes sense I think. One alternative, a new storage type (e.g. disk, s3, disk-backed s3) would I think have cleaner code, but a worse interface and performance. Making people choose a new storage type to handle this is kind of smelly, and since experiments use the same project path for both matrices and models, it would make all models go through a disk intermediary which would probably be bad performance-wise |
Pandas' read_hdf does not implement a method for only loading the first
n rows that works reliably on different datasets, particularly with multi-indices.
This is problematic because head_of_matrix is how the MatrixStore
implements empty() and columns() methods.
Instead of implementing head_of_matrix, we implement
empty() and columns() on HDFStore directly, obviating the need for a
head_of_matrix method that only reads the first row.