WIP: Displaying a way to break the chain_db when mongo-only collections are present #799
Conversation
|
@sbillinge this is to show how I am playing with the chaining. Details in above comment. (most file changes are from an inconsequential change, above comment points to consequential changes) |
|
Thanks Zach,
I am focussed rn working with Connor to get 0.6.0 release done, but I am
excited to turn back to this after that. I think a short call would work
best so I can decide how to move forward. I am routinely using the mongo
backend for todos now (which reminds me, I should back them up!), and
everything is working really sweetly! All the dbs playing together rather
well.
S
…On Wed, Jul 14, 2021 at 3:07 PM zthatch ***@***.***> wrote:
@sbillinge <https://github.com/sbillinge> this is to show how I am
playing with the chaining. Details in above comment.
(most file changes are from an inconsequential change, above comment
points to consequential changes)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#799 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABAOWUJLV7TF24QGGWSC3CDTXXN6PANCNFSM5AMBP74A>
.
--
Simon Billinge
Professor, Columbia University
Physicist, Brookhaven National Laboratory
|
…t doesn't break tests, but does remove ability to chain across mongo and fs. Commented code in chain_db en route to fix this.
… on each of the milestones, deliverables, and kickoffs before the updater runs at all. Likely because of a negligence to implement a deepcopy method in the chained collections leading to issues after the milestones have been listed.
…nd fs to mongo all broken
…base.xsh to put collections into their respective maps rather than the outputs of the iter method.
|
Refactored in such a way that we can now do everything we have ever wanted with minimal transitions. The databases used to chain at only the document level, which made it impossible to use the chained database as anything other than a reference when attempting to chain across the fs and mongo. This has been changed such that there is now a chaining object at the collection level as well, which makes a distinction between mongo and fs collections and handles the chaining in a way that allows the helper maintainers to check for a fs components, and if it is not present they can use mongo operations. This was done in such a way that we can maintain the original functionality of chaining across mongo and filesystem collections, contrary to the previous attempt. Test failures are not happening on windows, resolving on VM. |
|
wow, this is exciting news! |
…d it must load all of the mongo collections into memory.
Codecov Report
@@ Coverage Diff @@
## master #799 +/- ##
==========================================
- Coverage 72.17% 72.00% -0.18%
==========================================
Files 71 71
Lines 7501 7526 +25
==========================================
+ Hits 5414 5419 +5
- Misses 2087 2107 +20
Continue to review full report at Codecov.
|
Breaks the ability to chain across mongo and filesystem.
Power shown in a_proprevhelper.py, which realizes that we have a mongo collection and decides not to load all documents.
Main changes in client_manager.py (allows you to still load all from collections if you so choose), mongoclient line 380, and database.xsh.
Commented code in chain_db is potentially exciting because an update to getitem (not there yet) could allow the typical chaining method to work by running finds and chaining their results at get-time (all documents time), but also allow helper maintainers to check and see if their collection is purely mongo-backed, and if so run mongo commands.