-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
iter_collections
and ls-file-collection
#343
Commits on May 9, 2023
-
Establish
iter_collections
module, importiterdir()
from gooeyThis is a start for consolidating common functionality scattered around various extensions into a single implementation (pattern). This changeset import `iterdir()` from `datalad-gooey`. In contrast to the original implementation, this new one is using a stricter approach to types, and overfits less to a dataset-aware use case. However, it is not meant to be the exclusive implementation, but merely a start and a place to migrate directory iterators into. Ping datalad#323
Configuration menu - View commit details
-
Copy full SHA for ae185e3 - Browse repository at this point
Copy the full SHA ae185e3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 521d751 - Browse repository at this point
Copy the full SHA 521d751View commit details -
Configuration menu - View commit details
-
Copy full SHA for f2502b8 - Browse repository at this point
Copy the full SHA f2502b8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9e3064e - Browse repository at this point
Copy the full SHA 9e3064eView commit details -
Add support for TAR archive file collections
This is trying to be structurally similar to the `directory` collection implementation.
Configuration menu - View commit details
-
Copy full SHA for 6b070b1 - Browse repository at this point
Copy the full SHA 6b070b1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 01c3d27 - Browse repository at this point
Copy the full SHA 01c3d27View commit details -
Configuration menu - View commit details
-
Copy full SHA for 09632d6 - Browse repository at this point
Copy the full SHA 09632d6View commit details -
Consolidate and deduplicate across collection iterators
Only one path type enum, only one item dataclass (for now).
Configuration menu - View commit details
-
Copy full SHA for 3ae477b - Browse repository at this point
Copy the full SHA 3ae477bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6568fc2 - Browse repository at this point
Copy the full SHA 6568fc2View commit details -
Configuration menu - View commit details
-
Copy full SHA for 09a7513 - Browse repository at this point
Copy the full SHA 09a7513View commit details
Commits on May 10, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 059070a - Browse repository at this point
Copy the full SHA 059070aView commit details -
Standardize
itertar()
to always report (platform)PurePath
Now also for link targets.
Configuration menu - View commit details
-
Copy full SHA for 3fa4db0 - Browse repository at this point
Copy the full SHA 3fa4db0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8832b9b - Browse repository at this point
Copy the full SHA 8832b9bView commit details -
Using a pre-crafted (300 byte) tarball that is put on github, and is downloaded once per session.
Configuration menu - View commit details
-
Copy full SHA for 40c4f55 - Browse repository at this point
Copy the full SHA 40c4f55View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6683ddb - Browse repository at this point
Copy the full SHA 6683ddbView commit details -
Clarify how joint validators must report violations
Also include a check for these requirements that is executed in the error case (no performance critical), to inform developers about obvious implementation issues. Closes datalad#348
Configuration menu - View commit details
-
Copy full SHA for b1655cc - Browse repository at this point
Copy the full SHA b1655ccView commit details -
Configuration menu - View commit details
-
Copy full SHA for fcd0d0c - Browse repository at this point
Copy the full SHA fcd0d0cView commit details -
New
ls-file-collection
commandThis is (also) an alternative approach to `add-archive-content`. In comparison to the former, this is largely metadata driven, and works without (local) extraction of a tarball. This saves storage overhead, and makes it possible to run some parts of the ingestion pipeline on a remote system. Closes datalad#183
Configuration menu - View commit details
-
Copy full SHA for 5ed2248 - Browse repository at this point
Copy the full SHA 5ed2248View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4adba2c - Browse repository at this point
Copy the full SHA 4adba2cView commit details