Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge of rn/zarr and mpiarray for testing #195

Closed
wants to merge 197 commits into from
Closed

Merge of rn/zarr and mpiarray for testing #195

wants to merge 197 commits into from

Conversation

anjakefala
Copy link
Contributor

@anjakefala
feat(mpiarray): a first pass at implementing naive ufunc handling
278ee10 
- Upon a ufunc being executed, both inputs and outputs are converted to
standard nparrays, and then the nparray ufunc is called
- if no inputs are provided, the results are then converted back to MPIArrays distributed along the axis
of the input arrays
- all inputted MPIArrays *must* be distributed along the same axes

chime-experiment/Pipeline#81
@anjakefala
feat(mpiarray): build up attributes for new MPIArray in __array_final…
a83940a 
…ize__
@anjakefala
feat(mpiarray): handle case when result is a scalar
a5b7520 
- scalars should just be returned as-is, not converted into MPIArrays
@anjakefala
docs(mpiarray): improve comments and docs
6b39f46
@anjakefala
feat(mpiarray): experimental, convert the local array into ndarray
8210548 
- must try with MPI
@anjakefala
docs(mpiarray): more documentation addition
bcbd75f
@anjakefala
style: black
fb5c03e
@anjakefala
fix(mpiarray): dictionaries have keys, not attributes
d2c1a05
@anjakefala
style(mpiarray): general clean-up
275fbbf
@anjakefala
feat: return scalars as 1-d vectors across distributed axis
1e64c48
@anjakefala
docs(mpiarray): expand documentation
dc2a9f1
@anjakefala
feat(mpiarray): if the method is reduce, check if the distributed axe…
0ecf78b 
…s needs recalculation
@anjakefala
style: run black
4534dbc
@anjakefala
Merge branch 'master' into mpiarray
3c5b617
@anjakefala
fix(mpiarray): check if result is scalar, before adjusting distribute…
7a527cd 
…d axis
@anjakefala
feat(mpiarray): clean ups and refactoring
d38969d
@anjakefala
fix(mpiarray): handle linter errors
3ecadef
@anjakefala
build: ignore R1710 linter, does not make sense with array_ufunc
17e7d87 
Either all return statements in a function should return an expression,
or none of them should. (inconsistent-return-statements)

This does not make sense because __array_ufunc__ is a general function
which handles various ufuncs, some which return nothing (like when the
method "at" is used.)
@anjakefala
fix: improve logic for re-calculating distributed axes
b321b85
@anjakefala
fix: fix global slicing; do not use allreduce to grab global_shape le…
4286cee 
…ngth
@anjakefala
build(mpiarray): add basic tests
6d28085
@anjakefala
build(mpiarray): test reductions with different axes set-up
d30e39f
@anjakefala
fix: add further tests for outer_ufunc; change assert to ValueError
1b1f37d
@anjakefala
build: check failure to broadcast
ccc35d4
@anjakefala
docs: wrap docstrings to 80 chars
dcefcd0
@anjakefala
fix(mpiarray): if it cannot reconstruct the distributed axis, then an…
62ca3c0 
… exception should be thrown.
@anjakefala
style: improve readability
fb11885
@anjakefala
fix: establish axis change in reduce case
cac8103 
If distributed axis is greater than reduction axis, reduce it by 1.
Otherwise, keep it the same.

If keepdims is passed and true, also keep it the same.
@anjakefala
fix: fix bad offset
2f123ef
@anjakefala
fix: do not raise exception for ndarray.view() case that occurs withi…
bf4b53a 
…n wrap()
tristpinsm and others added 23 commits March 1, 2022 16:07
@tristpinsm
refactor(truncate): Allow non-contiguous arrays.
5aca3b8
@tristpinsm
fix(memh5.copyattrs): Convert numpy types.
ef44b61
@anjakefala
fix: compression_kwargs is a dict
380edc7
@anjakefala
fix: more minor fixes based on comments
7bf6279
@anjakefala
style: ran black
3765b3d
@anjakefala
style(memh5): clean up compression kwargs
8c749c9
@anjakefala
Merge branch 'master' into rn/zarr
2016b92
@tristpinsm
doc(truncate): Add note explaining why non-cont arrays are supported.
cc9a5bb
@tristpinsm
fix(config): file format property check.
c1355c4
@tristpinsm
fix(memh5): don't assume dataset has compression attributes.
e02bd97
@tristpinsm
test(test_memh5): Add compression test.
e18479a
@tristpinsm
test(test_memh5_parallel): update test with compression.
e170ae7
@tristpinsm
fix(config): Allow None default in file_format
1a23d8d
@jrs65
fix(profile): workaround missing metrics on macOS
e4928e1
@jrs65
test: fix test suite after merge issues
82d235c
@jrs65
style: apply black
cb186b8
@jrs65
test: remove duplication
3b5c5c8
@jrs65
fix: linter error
dc3074a
@jrs65
ci: remove requirement for binary wheels to speed up build
5f04d9f
@jrs65
style: reapply black
988e80c
@jrs65
ci: update workflow and remove another bitshuffle binary wheel
18d1863
@tristpinsm
fix(truncate): Mask for float mantissa overflow was wrong.
16735a9
@tristpinsm
test(truncate): Add special cases.
82eb13b
@anjakefala anjakefala marked this pull request as draft March 14, 2022 21:18
@anjakefala anjakefala force-pushed the develop branch 2 times, most recently from 7395be2 to ea8c381 Compare March 14, 2022 21:35
@jrs65
Copy link
Contributor

jrs65 commented May 30, 2022

Everything is now merged

@jrs65 jrs65 closed this May 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@anjakefala @jrs65 @nritsche @tristpinsm