API: Establish pydata_backend

[mtsokol]

Replaces #645

Hi @hameerabbasi,

Including the new backend in pydata/sparse ended up a bit convoluted, therefore I decided to break this task into smaller steps.

Here's the first step which moves existing implementation to sparse/pydata_backend directory with tests and adds context manager that tracks backend contextvar. The test suite has a lot of pydata specific calls so it will be a separate effort to make tests common for both backends.

I applied all your comments from #645 here.

The next PR will run some stub tests with finch to see that it can be installed and run in CI.

[github-actions]

Test Results

5 899 tests  +2   5 865 ✅  - 1   3m 18s ⏱️ -1s
    1 suites ±0      34 💤 +3 
    1 files   ±0       0 ❌ ±0 

Results for commit 4c309c7. ± Comparison against base commit 92e8078.

This pull request removes 5897 and adds 5899 tests. Note that renamed tests count towards both.

sparse.tests.test_array_function ‑ test_binary[dot-arg_order0]
sparse.tests.test_array_function ‑ test_binary[dot-arg_order1]
sparse.tests.test_array_function ‑ test_binary[dot-arg_order2]
sparse.tests.test_array_function ‑ test_binary[matmul-arg_order0]
sparse.tests.test_array_function ‑ test_binary[matmul-arg_order1]
sparse.tests.test_array_function ‑ test_binary[matmul-arg_order2]
sparse.tests.test_array_function ‑ test_binary[result_type-arg_order0]
sparse.tests.test_array_function ‑ test_binary[result_type-arg_order1]
sparse.tests.test_array_function ‑ test_binary[result_type-arg_order2]
sparse.tests.test_array_function ‑ test_binary[tensordot-arg_order0]
…

sparse.pydata_backend.tests.test_array_function ‑ test_binary[dot-arg_order0]
sparse.pydata_backend.tests.test_array_function ‑ test_binary[dot-arg_order1]
sparse.pydata_backend.tests.test_array_function ‑ test_binary[dot-arg_order2]
sparse.pydata_backend.tests.test_array_function ‑ test_binary[matmul-arg_order0]
sparse.pydata_backend.tests.test_array_function ‑ test_binary[matmul-arg_order1]
sparse.pydata_backend.tests.test_array_function ‑ test_binary[matmul-arg_order2]
sparse.pydata_backend.tests.test_array_function ‑ test_binary[result_type-arg_order0]
sparse.pydata_backend.tests.test_array_function ‑ test_binary[result_type-arg_order1]
sparse.pydata_backend.tests.test_array_function ‑ test_binary[result_type-arg_order2]
sparse.pydata_backend.tests.test_array_function ‑ test_binary[tensordot-arg_order0]
…

This pull request removes 31 skipped tests and adds 34 skipped tests. Note that renamed tests count towards both.

sparse.tests.test_compressed ‑ test_reductions_float16[f4-axis4-sum-kwargs0]
sparse.tests.test_compressed ‑ test_reductions_float16[f8-axis4-sum-kwargs0]
sparse.tests.test_compressed ‑ test_reductions_float16[i4-0-mean-kwargs1]
sparse.tests.test_compressed ‑ test_reductions_float16[i4-0-sum-kwargs0]
sparse.tests.test_compressed ‑ test_reductions_float16[i4-1-mean-kwargs1]
sparse.tests.test_compressed ‑ test_reductions_float16[i4-1-sum-kwargs0]
sparse.tests.test_compressed ‑ test_reductions_float16[i4-axis4-mean-kwargs1]
sparse.tests.test_compressed ‑ test_reductions_float16[i4-axis4-sum-kwargs0]
sparse.tests.test_compressed ‑ test_reductions_float16[i8-0-mean-kwargs1]
sparse.tests.test_compressed ‑ test_reductions_float16[i8-0-sum-kwargs0]
…

sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[f4-None-sum-kwargs0]
sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[f4-axis4-sum-kwargs0]
sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[f8-axis4-sum-kwargs0]
sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-0-mean-kwargs1]
sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-0-sum-kwargs0]
sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-1-mean-kwargs1]
sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-1-sum-kwargs0]
sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-axis4-mean-kwargs1]
sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i4-axis4-sum-kwargs0]
sparse.pydata_backend.tests.test_compressed ‑ test_reductions_float16[i8-0-mean-kwargs1]
…

♻️ This comment has been updated with latest results.

[hameerabbasi]

Thanks for the great work here, @mtsokol! A couple of comments.

[hameerabbasi]

Small changes.

[hameerabbasi]

We also need a rebase.

[rgommers]

Quick question: since pydata_backend and finch_backend are (I assume) private and are only supposed to be used via context manager backend selection, wouldn't it be better to prepend underscores to the directory names?

[mtsokol]

Sure! I think they can be prepended with an underscore. There will always be backend-specific members, like GCXS only present in pydata_backend, but importing it is hidden by backend selection mechanism (env-var or context manager).