Concatenate small input chunks before P2P rechunking

[hendrikmakait]

Blocked by (and includes) #8831

• Tests added / passed
• Passes pre-commit run --all-files

[github-actions]

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

    25 files  ± 0      25 suites  ±0   10h 9m 26s ⏱️ - 10m 16s
 4 118 tests + 6   4 000 ✅ + 5    111 💤  -  2  7 ❌ +3 
47 571 runs  +60  45 457 ✅ +81  2 106 💤  - 25  8 ❌ +4 

For more details on these failures, see this check.

Results for commit 5487c23. ± Comparison against base commit e9d8233.

This pull request removes 8 and adds 14 tests. Note that renamed tests count towards both.

distributed.tests.test_client ‑ test_client_connectionpool_semaphore_loop
distributed.tests.test_client ‑ test_client_gather_semaphore_loop
distributed.tests.test_utils_test ‑ test_dump_cluster_state
distributed.tests.test_utils_test ‑ test_dump_cluster_state_nannies
distributed.tests.test_utils_test ‑ test_dump_cluster_state_no_workers
distributed.tests.test_utils_test ‑ test_dump_cluster_state_timeout
distributed.tests.test_utils_test ‑ test_dump_cluster_state_unresponsive_local_worker
distributed.tests.test_utils_test ‑ test_dump_cluster_unresponsive_remote_worker

distributed.diagnostics.tests.test_install_plugin ‑ test_package_install_on_nanny
distributed.diagnostics.tests.test_install_plugin ‑ test_package_install_on_worker
distributed.shuffle.tests.test_rechunk ‑ test_calculate_prechunking_3d[old0-new0-expected0]
distributed.shuffle.tests.test_rechunk ‑ test_calculate_prechunking_3d[old1-new1-expected1]
distributed.shuffle.tests.test_rechunk ‑ test_calculate_prechunking_3d[old2-new2-expected2]
distributed.shuffle.tests.test_rechunk ‑ test_calculate_prechunking_3d[old3-new3-expected3]
distributed.shuffle.tests.test_rechunk ‑ test_calculate_prechunking_concatenation[1 B-expected0]
distributed.shuffle.tests.test_rechunk ‑ test_calculate_prechunking_concatenation[100 B-expected3]
distributed.shuffle.tests.test_rechunk ‑ test_calculate_prechunking_concatenation[20 B-expected1]
distributed.shuffle.tests.test_rechunk ‑ test_calculate_prechunking_concatenation[40 B-expected2]
…

♻️ This comment has been updated with latest results.

[hendrikmakait]

A/B tests performed on coiled/benchmarks#1532 show a significant improvement in runtime for tests with small input chunks:

As expected, we also see a significant increase in memory usage for those tests:

The memory usage is still within very safe bounds.

Interestingly, we also see an increase in tiles_to_rows, once again within safe bounds:

[phofl]

The memory usage is still within very safe bounds.

Could you quantify this a little?

[hendrikmakait]

The memory usage is still within very safe bounds.

Could you quantify this a little?

Average memory increases by 10%-30% of the original value.

[hendrikmakait]

@phofl: As discussed offline, I've extended the documentation. LMK if anything is still unclear.

[phofl]

few very small comment, but lgtm now!

Thanks for the comments, that was very helpful

[phofl]

This one worries me a little bit. the max input chunk is 2, max output chunk is 4 but the algorithm concatenates in a way that we end with 8, which is not great

[phofl]

is the block size limit the upper bound here?

[hendrikmakait]

Yes, https://github.com/dask/distributed/pull/8832/files#diff-0b80e83452ff3472b265026d4516846014500b991e12f3de4a41b39a990afbc6R494 is the limit here, so in this case it's 8 because array.chunk-size is 16 B.

[phofl]

Can you add a sentences what these 2 variables represent when you define them above? Took me a bit to figure this out

[hendrikmakait]

Done