Feat/neptune opencl

[porcuquine]

This PR adds support for a new feature flag, gpu2, which can be used in place of gpu. When this flag is active, the opencl flag of the neptune crate will be used. A new CI test demonstrates how to enable this option and establishes that it works (at least in the CI environment).

Because the opencl neptune flag makes GPU batch hashing almost twice as fast, we would expect to see that performance realized in GPU column and tree building here. However, the actual gains seen now are much less. My hypothesis is that this is because the parallelism and i/o here were optimized just well enough to not be the bottleneck previously. This hypothesis is partially confirmed by observation that GPU utilization with the gpu2 flag is only a little more than 50% — suggesting that the problem is indeed inability to feed the GPU fast enough.

I propose we merge this feature now and encourage users to experiment with the gpu2 flag in order to ensure it can indeed run successfully on deployed platforms. Performance should still be equivalent to or better than the old gpu usage.

Once we have established that gpu2 does not cause harm? That will free us to remove the old behavior and replace it with the new — so that the gpu flag also activates neptune's opencl flag. This will keep the code base simpler and remove one source of complexity when optimizing tree-building in order to exploit the new, better performance of neptune opencl.

[dignifiedquire]

@cryptonemo @porcuquine what's the status of this?

[cryptonemo]

@cryptonemo @porcuquine what's the status of this?

I haven't tested it, but the changes are mostly feature related, so it depends entirely on the state of neptune's usage of them. Can revisit after other issues are fixed.

[porcuquine]

This fully integrates the neptune OpenCL changes. It does not realize the full expected performance gains because we now hit a bottleneck in how the data is loaded onto the GPU from disk.

As @dignifiedquire and I discussed, the likely next step is to release this so users can test to ensure they are able to use the gpu2 flag without problems. Once we establish that, we can remove gpu2 and default to using neptune's opencl flag as the default (when gpu) is specified. This will keep the code simpler and could be accomplished when the further optimization to the code here is complete.

If it takes too long, or I have extra time, I can perform the optimization first — but the above is a logical progression and what we have discussed as the plan of record. I haven't had time to actually investigate the fix, but some discussion has prompted my memory, and I think I know the general shape of the problem.

[cryptonemo]

Looks good, will merge after a rebase

[porcuquine]

This is rebased now.

[cryptonemo]

Looks good -- clear to merge (squash and merge, preferred) when CI is happy.