[BUG] cudf::to_arrow is not properly synchronized

[harrism]

cudf::to_arrow copies data from device to host on a stream and never synchronizes that stream before making it available to the Arrow library, which does not use the same stream. Therefore I believe there is a potential race condition.

This is an example of one of the functions in to_arrow.cpp that copies data asynchronously to the host.

cudf/cpp/src/interop/to_arrow.cpp

Lines 39 to 58 in 3d44ed5

	template <typename T>
	std::shared_ptr<arrow::Buffer> fetch_data_buffer(column_view input_view,
	arrow::MemoryPool* ar_mr,
	cudaStream_t stream)
	{
	const int64_t data_size_in_bytes = sizeof(T) * input_view.size();
	auto result = arrow::AllocateBuffer(data_size_in_bytes, ar_mr);
	CUDF_EXPECTS(result.ok(), "Failed to allocate Arrow buffer for data");
	std::shared_ptr<arrow::Buffer> data_buffer = std::move(result.ValueOrDie());
	CUDA_TRY(cudaMemcpyAsync(data_buffer->mutable_data(),
	input_view.data<T>(),
	data_size_in_bytes,
	cudaMemcpyDeviceToHost,
	stream));
	return data_buffer;
	}

The to_arrow function ultimately calls this function and others like it, but never synchronizes before returning. Therefore an application could use this function to get an Arrow array/buffer, and then access that array on the host without synchronizing the stream.

I believe currently pageable host memory is used, which would cause cudaMemcpyAsync to return only once the copy is completed. But if this is switched to pinned host memory, it will be asynchronous, and we will have a race condition.

[harrism]

I believe the same is true of to_dlpack