Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RowToColumnTests.Biggest OOM on A3000 Laptop GPU with 6 GiB #2070

Open
gerashegalov opened this issue May 24, 2024 · 1 comment
Open

RowToColumnTests.Biggest OOM on A3000 Laptop GPU with 6 GiB #2070

gerashegalov opened this issue May 24, 2024 · 1 comment

Comments

@gerashegalov
Copy link
Collaborator

gerashegalov commented May 24, 2024

Repro:

./build/build-in-docker clean install \
  -DGPU_ARCHS=NATIVE \
  -DBUILD_TESTS=ON  \
  -DskipTests  \
  -Dlibcudf.clean.skip=false

Log file contet

RowToColumnTests.Biggest
unknown file: Failure
C++ exception with description "std::bad_alloc: out_of_memory: CUDA error at: /home/user/gits/NVIDIA/spark-rapids-jni/target/libcudf-install/include/rmm/mr/device/cuda_memory_resource.hpp:60: cudaErrorMemoryAllocation out of memory" thrown in the test body.

The same test passes on Quadro RTX 6000 with 24GiB

The test hard-codes a pretty large table in absolute terms:

constexpr auto num_rows = 2 * 1024 * 1024;
for (int i = 0; i < 128; ++i) {
cols.push_back(cudf::test::fixed_width_column_wrapper<int32_t>(r + num_rows * i,
r + num_rows * i + num_rows));
views.push_back(cols.back());
schema.push_back(cudf::data_type{cudf::type_id::INT32});
}

(2 * 1024 * 1024) rows * 128 cols * 4 = 1GiB

which is then replicated by conversion. Presumably the smaller GPU cannot pass this test as is.

The test should fail or be skipped with a clear message indicating that the GPU under test is too small.

@mattahrens
Copy link
Collaborator

Scope: update documentation related to test environment requirements for GPU memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants