Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] OOM crash on left merge of categoricals #1673

Closed
lmeyerov opened this issue May 8, 2019 · 3 comments
Closed

[BUG] OOM crash on left merge of categoricals #1673

lmeyerov opened this issue May 8, 2019 · 3 comments
Labels
bug Something isn't working Needs Triage Need team to review and classify

Comments

@lmeyerov
Copy link

lmeyerov commented May 8, 2019

Describe the bug

Left merge on even moderate categorical data causes RMM to OOM crash.

Demo: 500K rows -

bio_1col_gdf[['Throughput']].merge(bio_1col_gdf[['Throughput']], on='Throughput', how='left')

=>

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-84-dc34e62c26e9> in <module>()
----> 1 bio_1col_gdf[['Throughput']].merge(bio_1col_gdf[['Throughput']], on='Throughput', how='left')

/usr/local/lib/python3.6/site-packages/cudf/dataframe/dataframe.py in merge(self, right, on, how, left_on, right_on, left_index, right_index, lsuffix, rsuffix, type, method, indicator, suffixes)
   1551         # Compute merge
   1552         cols, valids = cpp_join.join(lhs._cols, rhs._cols, left_on, right_on,
-> 1553                                      how, method=method)
   1554 
   1555         # Output conversion - take cols and valids from `cpp_join` and

cudf/bindings/join.pyx in cudf.bindings.join.join()

cudf/bindings/join.pyx in cudf.bindings.join.join()

RuntimeError: RMM error encountered at: /conda/envs/gdf/conda-bld/libcudf_1557215827485/work/cpp/src/join/join_compute_api.h:322: 4 RMM_ERROR_OUT_OF_MEMORY

Steps/Code to reproduce bug
Run Issue 5 of https://colab.research.google.com/drive/1S5UhplicI4gpEIJz0jQdhGd3gyBAYbee#scrollTo=vpXDxGpAeysg .

Expected behavior
Left merge to terminate.

Environment details (please complete the following information):

  • !conda install -q -y --prefix /usr/local -c conda-forge
    -c rapidsai-nightly/label/cuda10.0 -c nvidia/label/cuda10.0
    cudf cuml => 0.7.0.dev0
  • Environment location: Colab
  • Method of cuDF install: Conda
  • Please run and attach the output of the cudf/print_env.sh script to gather relevant environment details
    git*
    Not inside a git repository

OS Information
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.2 LTS"
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
Linux fe4dd8671d4c 4.14.79+ #1 SMP Wed Dec 19 21:19:13 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

GPU Information
Wed May 8 00:28:11 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.56 Driver Version: 410.79 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:04.0 Off | 0 |
| N/A 73C P0 31W / 70W | 489MiB / 15079MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+

CPU
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 2
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 63
Model name: Intel(R) Xeon(R) CPU @ 2.30GHz
Stepping: 0
CPU MHz: 2300.000
BogoMIPS: 4600.00
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 46080K
NUMA node0 CPU(s): 0,1
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt arat arch_capabilities

CMake
/usr/local/bin/cmake
cmake version 3.12.0

CMake suite maintained and supported by Kitware (kitware.com/cmake).

g++
/usr/bin/g++
g++ (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

nvcc
/usr/local/cuda/bin/nvcc
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

Python
/usr/local/bin/python
Python 3.6.7

Environment Variables
PATH : /usr/local/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/tools/node/bin:/tools/google-cloud-sdk/bin:/opt/bin
LD_LIBRARY_PATH : /usr/lib64-nvidia
NUMBAPRO_NVVM : /usr/local/cuda/nvvm/lib64/libnvvm.so
NUMBAPRO_LIBDEVICE : /usr/local/cuda/nvvm/libdevice/
CONDA_PREFIX :
PYTHON_PATH :

conda packages
/usr/local/bin/conda

packages in environment at /usr/local:

Name Version Build Channel

arrow-cpp 0.12.1 py36h0e61e49_0 conda-forge
asn1crypto 0.24.0 py36_0
boost-cpp 1.68.0 h11c811c_1000 conda-forge
bzip2 1.0.6 h14c3975_1002 conda-forge
ca-certificates 2019.3.9 hecc5488_0 conda-forge
certifi 2019.3.9 py36_0 conda-forge
cffi 1.11.5 py36h9745a5d_0
chardet 3.0.4 py36h0f667ec_1
conda 4.6.14 py36_0 conda-forge
conda-env 2.6.0 h36134e3_1
cryptography 2.6.1 py36h72c5cf5_0 conda-forge
cudatoolkit 10.0.130 0
cudf 0.7.0.dev0 py36_1516 rapidsai-nightly/label/cuda10.0
cuml 0.8.0a cuda10.0_py36_539 rapidsai-nightly/label/cuda10.0
icu 58.2 hf484d3e_1000 conda-forge
idna 2.6 py36h82fb2a8_1
libblas 3.8.0 8_openblas conda-forge
libcblas 3.8.0 8_openblas conda-forge
libcudf 0.7.0.dev0 cuda10.0_1514 rapidsai-nightly/label/cuda10.0
libcuml 0.8.0a cuda10.0_539 rapidsai-nightly/label/cuda10.0
libcumlmg 0.0.0.dev0 cuda10.0_373 nvidia/label/cuda10.0
libedit 3.1.20170329 h6b74fdf_2
libffi 3.2.1 hd88cf55_4
libgcc-ng 8.2.0 hdf63c60_1
libgfortran-ng 7.3.0 hdf63c60_0
liblapack 3.8.0 8_openblas conda-forge
libnvstrings 0.7.0.dev0 cuda10.0_130 rapidsai-nightly/label/cuda10.0
libprotobuf 3.6.1 hdbcaa40_1001 conda-forge
librmm 0.7.0.dev0 cuda10.0_48 rapidsai-nightly/label/cuda10.0
libstdcxx-ng 8.2.0 hdf63c60_1
llvmlite 0.28.0 py36hdbcaa40_0 conda-forge
ncurses 6.1 hf484d3e_0
numba 0.43.1 py36hf2d7682_0 conda-forge
numpy 1.16.3 py36he5ce36f_0 conda-forge
nvstrings 0.7.0.dev0 py36_130 rapidsai-nightly/label/cuda10.0
openblas 0.3.6 h6e990d7_1 conda-forge
openssl 1.1.1b h14c3975_1 conda-forge
pandas 0.24.2 py36hf484d3e_0 conda-forge
parquet-cpp 1.5.1 4 conda-forge
pip 10.0.1 py36_0
pyarrow 0.12.1 py36hbbcf98d_0 conda-forge
pycosat 0.6.3 py36h0a5515d_0
pycparser 2.18 py36hf9f622e_1
pyopenssl 18.0.0 py36_0
pysocks 1.6.8 py36_0
python 3.6.7 h381d211_1004 conda-forge
python-dateutil 2.8.0 py_0 conda-forge
pytz 2019.1 py_0 conda-forge
readline 7.0 ha6073c6_4
requests 2.18.4 py36he2e5f8d_1
rmm 0.7.0.dev0 py36_48 rapidsai-nightly/label/cuda10.0
ruamel_yaml 0.15.37 py36h14c3975_2
setuptools 39.2.0 py36_0
six 1.11.0 py36h372c433_1
sqlite 3.26.0 h67949de_1001 conda-forge
thrift-cpp 0.12.0 h0a07b25_1002 conda-forge
tk 8.6.9 h84994c4_1001 conda-forge
urllib3 1.22 py36hbe7ace6_0
wheel 0.31.1 py36_0
xz 5.2.4 h14c3975_4
yaml 0.1.7 had09818_2
zlib 1.2.11 ha838bed_2

Additional context
Add any other context about the problem here.
@lmeyerov lmeyerov added Needs Triage Need team to review and classify bug Something isn't working labels May 8, 2019
@jrhemstad jrhemstad mentioned this issue May 8, 2019
Closed
@thomcom
Copy link
Contributor

thomcom commented May 9, 2019

Does #1689 improve this?

@lmeyerov
Copy link
Author

I don't think so. I may have tested incorrectly: I tried on the conda nightly, 0.7.0.dev0 py36_1567 rapidsai-nightly/label/cuda10.0

@kkraus14
Copy link
Collaborator

Connected with @lmeyerov and @trxcllnt offline about this and this use case is causing a cartesian product which just OOMs the GPU. Fix for their case is to call .unique() on the left table before merging. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working Needs Triage Need team to review and classify
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@thomcom @kkraus14 @lmeyerov