[BUG] BroadcastExchangeExec fails to fall back to CPU on driver node on GCP Dataproc

[sameerz]

Describe the bug
java.lang.UnsatisfiedLinkError: /tmp/cudf6291133341730615510.so: libnvrtc.so.10.2: cannot open shared object file: No such file or directory error when running the Mortgage ETL notebook with the sample mortgage data on a GCP Dataproc cluster, when the master node is not a GPU node.

Error message in this gist: https://gist.github.com/sameerz/b1df1130b0955b562a5801d7eb664811

Steps/Code to reproduce bug

1. Create a new cluster where the GPU node is CPU only and workers have CPU and GPU. Sample cluster creation scripts: https://gist.github.com/sameerz/d655ff262a295e66b3384fcbd15b67dd
2. Launch and run the notebook, instructions here: https://nvidia.github.io/spark-rapids/docs/get-started/getting-started-gcp.html

@tgravescs pointed out that this is avoidable if we set conf.set('spark.rapids.sql.exec.BroadcastExchangeExec', 'false'). However this notebook worked on Dataproc previously.

Expected behavior
GPU operations do not run on the master node if the master is CPU only.

Environment details (please complete the following information)

• Environment location: GCP Dataproc preview-ubuntu18, running either CUDA 10.2 or CUDA 11.0
• 2 GPU worker nodes
• Spark configuration settings related to the issue: Spark config settings are in the initialization script https://github.com/GoogleCloudDataproc/initialization-actions/blob/master/rapids/rapids.sh#L120 and in the Define Spark conf and Create Spark Session section of the notebook https://github.com/NVIDIA/spark-rapids/blob/gh-pages/docs/demo/GCP/Mortgage-ETL-GPU.ipynb.

One note, there is an unrelated error in this notebook - that same cell in the notebook needs to start with from pyspark import SparkConf , but that will be addressed in a separate PR.

Additional context
This error occurs running the 0.2 plugin and 0.15 cudf. Something may have changed in the Dataproc environment.

[jlowe]

Looks like it's trying to build an empty GPU table on the driver so it can turn around and serialize out an empty table. Not sure how we didn't hit this before, as this code has been there a while. Looks like this might be triggered in a case where the broadcast table ends up being empty which I wouldn't expect in the mortgage query.

[jlowe]

There might be an easy fix similar to what was done for rapidsai/cudf#5441, where we don't load native libs for HostColumnVector but instead defer the native libs loading to any class used by HostColumnVector that needs those libs. HostColumnVector doesn't have any native methods itself.

[aroraakshit]

PR mentioned in this bug (around sparkconf import) filed here: #991