Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unshim ExclusiveModeGpuDiscoveryPlugin #3590

Conversation

gerashegalov
Copy link
Collaborator

Fixes #3589

As one of the publicly documented classes loaded by Spark outside the
plugin control, ExclusiveModeGpuDiscoveryPlugin may not reside in the
Parallel World locations. Binary dedupe shows
ExclusiveModeGpuDiscoveryPlugin is bitwise-identical across all shims

Signed-off-by: Gera Shegalov gera@apache.org

As one of the publicly documented classes loaded by Spark outside the
plugin control, ExclusiveModeGpuDiscoveryPlugin may not reside in the
Parallel World locations. Binary dedupe shows
ExclusiveModeGpuDiscoveryPlugin is bitwise-identical across all shims

Signed-off-by: Gera Shegalov <gera@apache.org>
@gerashegalov gerashegalov self-assigned this Sep 21, 2021
@gerashegalov gerashegalov added the bug Something isn't working label Sep 21, 2021
@gerashegalov gerashegalov added this to the Sep 13 - Sep 24 milestone Sep 21, 2021
@gerashegalov
Copy link
Collaborator Author

 jar tvf dist/target/rapids-4-spark_2.12-21.10.0-SNAPSHOT.jar | grep ExclusiveModeGpu
 10734 Tue Sep 21 16:18:04 PDT 2021 com/nvidia/spark/ExclusiveModeGpuDiscoveryPlugin.class

@gerashegalov
Copy link
Collaborator Author

build

tgravescs
tgravescs previously approved these changes Sep 21, 2021
@gerashegalov
Copy link
Collaborator Author

build

@tgravescs
Copy link
Collaborator

we need to finish testing this manually

via ShimLoader

Signed-off-by: Gera Shegalov <gera@apache.org>
@tgravescs
Copy link
Collaborator

weird ci hasn't reported, I don't see anything running, kicking again

@tgravescs
Copy link
Collaborator

build

@tgravescs
Copy link
Collaborator

I tried this out manually and its working for me as well:
21/09/22 06:55:01 INFO InternalExclusiveModeGpuDiscoveryPlugin: Running ExclusiveModeGpuDiscoveryPlugin to acquire 1 GPU(s), host has 1 GPU(s)

@tgravescs
Copy link
Collaborator

failed with the scalastyle issue because of import order I mention above

@gerashegalov
Copy link
Collaborator Author

build

@tgravescs tgravescs merged commit f4465cb into NVIDIA:branch-21.10 Sep 22, 2021
@gerashegalov gerashegalov deleted the unshimExclusiveModeGpuDiscoveryPlugin branch November 16, 2022 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants