Fix MPI on Ubuntu 18.04 with CUDA #2271
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
OpenMPI 2.1.1 has a broken vader BTL, requiring us to disable its single-copy mode
mkuron
changed the title
mkuron
changed the title
fweik
approved these changes
Sep 18, 2018
|
Please milestone for Espresso 4.0.1 |
RudolfWeeber
pushed a commit
to RudolfWeeber/espresso
that referenced
this pull request
Oct 15, 2018
Fix MPI on Ubuntu 18.04 with CUDA
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
OpenMPI 2.1.1 has a broken vader BTL (byte-transport layer), requiring us to disable its single-copy mode. The brokenness manifests itself in messages like
and leads to broken MPI communication between multiple ranks on the same machine. This seems to somehow be CUDA-related as Espresso compiled without CUDA support does not have the issue. As Ubuntu hasn't backported the fix from 2.1.3 (https://www.mail-archive.com/users@lists.open-mpi.org/msg32357.html), we have to resort to disabling vader's single-copy mode. This makes vader behave similar to the old sm BTL, which was the default before OpenMPI 2.
This patch has the same effect as setting the environment variable
OMPI_MCA_btl_vader_single_copy_mechanism=noneon OpenMPI 2.0-2.1.2 and 3.0.0.However, we cannot set that from inside Espresso, so we use the MPI_T interface instead.Reported by @pkreissl.