Fix MPI example not working in IPv6 #2714

gengwg · 2023-02-28T03:48:17Z

This is to fix errors for the MPI example in #2713.

This PR does 2 things:

By updating the Docker base image, the OpenMPI version is effectively upgraded.
Add --mca option to mpiexec. This seems to have fixed the IPv6 issue.

Before:

$ k -n gengwg exec -it lm-mpi-job-mpiworker-0 -- mpirun --version
mpirun (Open MPI) 1.10.2

Besides, getting the mount error in #2713. Example:

Unexpected end of /proc/mounts line `overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/4GCDNOZIAYJDMKBWJVVSMRDLWE:/var/lib/docker/overlay2/l/RJ4EXRGNG4CKKFCJPAANAG6KWQ:/var/lib/docker/overlay2/l/3GCIAIOLS32GIVNV3KMWN6TPNN:/var/lib/docker/overlay2/l/RDG5AJQ3G64UCFYZPHBDGC7PR6:/var/lib/docker/overlay2/l/LZSWVN7NELVR3CTN6D254MGKD6:/var/lib/docker/overlay2/l/OMLRKFEAEDPGWVCU5JGAZVSL3X:/var/lib/docker/overlay2/l/FTXOND2LJMKTD5FQJSXBG6JEWN,upperdir=/var/lib/docker/overlay2/fbfe3afd54f7512357817dde4c887fd8083bbcb9508d3c4265dfb4344f4c'

After:

$ k -n gengwg exec -it lm-mpi-job-mpimaster-0 -- mpirun --version
mpirun (Open MPI) 2.1.1

No mount error any more. Also MPI runs successful:

$ k logs lm-mpi-job-mpimaster-0 -n gengwg
Hello world from processor lm-mpi-job-mpimaster-0, rank 1 out of 2 processors
Hello world from processor lm-mpi-job-mpimaster-0, rank 0 out of 2 processors

Would it be possible for someone to perform a test of this on a cluster that uses only IPv4? (should still be many in current world). This would ensure that backward compatibility is maintained. It's worth noting that our environment consists of hosts and pods that are IPv6 only, so it's a bit hard for me to test IPv4.

gengwg · 2023-02-28T04:24:22Z

/assign @Thor-wl

hwdef · 2023-02-28T06:31:59Z

example/integrations/mpi/Dockerfile

@@ -1,4 +1,4 @@
-FROM ubuntu:16.04
+FROM ubuntu:18.04


can we update this to 22.04?

I was thinking of that too, but I only tested 18.04 so far. I can test 22.04 tomorrow.

I test 22.04 on arm64. It is OK.

I tested 22.04, and confirmed this works on x86_64 IPv6 too.

$ k -n gengwg logs lm-mpi-job-mpimaster-0 Hello world from processor lm-mpi-job-mpimaster-0, rank 0 out of 2 processors Hello world from processor lm-mpi-job-mpimaster-0, rank 1 out of 2 processors $ k -n gengwg get po -w | grep mpi lm-mpi-job-mpimaster-0 1/1 Running 0 4m51s lm-mpi-job-mpiworker-0 1/1 Running 0 4m51s lm-mpi-job-mpiworker-1 1/1 Running 0 4m51s $ k -n gengwg exec -it lm-mpi-job-mpimaster-0 -- mpirun --version mpirun (Open MPI) 4.1.2

hwdef · 2023-02-28T06:32:20Z

example/integrations/mpi/mpi-example.yaml

@@ -23,7 +23,7 @@ spec:
                - |
                  MPI_HOST=`cat /etc/volcano/mpiworker.host | tr "\n" ","`;
                  mkdir -p /var/run/sshd; /usr/sbin/sshd;
-                  mpiexec --allow-run-as-root --host ${MPI_HOST} -np 2 mpi_hello_world;
+                  mpiexec --allow-run-as-root --mca --host ${MPI_HOST} -np 2 mpi_hello_world;


Does this parameter have any effect on ipv4?

I'm not entirely sure. That's why I wanted someone to test it on ipv4. If we don't find someone to help, I can try spining up minikube, and test it there.

I test in kind, with arm64 and ipv4, it works fine.But we'd better explain what this parameter is for

But we'd better explain what this parameter is for

That's a great question, and one that I was also curious about. I am actually new to learning MPI myself, and I have been experimenting with various methods. It turns out that using the "--mca" flag has been successful for me, but I did some further research and discovered that it actually requires a key value pair!

-mca, --mca <key> <value> Send arguments to various MCA modules. See the "MCA" section, below.

https://www.open-mpi.org/doc/v2.1/man1/mpiexec.1.php

I'm confused now. Hope some expert in OpenMPI can provide clarification and help me understand it better. I can also dig more tomorrow. Time for bed now. Thanks for the help!

I retested using 22.04, and it appears to be working without the need for the --mca option.

$ grep mpiexec mpi-example.yaml mpiexec --allow-run-as-root --host ${MPI_HOST} -np 2 mpi_hello_world; sleep 600; $ k -n gengwg logs lm-mpi-job-mpimaster-0 Warning: Permanently added 'lm-mpi-job-mpiworker-0.lm-mpi-job' (ED25519) to the list of known hosts. Warning: Permanently added 'lm-mpi-job-mpiworker-1.lm-mpi-job' (ED25519) to the list of known hosts. Hello world from processor lm-mpi-job-mpiworker-0, rank 0 out of 2 processors $ k -n gengwg exec -it lm-mpi-job-mpimaster-0 -- mpirun --version mpirun (Open MPI) 4.1.2

Let me update this PR to remove the --mca. Image/MPI version update seems sufficient.

Done. Maybe someone could rebuild and upload the image to Docker Hub, such as example-mpi:0.0.3, if it is not built automatically. It would be beneficial for others experiencing the same issue.

After the pr is merged, I can push the new example-mpi image.

After the pr is merged, I can push the new example-mpi image.

please also modify

volcano/example/integrations/mpi/mpi-example.yaml

Line 27 in 9d0a5da

image: volcanosh/example-mpi:0.0.1

Yes, you are right, thanks!

hwdef · 2023-02-28T06:40:34Z

And please make the DCO CI happy

Ensure you have a local copy of your branch by checking out the pull request locally via command line.
In your local branch, run: git rebase HEAD~1 --signoff
Force push your changes to overwrite the branch: git push --force-with-lease origin master

gengwg · 2023-02-28T07:46:41Z

And please make the DCO CI happy

1. Ensure you have a local copy of your branch by [checking out the pull request locally via command line](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/checking-out-pull-requests-locally).

2. In your local branch, run: git rebase HEAD~1 --signoff

3. Force push your changes to overwrite the branch: git push --force-with-lease origin master

Thanks. Just tried. Seems working now.

 DCO / DCO succeeded Feb 27, 2023 in 0s
DCO

All commits are signed off!

hwdef · 2023-02-28T07:53:17Z

/ok-to-test

hwdef · 2023-03-01T07:41:36Z

I think it's better to squash your commits to one commit. The three commits now are too many, which will pollute the commit history

gengwg · 2023-03-01T08:20:03Z

I think it's better to squash your commits to one commit. The three commits now are too many, which will pollute the commit history

I'm not very familiar with squashing commits since I don't use git frequently. However, I'll do some research how to do it. If I'm unable to do so, I'll create a new pull request as a last resort.

Signed-off-by: gengwg <genwg@users.noreply.github.com> Update image to 22.04 Signed-off-by: gengwg <genwg@users.noreply.github.com> remove --mca option Signed-off-by: gengwg <genwg@users.noreply.github.com>

hwdef · 2023-03-01T08:29:11Z

/lgtm

gengwg · 2023-03-01T08:32:50Z

Squashed. I followed the instructions here. I think better than ChatGPT :)

volcano-sh-bot · 2023-03-06T02:54:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Thor-wl

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [Thor-wl]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

volcano-sh-bot requested review from hzxuzhonghu and kevin-wangzefeng February 28, 2023 03:48

volcano-sh-bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Feb 28, 2023

volcano-sh-bot assigned Thor-wl Feb 28, 2023

hwdef reviewed Feb 28, 2023

View reviewed changes

volcano-sh-bot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Feb 28, 2023

Fix MPI example not working in IPv6

ada5110

Signed-off-by: gengwg <genwg@users.noreply.github.com> Update image to 22.04 Signed-off-by: gengwg <genwg@users.noreply.github.com> remove --mca option Signed-off-by: gengwg <genwg@users.noreply.github.com>

volcano-sh-bot assigned hwdef Mar 1, 2023

volcano-sh-bot added the lgtm Indicates that a PR is ready to be merged. label Mar 1, 2023

Thor-wl approved these changes Mar 6, 2023

View reviewed changes

volcano-sh-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 6, 2023

volcano-sh-bot merged commit 7ec3f2d into volcano-sh:master Mar 6, 2023

wangyang0616 mentioned this pull request Mar 6, 2023

Use the latest example-mpi mirror #2722

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix MPI example not working in IPv6 #2714

Fix MPI example not working in IPv6 #2714

gengwg commented Feb 28, 2023

gengwg commented Feb 28, 2023

hwdef Feb 28, 2023

gengwg Feb 28, 2023

hwdef Feb 28, 2023

gengwg Mar 1, 2023

hwdef Feb 28, 2023

gengwg Feb 28, 2023

hwdef Feb 28, 2023

gengwg Feb 28, 2023

gengwg Mar 1, 2023

gengwg Mar 1, 2023

wangyang0616 Mar 3, 2023

hwdef Mar 3, 2023

wangyang0616 Mar 5, 2023 •

edited

Loading

hwdef commented Feb 28, 2023

gengwg commented Feb 28, 2023

hwdef commented Feb 28, 2023

hwdef commented Mar 1, 2023 •

edited

Loading

gengwg commented Mar 1, 2023

hwdef commented Mar 1, 2023

gengwg commented Mar 1, 2023

volcano-sh-bot commented Mar 6, 2023

Fix MPI example not working in IPv6 #2714

Fix MPI example not working in IPv6 #2714

Conversation

gengwg commented Feb 28, 2023

Before:

After:

gengwg commented Feb 28, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangyang0616 Mar 5, 2023 • edited Loading

Choose a reason for hiding this comment

hwdef commented Feb 28, 2023

gengwg commented Feb 28, 2023

hwdef commented Feb 28, 2023

hwdef commented Mar 1, 2023 • edited Loading

gengwg commented Mar 1, 2023

hwdef commented Mar 1, 2023

gengwg commented Mar 1, 2023

volcano-sh-bot commented Mar 6, 2023

wangyang0616 Mar 5, 2023 •

edited

Loading

hwdef commented Mar 1, 2023 •

edited

Loading