How to update UCX while keeping OpenMPI unchanged

[kingold5]

Hi,

the system administrator has updated newer version UCX/1.11.1 for me. When using the OpenMPI, UCX still reports the error "ucp_worker.c:1836 UCX ERROR too many ep configurations: 16 (max: 16)" for me, which should be solved by the newer version. I used OpenMPI/4.1.1 configured with --with-ucx=UCX/1.10.1, and the newer UCX/1.11.1 was loaded with module load (LD_LIBRARY_PATH was set to UCX/1.11.1).

So my questions are, how can I know which version of UCX being used in OpenMPI? Is it possible to assign OpenMPI to a newer version of UCX without recompiling OpenMPI?

P.S. only srun could be used, no mpiexec or mpirun on this system.

Best regards

[yosefe]

OmpiMPI is could be compiled with "rpath" in <prefix>/lib/openmpi/mca_pml_ucx.so which points to specific UCX location (use chrpath command to check it).
In this case need to rebuild OpenMPI with new UCX version, remove the rpath tag manually from OpenMPI UCX components (<prefix>/lib/openmpi/*_ucx.so), or use LD_PRELOAD to load the new UCX version.

[William-Mou]

Hi @yosefe, Thank you for this discussion and the answer! It has been very beneficial, and the solution worked for me.
I want to share my easy script for development purposes: by naming this script as mpirun-ucx and executing it, it allows using a new version of UCX without modifying MPI.

#!/bin/bash

# Set the directory for UCX libraries
ucx_lib_directory="$HOME/pp-hw4/ucx-1.15.0/build/opt/lib"

# Build the LD_PRELOAD string
# It searches for specific UCX library files (libucp.so.0, libuct.so.0, libucm.so.0, libucs.so.0)
# in the specified directory, then concatenates their paths, separated by colons
ld_preload_paths=$(find "$ucx_lib_directory" -name 'libucp.so.0' -o -name 'libuct.so.0' -o -name 'libucm.so.0' -o -name 'libucs.so.0' | tr '\n' ':' | sed 's/:$//')

# Set the LD_PRELOAD environment variable with the paths of the UCX libraries
export LD_PRELOAD=$ld_preload_paths

# Run the mpirun command, passing all provided arguments
# "$@" is a special variable that holds all the arguments passed to the script
mpirun -x LD_PRELOAD=$LD_PRELOAD "$@"

[yosefe]

Cool! I am using a similar script myself

[William-Mou]

@yosefe Thanks for your response! I'm a TA for a parallel programming course at National Tsing Hua University, Taiwan. Having participated in SC22 and ISC23, I plan to introduce a lab for students to develop observability code (something like UCX_LOG_LEVEL=info) in UCX, helping them grasp its architecture.
(If I am fortunate enough to receive your valuable suggestions, please feel free to let me know!)