Skip to main content

HPCX

AIStation platform, a virtualization platform built on Docker, suitable for AI training and GPU-accelerated computing.

1. Usage Instructions​

## 1. The shared directory is “/USERNAME_DIRECTORY”. Note that all development environments created under the current user share the same directory.
## 2. Network supports IB. IB drivers exist on the physical machine but are not present in the current image. You can invoke them by specifying the network interface card.
## 3. Since network drivers exist on the physical machine, it is not recommended to compile MPI from source. Instead, use the HPC-X environment to install MPI.

2. Basic Platform Operations​

Refer to the Inspur General User Operation Manual

3. Download HPC-X​

https://developer.nvidia.com/networking/hpc-x ### Download link

4. Installation Steps​

cd hpcx
export HPCX_HOME=$PWD

Compile OpenMPI based on the current environment​

$ tar xfp ${HPCX_HOME}/sources/openmpi-gitclone.tar.gz
$ cd ${HPCX_HOME}/sources/openmpi-gitclone
$ ./configure CC=icc CXX=icpc F77=ifort FC=ifort --prefix=${HPCX_HOME}/ompi-icc \
--with-hcoll=${HPCX_HOME}/hcoll \
--with-ucx=${HPCX_HOME}/ucx \
--with-platform=contrib/platform/mellanox/optimized \
2>&1 | tee config-icc-output.log
$ make -j32 all 2>&1 | tee build_icc.log && make -j24 install 2>&1 | tee install_icc.log

Activate HPC-X​

export HPCX_HOME=$PWD
cd hpcx-v2.12-gcc-MLNX_OFED_LINUX-5-ubuntu18.04-cuda11-gdrcopy2-nccl2.12-x86_64/
source hpcx-init.sh
hpcx_load

## Load via module
module use $HPCX_HOME/modulefiles
module load hpcx

mpirun command parameters​

QUDA_ENABLE_P2P=3 /yangybai11/hpcx-v2.12-gcc-MLNX_OFED_LINUX-5-ubuntu18.04-cuda11-gdrcopy2-nccl2.12-x86_64/ompi/bin/mpirun --allow-run-as-root  -np 16 --host 192.208.79.37:8,192.224.8.14:8  -bind-to none -map-by slot -x LD_LIBRARY_PATH -x HOROVOD_MPI_THREADS_DISABLE=1 -x PATH -mca pml ucx -x NCCL_DEBUG=INFO -x NCCL_TREE_THRESHOLD=0 -x UCX_LOG_LEVEL=info ./hmc -i s1.0_restart_37540.xml -geom 1 2 2 4
## --allow-run-as-root allows root to execute mpirun
## -mca pml ucx Explicitly use UCX with OpenSHMEM 	
## 
/etc/hosts ## When running cross-node jobs, SSH connections between nodes may be required
~/.ssh/known_hosts

Note: This platform's network exists on the physical machine; accessing it within the instance environment may cause issues. Use https://developer.nvidia.com/networking/hpc-x

https://content.mellanox.com/hpc/hpc-x/v2.12/hpcx-v2.12-gcc-MLNX_OFED_LINUX-5-ubuntu18.04-cuda11-gdrcopy2-nccl2.12-x86_64.tbz