Using MPICH2 Under Maui on CAC's V4 Cluster
MPICH2 is not the default MPI on the CAC V4 cluster, but some codes run better under MPICH. Here's how you compile and run a batch code using MPICH2.
First set environment variables to put MPICH2's directory in your shell's path. This shows how to do it in Bash shell.
export MPICH2=/opt/mpich2/gnu export PATH=$MPICH2/bin:$PATH export LD_LIBRARY_PATH=$MPICH2/lib:$LD_LIBRARY_PATH
Compiling with mpicc, mpicxx, or mpif77 is the easiest way to link properly. If you want to use a different compiler, you can find the correct libraries for linking by looking at those commands. For example, "less `which mpicxx`" would tell you that you need to link to "-lmpich -lpthreads -lrt -luuid". The relevant include and lib directories are $MPICH2/include and $MPICH2/lib.
In your batch script, ensure MPICH2 is in your path and use mpiexec instead of mpirun.
#!/bin/sh #PBS -l walltime=00:59:00,nodes=2 #PBS -A your_project_id #PBS -j oe #PBS -N batchtest #PBS -q v4dev #PBS -S /bin/bash # Turn on echo of shell commands set -x export MPICH2=/opt/mpich2/gnu export PATH=$MPICH2/bin:$PATH export LD_LIBRARY_PATH=$MPICH2/lib:$LD_LIBRARY_PATH NODECNT=`cat $PBS_NODEFILE|wc -l` CORESPERNODE=`grep processor /proc/cpuinfo|wc -l` CORECNT=$((NODECNT * CORESPERNODE)) mpdboot -n $NODECNT --file=$PBS_NODEFILE --verbose # This will print a list of nodes on which I can run MPI programs. # It's just for debugging whether mpd started correctly. mpdtrace -l cd $HOME/dev/lammps/examples/crack mpiexec -n $CORECNT ./lmp_g++ < in.crack mpdallexit
You can cut out the mpdtrace command once you feel things are working.