Using MPICH2

Using MPICH2 Under Maui on CAC's V4 Cluster

MPICH2 is not the default MPI on the CAC V4 cluster, but some codes run better under MPICH. Here's how you compile and run a batch code using MPICH2.

First set environment variables to put MPICH2's directory in your shell's path. This shows how to do it in Bash shell.

export MPICH2=/opt/mpich2/gnu
export PATH=$MPICH2/bin:$PATH
export LD_LIBRARY_PATH=$MPICH2/lib:$LD_LIBRARY_PATH

Compiling with mpicc, mpicxx, or mpif77 is the easiest way to link properly. If you want to use a different compiler, you can find the correct libraries for linking by looking at those commands. For example, "less `which mpicxx`" would tell you that you need to link to "-lmpich -lpthreads -lrt -luuid". The relevant include and lib directories are $MPICH2/include and $MPICH2/lib.

In your batch script, ensure MPICH2 is in your path and use mpiexec instead of mpirun.

#!/bin/sh
#PBS -l walltime=00:59:00,nodes=2
#PBS -A your_project_id
#PBS -j oe
#PBS -N batchtest
#PBS -q v4dev
#PBS -S /bin/bash

# Turn on echo of shell commands
set -x

export MPICH2=/opt/mpich2/gnu
export PATH=$MPICH2/bin:$PATH
export LD_LIBRARY_PATH=$MPICH2/lib:$LD_LIBRARY_PATH

NODECNT=`cat $PBS_NODEFILE|wc -l`
CORESPERNODE=`grep processor /proc/cpuinfo|wc -l`
CORECNT=$((NODECNT * CORESPERNODE))

mpdboot -n $NODECNT --file=$PBS_NODEFILE --verbose

# This will print a list of nodes on which I can run MPI programs.
# It's just for debugging whether mpd started correctly.
mpdtrace -l

cd $HOME/dev/lammps/examples/crack

mpiexec -n $CORECNT ./lmp_g++ < in.crack

mpdallexit

You can cut out the mpdtrace command once you feel things are working.