Testing parallel file copying

From CAC Documentation wiki
Jump to navigation Jump to search

This example creates some large blank files and copies them simultaneously to nodes in order to test throughput on the cluster's filesystem in an ad hoc way. The command, parallel, is Gnu Parallel.

#!/bin/sh
#PBS -l walltime=00:60:00,nodes=2
#PBS -A shm7_0003
#PBS -j oe
#PBS -N batchtest
#PBS -q v4dev
#PBS -S /bin/bash

# This batch script should test the /v4scratch space by creating
# large files and copying them to scratch. Whee!


# Turn on echo of shell commands
set -x

SCRATCH=/v4scratch/$USER
TRACE=$PBS_O_WORKDIR/time${PBS_JOBID%%.*}.txt

KB=1024
MB=1048576

file_of_size ()
{
  dd if=/dev/zero of=data bs=$2 count=$1
  echo 'size ' $(($2 * $1)) > $TRACE
}

cd $SCRATCH

if df .|grep cac_home>/dev/null
then
  echo Not on scratch, when we should be.
  echo We are on cac_home, which we would trash,
  echo so quit instead.
  exit 2
fi


rm -rf $SCRATCH/data*
file_of_size $((3 * 1024)) MB
NODECNT=$(wc -l < "$PBS_NODEFILE")
TASKCNT=$((8 * NODECNT))

TRACEBASE=$PBS_O_WORKDIR/time${PBS_JOBID%%.*}

echo tostart `date`
seq 1 $TASKCNT|parallel --sshloginfile "$PBS_NODEFILE" $PBS_O_WORKDIR/copy_one.sh $TMPDIR $TRACEBASE
echo tofinish `date`

echo fromstart `date`
seq 1 $TASKCNT|parallel --sshloginfile "$PBS_NODEFILE" $PBS_O_WORKDIR/copy_from.sh $TMPDIR $TRACEBASE
echo fromfinish `date`

And the helper files, copy_one and copy_from:

#!/bin/bash
# We have to pass in these variables because remote ssh means the new
# process does not inherit variables from the parent.
# Start this process in scratch. Pass it the location of TMPDIR.
TMPDIR=$1
TRACEBASE=$2
PID=$3

SCRATCH=/v4scratch/$USER
cd $SCRATCH

# Copy a file under Gnu Parallel from scratch to TMPDIR and back.
TRACE=${TRACEBASE}_${PARALLEL_SEQ}.txt
touch $TRACE

LOCALNAME=$TMPDIR/data${PARALLEL_SEQ}
(/usr/bin/time cp data $LOCALNAME >/dev/null) 2>> $TRACE

Here is copy_from.sh.

#!/bin/bash
# We have to pass in these variables because remote ssh means the new
# process does not inherit variables from the parent.
# Start this process in scratch. Pass it the location of TMPDIR.
TMPDIR=$1
TRACEBASE=$2
PID=$3

SCRATCH=/v4scratch/$USER
cd $SCRATCH

# Copy a file under Gnu Parallel from scratch to TMPDIR and back.
TRACE=${TRACEBASE}_${PARALLEL_SEQ}.txt

LOCALNAME=$TMPDIR/data${PARALLEL_SEQ}

(/usr/bin/time cp $LOCALNAME data${PARALLEL_SEQ} >/dev/null) 2>> $TRACE