Multiple Batch Tasks

From CAC Documentation wiki
Revision as of 17:11, 21 September 2015 by Ad876 (talk | contribs) (Created page with "==Multiple Batch Tasks== You can add the -t switch to tell the scheduler to run multiple copies of your batch file. It's just like telling the printer to print five copies of...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Multiple Batch Tasks

You can add the -t switch to tell the scheduler to run multiple copies of your batch file. It's just like telling the printer to print five copies of your document. This example starts ten tasks. Each knows its task number when it starts. They each are scheduled as separate jobs, so they don't have to start at the same time and generally will not start simultaneously.

#!/bin/sh
#PBS -A AcctNumber
#PBS -l walltime=02:00,nodes=1
#PBS -N taskTest
#PBS -j oe
#PBS -q v4
#PBS -t 10

echo I am job ${PBS_JOBID} and task ${PBS_TASKNUM}.

You can start a thousand tasks this way with one command! If you tried using nsub a thousand times, the scheduler would get flooded, but the -t switch makes it easier. A feature of -t is that you can specify ranges and strides, as in -t 3-5 and -t 1-100,2.

How might I use this switch?

  • Make a list of batch parameters in a file. Use $PBS_TASKNUM to have each task pull that line from the file.
INDATA=parameters.txt
PARAMS=$(awk "{ if (NR==$PBS_TASKNUM) print \$0 }" $INDATA)
  • Work through the entries in a directory.
DATAFILE=$(ls -1 *.dat | awk "{ if (NR==$PBS_TASKNUM) print \$0 }")

The qdel, mjobctl, and showq commands do not agree on whether the multiple tasks are separate jobs or part of a job. You may be able to kill all of the jobs with qdel <jobid>, but you may need to specify each task with qdel <jobid>-1, qdel <jobid>-2, and so on. Use a for loop with a sleep in there so the scheduler doesn't get overloaded.

for i in {1..100}; do qdel 1003172-$i; sleep 0.5; done