Difference between revisions of "Multiple Batch Tasks"

From CAC Documentation wiki
Jump to navigation Jump to search
(Created page with "==Multiple Batch Tasks== You can add the -t switch to tell the scheduler to run multiple copies of your batch file. It's just like telling the printer to print five copies of...")
 
Line 14: Line 14:
 
</source>
 
</source>
  
You can start a thousand tasks this way with one command! If you tried using nsub a thousand times, the scheduler would get flooded, but the -t switch makes it easier. A feature of -t is that you can specify ranges and strides, as in -t 3-5 and -t 1-100,2.
+
You can start a thousand tasks this way with one command! BUT PLEASE DON'T. You might crash the scheduler! Think about it: if you tried using qsub a thousand times, the scheduler would get flooded. The -t switch just makes it easier to cause such a flood.
 +
 
 +
A nice feature of -t is that you can specify ranges and strides, as in -t 3-5 and -t 1-100,2.
  
 
How might I use this switch?
 
How might I use this switch?
Line 22: Line 24:
 
PARAMS=$(awk "{ if (NR==$PBS_TASKNUM) print \$0 }" $INDATA)
 
PARAMS=$(awk "{ if (NR==$PBS_TASKNUM) print \$0 }" $INDATA)
 
</source>
 
</source>
* Work through the entries in a directory.
+
* Work through the entries in a directory, again by line number.
 
<source lang=bash>
 
<source lang=bash>
 
DATAFILE=$(ls -1 *.dat | awk "{ if (NR==$PBS_TASKNUM) print \$0 }")
 
DATAFILE=$(ls -1 *.dat | awk "{ if (NR==$PBS_TASKNUM) print \$0 }")

Revision as of 18:06, 18 January 2017

Multiple Batch Tasks

You can add the -t switch to tell the scheduler to run multiple copies of your batch file. It's just like telling the printer to print five copies of your document. This example starts ten tasks. Each knows its task number when it starts. They each are scheduled as separate jobs, so they don't have to start at the same time and generally will not start simultaneously.

#!/bin/sh
#PBS -A AcctNumber
#PBS -l walltime=02:00,nodes=1
#PBS -N taskTest
#PBS -j oe
#PBS -q v4
#PBS -t 10

echo I am job ${PBS_JOBID} and task ${PBS_TASKNUM}.

You can start a thousand tasks this way with one command! BUT PLEASE DON'T. You might crash the scheduler! Think about it: if you tried using qsub a thousand times, the scheduler would get flooded. The -t switch just makes it easier to cause such a flood.

A nice feature of -t is that you can specify ranges and strides, as in -t 3-5 and -t 1-100,2.

How might I use this switch?

  • Make a list of batch parameters in a file. Use $PBS_TASKNUM to have each task pull that line from the file.
INDATA=parameters.txt
PARAMS=$(awk "{ if (NR==$PBS_TASKNUM) print \$0 }" $INDATA)
  • Work through the entries in a directory, again by line number.
DATAFILE=$(ls -1 *.dat | awk "{ if (NR==$PBS_TASKNUM) print \$0 }")

The qdel, mjobctl, and showq commands do not agree on whether the multiple tasks are separate jobs or part of a job. You may be able to kill all of the jobs with qdel <jobid>, but you may need to specify each task with qdel <jobid>-1, qdel <jobid>-2, and so on. Use a for loop with a sleep in there so the scheduler doesn't get overloaded.

for i in {1..100}; do qdel 1003172-$i; sleep 0.5; done