A Four Part Job

We are now going to combine mkrandfile and xrandfile to construct a PBS job that has four distinct parts:

1.

Prepare a directory on the AVIDD GPFS.

2.

Generate a data file in the directory.

3.

Process the data file.

4.

Clean up the GPFS directory and exit.

This is a simplistic prototype of a typical supercomputer job. The ``generate a data file'' part may include staging the file from, e.g., HPSS, prior to executing other parts of the code. The ``clean up'' part may include uploading a modified data file to HPSS and deleting any unnecessary junk that the job may have generated on the GPFS.

Here is the script:

[gustav@bh1 PBS]$ cat process.sh
#PBS -S /bin/bash
#PBS -N process
#PBS -o process_out
#PBS -e process_err
#PBS -q bg
#PBS -m abe
#PBS -M gustav@indiana.edu
#
# Prepare a directory on the AVIDD GPFS.
[ -d /N/gpfs/gustav ] || mkdir /N/gpfs/gustav
cd /N/gpfs/gustav
rm -f test
# Generate a data file in the directory.
mkrandfile -f test -l 100
# Process the data file.
xrandfile -f test -l 4
# Clean up the GPFS directory and exit.
rm -f test
exit 0
[gustav@bh1 PBS]$

Let us submit the script and inspect the output of the job:

[gustav@bh1 PBS]$ qsub process.sh
13744.bh1.avidd.iu.edu
[gustav@bh1 PBS]$ qstat 13744.bh1.avidd.iu.edu
Job id           Name             User             Time Use S Queue
---------------- ---------------- ---------------- -------- - -----
13744.bh1        process          gustav                  0 R bg              
[gustav@bh1 PBS]$ qstat 13744.bh1.avidd.iu.edu
qstat: Unknown Job Id 13744.bh1.avidd.iu.edu
[gustav@bh1 PBS]$ cat process_out
writing on test
writing 100 blocks of 1048576 random integers
reading test
reading in chunks of size 16777216 bytes
allocated 16777216 bytes to junk
read 419430400 bytes
[gustav@bh1 PBS]$

Well, it all works as expected.

In the following sections we are going to learn how we can submit a multi-stage task like this in the form of several mutually dependent PBS jobs.