Job Script Example 02a Many Input Files

Description

This is similar to Job Script Example 02 Many Input Files, except that the input files are in different subdirectories and all the files themselves have the same name "input.dat".

This example has two steps:

1. create the subdirectories and input files -- done interactively in a login node
2. submit the job

The example script and input files are in:

/mnt/HA/opt/Examples/Example02a

Computation "Program"

As a stand-in for an actual computation, we have a trivial bash script which just prints out the contents of the input file.

#!/bin/bash
### compute.sh

cat input.dat

Input Files

The inputs are created by this bash script:

#!/bin/bash
### make_inputs.sh

### there are 7 elements in T
T=(111 222 333 444 555 666 777)

### there are 4 elements in U
U=(5 7 11 13)

### we create one subdirectory for each (T,U) combination
outdir=out

for t in ${T[@]}
do
    for u in ${U[@]}
    do
        mkdir -p $outdir/T$t/U$u
        ### do stuff here like, cp input${t}${u}.dat $outdir/$t/$u
        ### this line is just an example
        echo "INPUTS: t = $t, u = $u" > $outdir/T$t/U$u/input.dat
    done
done

This creates some directories "T???" each with some subdirectories "U?". Each of these subdirectories has one input file named input.dat which contains the line of text

INPUTS: t = ?, u = ?

Script

Once all the inputs have been created, submit this job script.

#!/bin/bash
#$ -S /bin/bash
#$ -cwd
#$ -j y
#$ -M myname@drexel.edu
#$ -P myrsrchPrj
#$ -l h_rt=0:05:00
#$ -l m_mem_free=128M
#$ -l h_vmem=256M
#$ -t 1:28:1
#$ -q all.q
. /etc/profile.d/modules.sh
module load shared
module load proteus
module load sge/univa

###
### example02a.sh
###

### there are 7 elements in T
T=(111 222 333 444 555 666 777)

### there are 4 elements in U
U=(5 7 11 13)

### Note that the task specification in the header
### must make task IDs that start with 1. We need
### 28 separate tasks, hence:
###     #$ -t 1:28:1

### Most of the following is from the make_inputs.sh script
### except that here we do not copy or create the directories.
### Putting the directory and data creation here would mean
### that ### process would be repeated for each task, all
### writing over the same input files.

### We have one subdirectory for each (T,U) combination
outdir=out

declare -a tu_arr
idx=1
for t in ${T[@]}
do
    for u in ${U[@]}
    do
        echo "idx = $idx, t = $t, u = $u "
        tu_arr[$idx]="${t},${u}"
        ((idx++))
    done
done

echo ""

### this prints all elements of tu_arr
echo "tu_arr = ${tu_arr[*]}"

### this prints the number of elements in tu_arr
echo "no. elems in tu_arr = ${#tu_arr[*]}"

echo ""

### Our program expects to run in the appropriate subdirectories;
### we extract the values t and u from tu_arr
t=$( echo ${tu_arr[$SGE_TASK_ID]} | cut -f1 -d, )
u=$( echo ${tu_arr[$SGE_TASK_ID]} | cut -f2 -d, )

echo "SGE_TASK_ID = $SGE_TASK_ID"
echo "t = $t"
echo "u = $u"

echo ""
echo "- - - - -"
echo "Computation:"

program=`pwd`/compute.sh
cd $outdir/T$t/U$u
$program input.dat