Slurm - Job Script Example 02 Many Input Files
Description
This is similar to Slurm - Job Script Example 01 Many Input Files, except that the files are not named sequentially. Instead, the needed file names are listed in a separate file.
The example script and input files are in:
/ifs/opt/Examples/Example02
Input File
The input file is named list_of_files.txt. Contents are:
2KontuepFego.txt
atph7QuodsId.txt
glyrydrivUs5.txt
hidAjyonOct2.txt
ikDugOdcayp4.txt
irdaikIbDik8.txt
JuiberAnNup1.txt
KrighwennAr6.txt
mepViavejub7.txt
NidgitOtElm0.txt
rosivPicdon9.txt
scomghovJer1.txt
SwiviphEpur5.txt
tyWocaibyav3.txt
VoryifuttEk1.txt
Whan8Harhij6.txt
wivErtAcper3.txt
yabhavnekIb9.txt
To create the files with the appropriate content:
[juser@picotte001 SlurmTest]$ for fn in $(cat list_of_files.txt) ; do echo "hello" > $fn ; done
Script
File names are listed in a file
This script demonstrates a useful pattern when the input arguments are not numerically sequential. Create an array variable[1] to store the arguments to be passed, and index into that array using the SLURM_ARRAY_TASK_ID.
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=18
#SBATCH --time=00:03:00
#SBATCH --mem=1GB
#SBATCH --partition=def
#SBATCH --array=1-18
### NOTE: you must know the number of files at the time of sbatch
declare -a filenames=( $( cat list_of_files.txt ) )
taskid=$(printf %02d $SLURM_ARRAY_TASK_ID)
sed -e 's/hello/goodbye/' ${filenames[$( expr $SLURM_ARRAY_TASK_ID - 1 )]} > moddata${taskid}.txt
Input files are sequential but not listed in a separate file
This is similar to the above, but the files are named in some sequential manner.
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=18
#SBATCH --time=00:03:00
#SBATCH --mem=1GB
#SBATCH --partition=def
#SBATCH --array=1-18
### NOTE: you must know the number of files at the time of qsub
. /etc/profile
module load shared
module load slurm/picotte
declare -a filenames=( $( /bin/ls -1 *.input ) )
taskid=$(printf %02d $SLURM_ARRAY_TASK_ID)
sed -e 's/hello/goodbye/' ${filenames[$( expr $SLURM_ARRAY_TASK_ID - 1 )]} > moddata${taskid}.txt