Slurm - Job Script Example 01 Many Input Files
Description
You have many input files to process: maybe you are converting data from one format to another. The file names are in sequential order:
rawdata01.txt
rawdata02.txt
...
rawdata13.txt
The example script plus input files are in
/ifs/opt/Examples/Example01
Script
#!/bin/bash
#SBATCH --partition=def
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=13
#SBATCH --time=00:03:00
#SBATCH --mem=1GB
#SBATCH --array=1-13
. /etc/profile.d/modules.sh
module load shared
module load slurm/picotte
cd /ifs/groups/myGrp/juser
### Since we will have many simultaneous processes on possibly many nodes
### writing to the same directory, we use the BeeGFS filesystem
DATADIR=/beegfs/scratch/juser/Examples/Example01
### This makes the fileid an integer with zeroes in front so that all fileids are the same number of characters
fileid=$(printf %02d $SLURM_ARRAY_TASK_ID)
sed -e 's/hello/goodbye/' ${DATADIR}/rawdata${fileid}.txt > ${DATADIR}/moddata${fileid}.txt