Skip to content

Slurm - Job Script Example 02a Many Input Files in Separate Directories with Python

Description♯

You have multiple input files in multiple directories. The paths are:

  /ifs/groups/myGrp/Inputs/1/data.txt
  /ifs/groups/myGrp/Inputs/2/data.txt
  /ifs/groups/myGrp/Inputs/3/data.txt
  /ifs/groups/myGrp/Inputs/4/data.txt
  /ifs/groups/myGrp/Inputs/5/data.txt
  /ifs/groups/myGrp/Inputs/6/data.txt
  /ifs/groups/myGrp/Inputs/7/data.txt
  /ifs/groups/myGrp/Inputs/8/data.txt
  /ifs/groups/myGrp/Inputs/9/data.txt

You have a Python script which reads one of those data files.

You want to run nine distinct copies of the Python script separately on each of those data files.

Python Script♯

#!/usr/bin/env python3
import os

task_id = os.environ['SLURM_ARRAY_TASK_ID']
print('task_id = {}'.format(task_id))

datafile = '/ifs/groups/myGrp/Inputs/{}/data.txt'.format(task_id)
with open(datafile, 'r') as f:
    for line in f:
        print(line)

Slurm Job Script♯

#!/bin/bash -l
#SBATCH --array=1-9

python3 arr_test.py