Historical Job Data in Slurm
To view historical job and usage data on Picotte.
List of Jobs for Certain Time Period
sacct[1] can list completed jobs (including failed ones).
Here is an example showing all jobs for the user juser for the period
Apr 1, 2021 through (and including) Apr 30, 2021. A selection of fields
is specified with the “--format” option.
$ sacct --starttime 2021-04-01 --endtime 2021-05-01 --user juser --format=User,JobID,partition,reqcpus,reqmem,state,start,end,elapsed
User JobID Partition ReqCPUS ReqMem State Start End Elapsed
--------- ------------ ---------- -------- ---------- ---------- ------------------- ------------------- ----------
juser 96651_1 def 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_1.bat+ 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_1.ext+ 4 2Gc COMPLETED 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
juser 96651_5 def 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_5.bat+ 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_5.ext+ 4 2Gc COMPLETED 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
juser 96651_9 def 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_9.bat+ 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_9.ext+ 4 2Gc COMPLETED 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
juser 96651_13 def 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_13.ba+ 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_13.ex+ 4 2Gc COMPLETED 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
juser 96651_17 def 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_17.ba+ 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_17.ex+ 4 2Gc COMPLETED 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
juser 96651_21 def 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_21.ba+ 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_21.ex+ 4 2Gc COMPLETED 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
juser 96651_25 def 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_25.ba+ 4 2Gc OUT_OF_ME+ 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
96651_25.ex+ 4 2Gc COMPLETED 2021-04-02T09:49:47 2021-04-02T09:50:19 00:00:32
...
juser 307745_6 def 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
307745_6.ba+ 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
307745_6.ex+ 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
juser 307745_7 def 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
307745_7.ba+ 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
307745_7.ex+ 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
juser 307745_8 def 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
307745_8.ba+ 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
307745_8.ex+ 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
juser 307745_9 def 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
307745_9.ba+ 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
307745_9.ex+ 1 3900Mc COMPLETED 2021-04-27T14:29:10 2021-04-27T14:29:11 00:00:01
juser 313663 gpu 1 0n COMPLETED 2021-04-29T11:48:35 2021-04-30T03:12:41 15:24:06
313663.batch 1 0n COMPLETED 2021-04-29T11:48:35 2021-04-30T03:12:41 15:24:06
313663.exte+ 1 0n COMPLETED 2021-04-29T11:48:35 2021-04-30T03:12:41 15:24:06
There are many options to sacct, and many possible fields to display.
See the official documentation, linked below in the references.
To see fields that can be displayed, do “sacct -e”.
Usage Data
sreport[2] lists historical usage by various “Trackable Resources” (TRES). In particular, the Billing TRES corresponds to the Service Units (SUs) used for billing: 60 Billing TRES = 1 SU.
Here is an example showing all TRES available.
$ sreport cluster AccountUtilizationByUser Account=somethingPrj tree Start=2021-05-01 End=2021-06-01 --tres ALL
--------------------------------------------------------------------------------
Cluster/Account/User Utilization 2021-05-01T00:00:00 - 2021-05-31T23:59:59 (2678400 secs)
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
Cluster Account Login Proper Name TRES Name Used
--------- -------------------- --------- --------------- -------------- --------
picotte somethingprj cpu 2885
picotte somethingprj mem 0
picotte somethingprj energy 0
picotte somethingprj node 399
picotte somethingprj billing 18842
picotte somethingprj fs/disk 0
picotte somethingprj vmem 0
picotte somethingprj pages 0
picotte somethingprj gres/gpu 414
picotte somethingprj gres/gpu:v100 0
picotte somethingprj juser J. User cpu 2885
picotte somethingprj juser J. User mem 0
picotte somethingprj juser J. User energy 0
picotte somethingprj juser J. User node 399
picotte somethingprj juser J. User billing 18842
picotte somethingprj juser J. User fs/disk 0
picotte somethingprj juser J. User vmem 0
picotte somethingprj juser J. User pages 0
picotte somethingprj juser J. User gres/gpu 414
picotte somethingprj juser J. User gres/gpu:v100 0
Compute Usage and Charge Report
To view compute only usage, i.e. without storage usage, use the
“billing_report” command. Built-in help is available using the -h or
--help flag. Billing report for the current month can also be
displayed, but the data will be partial.
Example usage:
[juser@picotte001]$ billing_report -a somethingPrj -w 2021-04
USAGE REPORT FOR somethingprj ON CLUSTER picotte - April 2021
Rate = $ 0.0123 per SU
Compute usage: 7.435367e+03 SU
Charge: $ 91.46
Per-user usage and charge
Name User ID Usage (SU) Charge
Jane Euler je001 1.955650e+3 $ 24.05
Carl Fred cf123 5.390267e+3 $ 66.30
Ellen Turing et456 8.945000e+1 $ 1.10
See also: URCF official rate schedule