Prolog and Epilog Guide
Slurm supports a multitude of prolog and epilog programs. Note that for security reasons, these programs do not have a search path set. Either specify fully qualified path names in the program or set the PATH environment variable. The first table below identifies what prologs and epilogs are available for job allocations, when and where they run.
|
Parameter |
Location |
Invoked by |
User |
When executed |
|
Prolog (from slurm.conf) |
Compute or front end node |
slurmd daemon |
SlurmdUser (normally user root) |
First job or job step initiation on that node (by default); PrologFlags=Alloc will force the script to be executed at job allocation |
|
PrologSlurmctld (from slurm.conf) |
Head node (where slurmctld daemon runs) |
slurmctld daemon |
SlurmctldUser |
At job allocation |
|
Epilog (from slurm.conf) |
Compute or front end node |
slurmd daemon |
SlurmdUser (normally user root) |
At job termination |
|
EpilogSlurmctld (from slurm.conf) |
Head node (where slurmctld daemon runs) |
slurmctld daemon |
SlurmctldUser |
At job termination |
This second table below identifies what prologs and epilogs are available for job step allocations, when and where they run.
|
Parameter |
Location |
Invoked by |
User |
When executed |
|
SrunProlog (from slurm.conf) or srun --prolog |
srun invocation node |
srun command |
User invoking srun command |
Prior to launching job step |
|
TaskProlog (from slurm.conf) |
Compute node |
slurmstepd daemon |
User invoking srun command |
Prior to launching job step |
|
srun --task-prolog |
Compute node |
slurmstepd daemon |
User invoking srun command |
Prior to launching job step |
|
TaskEpilog (from slurm.conf) |
Compute node |
slurmstepd daemon |
User invoking srun command |
Completion job step |
|
srun --task-epilog |
Compute node |
slurmstepd daemon |
User invoking srun command |
Completion job step |
|
SrunEpilog (from slurm.conf) or srun --epilog |
srun invocation node |
srun command |
User invoking srun command |
Completion job step |
By default the Prolog script is only run on any individual node when it first sees a job step from a new allocation; it does not run the Prolog immediately when an allocation is granted. If no job steps from an allocation are run on a node, it will never run the Prolog for that allocation. This Prolog behaviour can be changed by the PrologFlags parameter. The Epilog, on the other hand, always runs on every node of an allocation when the allocation is released.
The task prolog is executed with the same environment as the user tasks to
be initiated. The standard output of that program is read and processed as
follows:
export name=value
sets an environment variable for the user task
unset name
clears an environment variable from the user task
print ...
writes to the task's standard output.
The above functionality is limited to the task prolog script.
Unless otherwise specified, these environment variables are available to all of the programs.
- CUDA_MPS_ACTIVE_THREAD_PERCENTAGE Specifies the percentage of a GPU that should be allocated to the job. The value is set only if the gres/mps plugin is configured and the job requests those resources. Available in Prolog and Epilog only.
- CUDA_VISIBLE_DEVICES Specifies the GPU devices that should be allocated to the job. The value is set only if the gres/gpu or gres/mps plugin is configured and the job requests those resources. Note that the environment variable set for the job may differ from that set for the Prolog and Epilog if Slurm is configured to constrain the device files visible to a job using Linux cgroup. This is because the Prolog and Epilog programs run outside of any Linux cgroup while the job runs inside of the cgroup and may thus have a different set of visible devices. For example, if a job is allocated the device "/dev/nvidia1", then CUDA_VISIBLE_DEVICES will be set to a value of "1" in the Prolog and Epilog while the job's value of CUDA_VISIBLE_DEVICES will be set to a value of "0" (i.e. the first GPU device visible to the job). Available in Prolog and Epilog only.
- SLURM_ARRAY_JOB_ID If this job is part of a job array, this will be set to the job ID. Otherwise it will not be set. To reference this specific task of a job array, combine SLURM_ARRAY_JOB_ID with SLURM_ARRAY_TASK_ID (e.g. scontrol update ${SLURM_ARRAY_JOB_ID}_{$SLURM_ARRAY_TASK_ID} ...); Available in PrologSlurmctld and EpilogSlurmctld only.
- SLURM_ARRAY_TASK_ID If this job is part of a job array, this will be set to the task ID. Otherwise it will not be set. To reference this specific task of a job array, combine SLURM_ARRAY_JOB_ID with SLURM_ARRAY_TASK_ID (e.g. scontrol update ${SLURM_ARRAY_JOB_ID}_{$SLURM_ARRAY_TASK_ID} ...); Available in PrologSlurmctld and EpilogSlurmctld only.
- SLURM_ARRAY_TASK_MAX If this job is part of a job array, this will be set to the maximum task ID. Otherwise it will not be set. Available in PrologSlurmctld and EpilogSlurmctld only.
- SLURM_ARRAY_TASK_MIN If this job is part of a job array, this will be set to the minimum task ID. Otherwise it will not be set. Available in PrologSlurmctld and EpilogSlurmctld only.
- SLURM_ARRAY_TASK_STEP If this job is part of a job array, this will be set to the step size of task IDs. Otherwise it will not be set. Available in PrologSlurmctld and EpilogSlurmctld only.
- SLURM_CLUSTER_NAME Name of the cluster executing the job.
- SLURM_JOB_GPUS GPU IDs allocated to the job (if any). Available in the Prolog only.
- SLURM_JOB_ACCOUNT Account name used for the job. Available in PrologSlurmctld and EpilogSlurmctld only.
- SLURM_JOB_CONSTRAINTS Features required to run the job. Available in Prolog, PrologSlurmctld and EpilogSlurmctld only.
- SLURM_JOB_DERIVED_EC The highest exit code of all of the job steps. Available in EpilogSlurmctld only.
- SLURM_JOB_EXIT_CODE The exit code of the job script (or salloc). The value is the status as returned by the wait() system call (See wait(2)). Available in EpilogSlurmctld only.
- SLURM_JOB_EXIT_CODE2 The exit code of the job script (or salloc). The value has the format <exit>:<sig>. The first number is the exit code, typically as set by the exit() function. The second number is the signal that caused the process to terminate if it was terminated by a signal. Available in EpilogSlurmctld only.
- SLURM_JOB_GID Group ID of the job's owner. Available in PrologSlurmctld and EpilogSlurmctld only.
- SLURM_JOB_GROUP Group name of the job's owner. Available in PrologSlurmctld and EpilogSlurmctld only.
- SLURM_JOB_ID Job ID. CAUTION: If this job is the first task of a job array, then Slurm commands using this job ID will refer to the entire job array rather than this specific task of the job array.
- SLURM_JOB_NAME Name of the job. Available in PrologSlurmctld and EpilogSlurmctld only.
- SLURM_JOB_NODELIST Nodes assigned to job. A Slurm hostlist expression. scontrol show hostnames can be used to convert this to a list of individual host names. Available in PrologSlurmctld and EpilogSlurmctld only.
- SLURM_JOB_PARTITION Partition that job runs in. Available in Prolog, PrologSlurmctld and EpilogSlurmctld only.
- SLURM_JOB_UID User ID of the job's owner.
- SLURM_JOB_USER User name of the job's owner.
- SLURM_SCRIPT_CONTEXT
Identifies which epilog or prolog program is currently running.
The value is one of the following:
- prolog_slurmctld
- epilog_slurmctld
- prolog_slurmd
- epilog_slurmd
- prolog_task
- epilog_task
- prolog_srun
- epilog_srun
- SLURM_WCKEY User name of the job's wckey (if any). Available in PrologSlurmctld and EpilogSlurmctld only.
Plugin functions may also be useful to execute logic at various well-defined points.
SPANK is another mechanism that may be useful to invoke logic in the user commands, slurmd daemon, and slurmstepd daemon.
Failure Handling
If the Epilog fails (returns a non-zero exit code), this will result in the node being set to a DRAIN state. If the EpilogSlurmctld fails (returns a non-zero exit code), this will only be logged. If the Prolog fails (returns a non-zero exit code), this will result in the node being set to a DRAIN state and the job requeued in a held state (unless nohold_on_prolog_fail is configured in SchedulerParameters). If the PrologSlurmctld fails (returns a non-zero exit code), this will cause the job to be requeued. Only batch jobs can be requeued. Interactive jobs (salloc and srun) will be cancelled if the PrologSlurmctld fails.
Based upon work by Jason Sollom, Cray Inc. and used by permission.
Last modified 24 March 2020