Slurm tasks

In this section we will do a deeper dive into what a slurm task is and how they relate to resource allocation strategies 1 and 2.

The task is the atomic unit of slurm. It cannot be broken down any further and every slurm job is composed of one or more tasks.

What is a task?

A task is a unit of work or one process that SLURM schedules. One task is usually:

• One run of a program or script

• One MPI rank in a parallel job

Say you run bwa, a popular read mapping tool, with eight threads: bwa mem -t 8. Do you have 8 tasks? No, you have a single task that is running on eight threads and therefore requires 8 CPUs. So you would request:

#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8

In slurm, a single task is invoke by calling srun which is why we use it inside the job script or when starting an interactive job.

Since the task is the atomic unit, resources assigned to the same task must automatically be on the same node. The above example will request 8 CPUs on the same node.

When you do per-resource allocation, you always start by thinking about the number of tasks you will need.

When would you use more than one task?

• If your application uses MPI (Message Passing Interface) each task is one MPI rank. They will be independent processes, but they can communicate with eachother via MPI.

• If you want to run the same application many times, i.e. a Monte Carlo Simulation, you can start each run as a separate task. If you want each task to process a different input file or use different output files, you’ll need to pass task-specific arguments using SLURM environment variables like $SLURM_PROCID or $SLURM_ARRAY_TASK_ID. A minimal toy example of this is shown in parallel_example.sh.

• Workflow managers like Snakemake, Nextflow, or Dask can launch multiple tasks to parallelize steps. You have to specify in the invocation of the tool that there are multiple tasks available. I.e. srun snakemake --cores 20 will let snakemake know it can start up to 20 parallel jobs.

How do I use --ntasks-per-node?

--ntasks-per-node is similar to --ntasks but explicit about how many tasks should run on each node. It only makes sense to use if you have more than 1 node, because if you request 1 node all your tasks must naturally run on that node.

#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4

• You are allocating per-node (strategy 1)

• You want 2 nodes

• You want 4 tasks on each node

• So SLURM will launch a total of 8 tasks (2 × 4) to run in parallel

• --ntasks-per-node is supplementary information for slurm on how you want to divide up the two nodes you have requested and how many tasks actually should run.