This tutorial tutorial demonstrates how to run serial and parallel jobs on the cluster. Before starting this tutorial please make sure you have completed How to Compile on chip first, as you will need those compiled programs for this tutorial.

Introduction

Running a program on chip is different than running one on a standard workstation. When we log into the cluster, we are interacting with the login node. But we would like our programs to run on the compute nodes, which is where the real computing power of the cluster is. We will walk through the processes of running serial and parallel code on the cluster, and then later discuss some of the finer details.

On chip, jobs must be run on the compute nodes of the cluster. You cannot execute jobs directly on the compute nodes yourself; you must request the cluster’s batch system do it on your behalf. To use the batch system, you will submit a special script which contains instructions to execute your job on the compute nodes. When submitting your job, you specify a partition (group of nodes, e.g., 2018, 2021, or 2018; see https://umbc.atlassian.net/wiki/x/AQB6Sg for more) and a QOS (a classification that determines what kind of resources your job will need; see https://umbc.atlassian.net/wiki/x/D4DwRQ for more). Your job will wait in the queue until it is “next in line”, and free processors on the compute nodes become available. Once a job is started, it continues to run until it either completes (with or without error) or reaches its time limit, in which case it is terminated by the scheduler.

During the runtime of your job, your instructions will be executed across the compute nodes. These instructions will have access to the resources of the nodes on which they are running. Notably, the memory, processors, and local disk space (/scratch space; see Storage for more). Note that the /scratch space is cleared after every job terminates.

The batch system (also called the scheduler or work load manager) used on chip is called SLURM (or Slurm), which is short for Simple Linux Utility for Resource Management.

Interacting with the SLURM Scheduling System

The examples below on this page are designed for running code on the CPU cluster such as the C programs in How to Compile on chip.

There are several basic commands you need to know to submit jobs, cancel them, and check their status. These are:

• sinfo – view the current status of the queues

• sbatch – submit a job to the batch queue system

• squeue – check the current jobs in the batch queue system

• scontrol – find out details about a particular job

• scancel – cancel a job

scancel

The first command we will mention is “scancel”. If you have submitted a job that you no longer want, you should be a responsible user and kill it. This will prevent resources from being wasted, and allows other users’ jobs to run. Jobs can be killed while they are pending (waiting to run), or while they are actually running. To remove a job from the queue or to cancel a running job cleanly, use the scancel command with the identifier of the job to be deleted, for instance:

Substitute the job identifier in the example above, “999999”, for the one you wish to cancel. The job identifier can be obtained from the job listing from squeue (see https://umbc.atlassian.net/wiki/spaces/faq/pages/edit-v2/1325957222#squeue ) or immediately after using batch (see https://umbc.atlassian.net/wiki/spaces/faq/pages/edit-v2/1325957222#sbatch), when you originally submitted the job (also below). See “man scancel” for more information.

sbatch

Now that we know how to cancel a job, we will see how to submit one. You can use the “sbatch” command to submit a script to the queue system:

In this example, run-nodesused-n2ppn4mpi.slurm is the script we are sending to the slurm scheduler. We will see shortly how to formulate such a script. Notice that sbatch returns a job identifier. We can use this to kill the job later if necessary (as in the scancel example above; see https://umbc.atlassian.net/wiki/spaces/faq/pages/edit-v2/1325957222#scancel), or to check its status. For more information, see “man sbatch”.

squeue

You can use the “squeue” command to check the status of jobs in the batch queue system. Here is an example of the basic usage on chip-cpu and for the user “userid”; to see your jobs, replace userid with your username on chip:

The most interesting column is the one titled “ST” for “status”. It shows what a job is doing at this point in time. The state “PD” (“pending”) indicates that the job has been queued. When enough free processor cores become available, it will change to the “R” (“running”) state and begin running. You may also see a job with status “CG” or “CF”, which means it is completing (such as still writing stdout and stderr), and about to exit the batch system. Other statuses are possible too, see “man squeue”. Once a job has exited the batch queue system, it will no longer show up in the squeue display.

We can also see several other pieces of useful information. The TIME column shows the current walltime used by the job up to the present time. For example, job 999999 has been running for 1 second so far. The NODES column shows the number of nodes used by the job and the NODELIST column shows which compute node(s) has(/have) been assigned to the job. For job 999999, the 2 nodes are c18-41 and c18-42, which get contracted together as c18-[41-42].

scontrol

After submitting a job while it is queued, during a job, and for a limited period of time after the job has finished (a few hours), the “scontrol” command can give access to the full details of the job. See “man scontrol” for more information; here we demonstrate basic output with “show job”, using the job ID from above as our example again:

This is very dense output, but if reading carefully, you can find all details of the job, such as number of nodes, node list, time limit, directory of the job, slurm script used, stdout and stderr files, etc.

sinfo

The “sinfo” command also shows the current status of the batch system, but from the point of view of the Slurm partitions. Here is a sample output that can show how many nodes in which partition are available:

To see more details of available equipment, use sinfo with options specified using format codes such as:

The format of each field is "%[[.]size]type[suffix]":

• “size”—Minimum field size. If no size is specified, whatever is needed to print the information will be used.

• “.”— Indicates the output should be right justified and size must be specified. By default output is left justified.

In this example we are asking sinfo to display the names of nodes used, the number of CPUs per node (N), size of memory per node in MB (m), available features on nodes (f), and any generic resources (G; which for chip will list GPUs available on GPU-type nodes). See “man sinfo” for all available format codes. Using our example above gives:

The key use of this output is to know how many nodes have what type of equipment exactly and how to spell their names to request them in your srun or sbatch commands.

Running Serial Hello World

This section assumes you have already compiled the serial “Hello, world!” example (see How to Compile on chip). Now we will see how to run it in several different ways.

Interactive run on a compute node

The most obvious way to run the program is interactively, that is, from the Linux command line on the compute node, where you are while compiling.

We can see the reported hostname that confirms that the program ran on the compute node c18-01 while we were running an interactive session.

Batch run using srun without a Slurm script

For jobs that take more than a few seconds, interactive running is not really appropriate. The srun command reserves a compute node for your job, runs the job there, and then releases the compute node again, so others can use it. A most basic srun command can be issued without a slurm script (see next sub-section), meaning that you issue the srun command from the Linux command line. However, note carefully that this srun command works only from the login node, not if you are in an interactive session on a compute node! This is as opposed to the sbatch command in the following sub-section that can be issued either from the login node or a compute node.

Batch run using sbatch with a Slurm script

To submit a batch job, it is best to assemble all srun options in a Slurm file and then use sbatch with this file. Download the Slurm script to your workspace where the executable “hello_serial” is already located, using either the link or wget (see How to Compile on chip).

DOWNLOAD LINK: https://umbc-research.github.io/wiki-code/Hello_Serial/run-hello_serial.slurm

The flags used in the Slurm script are described below:

• job-name: sets the string that is displayed as the name of the job in squeue.

• output/error: set the file names for capturing standard output (stdout) and standard error (stderr), respectively.

• cluster: sets the cluster (CPU or GPU) that the job will run on. Here we are requesting CPU nodes.

• account: the PI account which the job will associated with.

• partition: chooses the “general” partition of the CPU cluster to request for the job to run on. The QOS flag requests the short queue, since this particular “Hello, world!” job should only run for a few seconds or less.

• time: provides a more precise estimate of the maximum possible time for the job to take. After a job has reached its time limit, it is stopped by the scheduler. This is done to ensure that everyone has a fair chance to use the cluster. It is good practice to slightly overestimate this to ensure your job does not get terminated before completion.

• mem: requested memory per node for the job. Similar to the time flag it is good practice to slightly overestimate this to ensure your job does not run out of memory while running.

• nodes: the total number of nodes requested; set to one for this serial job example.

• ntasks-per-node: number of MPI tasks per node; set to one for this serial job example.

We are now ready to submit our job to the scheduler. To accomplish this, use the sbatch command as follows:

If the submission is successful, the sbatch command returns a job ID, here 222222. We can use this to check the status of the job (squeue), or delete it (scancel) if necessary. To check on running jobs by user “userid” in this example, use the squeue command like

Notice that this is actually NOT the job that we just submitted, see the job ID! Rather, this is showing the job of my interactive session itself from which the job was submitted. The issue is that this “Hello, world!” job just takes fractions of a second, so we are typically not able to catch it in squeue while it is running. But looking at the files in the directory now, we see that the files slurm.err and slurm.out exist, so the job definitely ran. If slurm.err is not empty (i.e., it contains error messages), check the contents carefully as something may have gone wrong. The file slurm.out contains our stdout output; it should contain the “hello world” message from our program.

Running Parallel Hello World

This section assumes you have already compiled the parallel “Hello, world!” example (see How to Compile on chip). Now we will see how to conduct a parallel run across multiple compute nodes. Download the Slurm script to your workspace where the executable “hello_parallel” is already located, using either the link or wget (see How to Compile on chip).

DOWNLOAD LINK: https://umbc-research.github.io/wiki-code/Hello_Parallel/run-hello_parallel-n2ppn4mpi.slurm

The Slurm script above is very similar to the serial slurm script in the options for sbatch set at the top of the file, except that we specify two nodes with the “--nodes=2” flag, and four processes per node with “--ntasks-per-node=4”. These are just examples. You can choose numbers of nodes from one to however many are available and processes per node from one to how many cores there on one node; e.g, on a 2018 node, there are two 18-core CPUs for a total of 36 cores (see https://umbc.atlassian.net/wiki/x/EQDcT for more).

A major difference of a parallel slurm script is the use of “mpirun” in front of the executable in the last line of the slurm script. There are also some extra lines before the mpirun line. The “module load” command is used to load the necessary Intel MPI libraries to run the program. This needs to be the same modules you used when compiling (see How to Compile on chip and https://umbc.atlassian.net/wiki/x/BQBHTw for more). The command we use here loads multiple modules simultaneously.

The mpirun option “-print-rank-map” prints the hostnames of the nodes assigned and a list of the MPI ranks on each node to stdout; this is perfectly optional, you do not have to use this option. The two lines (those starting with “unset” and “export”) before the mpirun line fix the MPI processes to the assigned cores, which may make the job more efficient; these lines are also optional, you do not have to use them.

You can now submit the script to the batch queue system:

The parallel Hello World program does not run long, but we managed to capture some output in squeue:

This time, squeue shows the job running briefly, with 1 second elapsed, before the job again completes quickly and is gone from the queue with the second output of squeue. Notice again that the job with the name “bash” is an interactive shell that can be seen along with the parallel Hello World job.

We check the output that we obtained now:

We see that the error file “slurm.err” is empty again, indicating that no error occurred. The output file “slurm.out” lists the MPI ranks and the compute node that each ran on, here c18-41 for MPI ranks 0, 1, 2, 3 and c18-42 for MPI ranks 4, 5, 6, 7. Clearly, the output is mixed up and in random order. This is to be expected, as several output streams write to the same file. Regardless, we clearly see that two nodes were used and that each node had four MPI processes per node running. The first two lines of slurm.out are caused by the “-print-rank-map” option to mpirun; they show the hostname and the MPI ranks run on each node.

Parallel Nodesused Runs

This section assumes you have already compiled the parallel “Nodesused” example (see How to Compile on chip). This program is similar to the parallel Hello world program, but it logs which nodes were used and outputs information in an ordered fashion to file instead in addition to stdout. Download the Slurm script to your workspace where the executable “nodesused” is already located, using either the link or wget (see How to Compile on chip).

DOWNLOAD LINK: https://umbc-research.github.io/wiki-code/Nodesused/run-nodesused-n2ppn4mpi.slurm

The submission script reads for the nodesused program, with minor changes to the job-name and the mpirun line. Submit the script with sbatch:

We can now examine the output:

Notice that the node numbers confirm that the job was run on two nodes with four processes per node again. As before for the parallel “Hello, world!” program, the order of output lines to stdout is random. However, the listing of the file “nodesused.log” shows that our code in the nodesused() function ordered the output by the MPI process IDs:

The output file “nodesused_cpuid.log” shows similar output, but also shows the CPU ID in addition to the other information.