1 Introduction
2 Files needed for this tutorial
- 2.1 Serial Example (SLURM script & Source Code)
- 2.2 Parallel Example (SLURM script & Source Code)
3 Compile and Run in Interactive Session
- 3.1 Compile Programs
- 3.2 Run Programs
  - 3.2.1 Try Using mpirun
4 Run with SLURM Script
- 4.1 Run MPI with Multiple Nodes
- 4.2 Dividing the Workload Further

Introduction

This tutorial is a quick start guide to introduce you to the concepts of running software on chip interactively, or via SLURM scripts, along with how to run your code in parallel. For more in depth tutorials on these concepts see the following:

Before starting this tutorial make sure you are familar with the concepts in https://umbc.atlassian.net/wiki/x/R4BPQg.

NOTE: Whenever you see “userid” this is a dummy username used in the tutorial. When you work through the tutorial, make sure to substitute your username in the relevant places. Similarly, for “pi_userid” this is a dummy PI account name. Please make sure you are referring to the PI account for your case.

Files needed for this tutorial

Create a folder on chip and upload the following files for this tutorial (see https://umbc.atlassian.net/wiki/x/BgA-RQ for more on that).

Serial Example (SLURM script & Source Code)

DOWNLOAD LINK: https://umbc-research.github.io/wiki-code/Quick_Run/run-complexProgram.slurm

DOWNLOAD LINK: https://umbc-research.github.io/wiki-code/Quick_Run/complexProgram.cpp

Parallel Example (SLURM script & Source Code)

DOWNLOAD LINK: https://umbc-research.github.io/wiki-code/Quick_Run/run-MPIcomplexProgram.slurm

DOWNLOAD LINK: https://umbc-research.github.io/wiki-code/Quick_Run/MPIcomplexProgram.cpp

Compile and Run in Interactive Session

Let us begin by starting an interactive session with the following command:

srun --cluster=chip-cpu --account=pi_userid --partition=2021 --qos=normal --time=1:00:00 --mem=5000 --pty $SHELL

Once you have started your interactive seeion load the Intel 2024a compiler and MPI software:

module load intel/2024a

You can check the modules you have loaded with “module list”:

[userid@c21-17 Quick_Run]$ module list                                                                         
                                                                                                                 
Currently Loaded Modules:                                                                                        
  1) slurm/chip-gpu/23.11.4   6) zlib/1.3.1-GCCcore-13.3.0      11) impi/2021.13.0-intel-compilers-2024.2.0      
  2) git/2.33.1               7) binutils/2.42-GCCcore-13.3.0   12) imkl/2024.2.0
  3) shared                   8) intel-compilers/2024.2.0       13) iimpi/2024a
  4) DefaultModules           9) numactl/2.0.18-GCCcore-13.3.0  14) imkl-FFTW/2024.2.0-iimpi-2024a
  5) GCCcore/13.3.0          10) UCX/1.16.0-GCCcore-13.3.0      15) intel/2024a

As you can see, loading the intel2024a compiler adds it to our module list and associated MPI software.

Compile Programs

We compile both programs using the command below. Notice how we use the same icpx compiler, but for the MPI version I just add mpi to the start to make it compatible. Besides that, they are essentially the same thing

To compile Non-MPI file:

icpx -O3 -xHost -qmkl=sequential complexProgram.cpp -o complexProgram

To compile MPI file:

mpiicpx -O3 -xHost -qmkl=sequential MPIcomplexProgram.cpp -o MPIcomplexProgram

Run Programs

Now you have compiled both programs you can use the follwoing commands to run the programs in your interactive session.

Non-MPI Run Command:

./complexProgram

MPI run Command:

./MPIcomplexProgram

Output:

[userid@c21-06 Quick_Run]$ ./complexProgram 
Result: 1e+08 (should be ~100000000)
Elapsed time: 2230076685 ns
[userid@c21-06 Quick_Run]$ ./MPIcomplexProgram 
Global sum: 1e+08 (should be ~100000000)
Elapsed time: 2238208222 ns
Number of processes: 1

As you can see, this makes sense. We are only using one node, and one process per node for the MPI version, which means no distribution of work (i.e., running in serial). This means the MPI and non-MPI versions will both take about the same time to run.

Try Using mpirun

Non-MPI Run Command (same as before):

./complexProgram

MPI run Command:

mpirun ./MPIcomplexProgram

Output:

[userid@c21-06 Quick_Run]$ ./complexProgram 
Result: 1e+08 (should be ~100000000)
Elapsed time: 2228666218 ns
[userid@c21-06 Quick_Run]$ mpirun ./MPIcomplexProgram 
Global sum: 1e+08 (should be ~100000000)
Elapsed time: 2230807479 ns
Number of processes: 1

As you can see, the MPI version still runs in essentially the same time, for the same reason as earlier: it is only using one node, and one processor per node, meaning the speed should be essentially the same as complexProgram.

Run with SLURM Script

Run MPI with Multiple Nodes

In theory, you can do this in the command line, but it is a bit complicated. An easier way to do this is just to use a SLURM script to run the job with the “sbatch” command. You can do this from inside the interactive session above, or directly from the login node.

For comparison, we will also use a slurm file for the non-mpi complex program.

To run with sbatch:

sbatch run-complexProgram.slurm

sbatch run-MPIcomplexProgram.slurm

Now, to read the outputs use the following:

cat outputcomplexProgram.out

Quick Run Guide: How to run things on chip, with/without parallelism, with different compilers, etc.