This compiling tutorial assumes that you are able to log in to the chip cluster in the UMBC High Performance Computing Facility. For information, how to log in and use your account up to this point, please visit the Getting Started on chip first.

Introduction and Compiling with the Intel Compiler

In this tutorial, we will illustrate how to compile C source code and run the resulting executable on the CPU cluster chip-cpu in chip in the UMBC High Performance Computing Facility. Working on a distributed cluster like chip is fundamentally different from working on a standard server or a personal computer, so please make sure to read and understand this material. We will first start with a classical serial example, and work our way to compiling parallel code using the most important parallel processing library MPI (Message Passing Interface). For more information on MPI see the Wikipedia article. We will assume that you know some basic programming concepts, so the code will not be explained in explicit detail. More details can be found in manual pages on the system that are available for Linux commands (e.g., try “man mkdir”, “man cd”, “man pwd”, “man ls”) as well as for C functions (e.g., try “man fprintf”).

Since we want to use parallel code with MPI, we need to use the Intel compiler suite, since that has MPI available with it. See the Getting Started on chip documentation on how to access chip. This tutorial assumes that you are logged in to your account on chip.

Loading the Intel Compiler

After logging in to chip’s login node (also known as the user node), you need to be on a compute node to compile with the Intel compiler and with MPI, since the module for the Intel compiler suite needs to be loaded (and loading of modules cannot be done on the login node). For more information about modules see https://umbc.atlassian.net/wiki/x/BQBHTw.

To find out the names of the modules available to load, do “module spider Intel“; this command can be issued on the login node or a compute node, but the actual “module load ...” command can only be issued on a compute node.

Before starting an interactive session you can check available resources on the cluster using the sinfo command (see sinfo manual for more information). Here is a sample case:

This shows the available resources by cluster type (CPU/GPU), partition, and if there are available nodes for computation. Once you have confirmed available resources use “srun” to start an interactive session on a node from the “general” partition, substituting “pi_userid” for your relevant account information:

Once you have started the interactive session, you can check which modules are currently loaded:

When you issue this command, the prompt (“userid@c18-04” in the example above) will change to show your username and the compute node you started the interactive session on. You can see that we need to load the Intel compilers. Now issue this command:

This will load the default Intel compilers for chip. You can request a specific version of the compiler by modifying the command slightly (e.g., module load intel/2024a will load the 2024a version of the Intel compiler). If we list the modules again, we can now see that the Intel compiler and corresponding MPI modules are now loaded:

To confirm that an MPI-capable MPI compiler has been loaded, issue the following:

The “which” command looks up the path associated with a command. If a path is returned like our sample case above, you have successfully loaded a compiler. One last preparation step is to create a shell environmental variable to tell the system the Intel compiler’s name:

This finishes the setup to use the Intel compiler suite with the icx C compiler. You have to repeat these steps every time you want to compile with the Intel compiler.

Serial Hello World

In this example, you will compile a simple serial program (a program that runs on one node) to run on the chip cluster. Make sure you have followed all the instructions above and are logged into a compute node with the compiler loaded before proceeding (see https://umbc.atlassian.net/wiki/spaces/faq/pages/edit-v2/1284276235#Loading-the-Intel-Compiler ).

It is a good idea to collect all the files for a project in a directory. This project is a serial version of the classic “Hello, world!” program. Therefore, use the mkdir (= “make directory”) command to create a directory “Hello_Serial” and cd (= “change directory”) into it.

Notice that the command prompt indicates that you are now in the “Hello_Serial” directory. Use the “pwd” (= “print working directory”) command any time to confirm where you are in your directory structure and “ll” (short for “ls -l”) to list the files that are there.

Given below is the C source code for our simple serial example:

DOWNLOAD LINK: https://umbc-research.github.io/wiki-code/Hello_Serial/hello_serial.c

You can list all files to see that the file is present now:

We have shown the prompt in the examples above to emphasize that a command is being issued. When following the examples, your prompt may look a bit different (e.g., your own username will be there instead of “userid”!), but be careful to only issue the command part, not the prompt or the example output.

Once you have saved this code to your directory, you have to compile it before you can execute it, since C is a source code programming language. Compile using the Intel icx compiler with the following command:

The -o option of icx specifies the desired name of the output file; it is customary to use the base filename without extension for an executable in C/C++. If successful, no errors or warnings will appear and an executable hello_serial will have been created, in addition to the source code in hello_serial.c.

Notice that the “x” in the permissions “-rwxrwx—” indicates that hello_serial is an executable. Depending on your terminal emulator, this may also be indicated by the asterisk “*” following its name (the “*” is not part of the filename, it is just an indication of an executable from the ls command). When a file is not an executable (or there is no permission to execute it), a dash “-” appears in place of the “x”; the dashes in “-rw-rw—-” for hello_serial.c confirm that this C source code is not executable in its source code form.

To see how to run your serial executable on the cluster, jump to https://umbc.atlassian.net/wiki/spaces/faq/pages/edit-v2/1325957222#Running-Serial-Hello-World.

Parallel Hello World

In this example, you will compile a simple parallel program (a program that runs on multiple nodes) to run on the chip cluster. Make sure you have followed all the instructions above and are logged into a compute node with the compiler loaded before proceeding (see https://umbc.atlassian.net/wiki/spaces/faq/pages/edit-v2/1284276235#Loading-the-Intel-Compiler).

You may want to create a new directory for this project using “mkdir Hello_Parallel”. Use the link or wget again (see above How to Compile on chip) to save the following code to your directory:

DOWLOAD LINK: https://umbc-research.github.io/wiki-code/Hello_Parallel/hello_parallel.c

This version of the “Hello, world!” program collects several pieces of information at each MPI process: the MPI processor name (i.e., the hostname), the process ID, and the number of processes in our job. Notice that we needed a new header file mpi.h to get access to the MPI commands. We also need to call MPI_Init before using any other MPI commands, and MPI_Finalize is needed at the end to clean up. Compile the code with the following command:

After a successful compilation with no errors or warnings, an executable “hello_parallel” should have been created, which we confirm by “ll”.

To see how to run your parallel executable on the cluster, see https://umbc.atlassian.net/wiki/spaces/faq/pages/edit-v2/1325957222#Running-Paralell-Hello-World.

In this example, we have written output from our MPI program to stdout. As a general guide, stdout and stderr should be used for reporting status information, and not for returning large datasets. If your program does need to write out a lot of data, it would be more appropriate to use file I/O instead.

Logging Which Nodes are Used

In this final example you will compile a parallel program that logs which nodes were used and outputs information in an ordered fashion to file instead in addition to stdout. Make sure you have followed all the instructions above and are logged into a compute node with the compiler loaded before proceeding (see https://umbc.atlassian.net/wiki/spaces/faq/pages/edit-v2/1284276235#Loading-the-Intel-Compiler).

For a parallel program, it is always a good idea to log which compute nodes you have used. We can extend our parallel “Hello, world!” program to accomplish this, namely in addition to printing the information to stdout, we will save it to file. This functionality is contained in a self-contained function nodesused() that you can also copy into other programs and then call from the main program, as shown in the code below:

DOWNLOAD LINK: https://umbc-research.github.io/wiki-code/Nodesused/nodesused.c

You may want to create a new directory for this project using “mkdir Nodesused”. Use the link or wget again (see above How to Compile on chip) to save the above code to your directory.

If you ran the parallel “Hello, world!” above (see https://umbc.atlassian.net/wiki/spaces/faq/pages/edit-v2/1325957222#Running-Paralell-Hello-World ) you will have noticed that the processes reported back in a random order to stdout. This is difficult to read for large numbers of processes, so for the output to file, we have the process with ID 0 receive the greeting message from each other process, in order by process ID, and only Process 0 will write the messages to file.

The code actually creates and writes to two files:

1. The file “nodes_used.log” contains only the process ID and hostname, which is the same information as printed stdout already, but ordered.

2. The file “nodesused_cpuid.log” additionally outputs the CPU ID, that is, the number of the computational core in the two CPUs on the node that the MPI process executed on.

Message sending is accomplished using the MPI_Send() function, and receiving with the MPI_Recv() function. Each process prepares its own message, then execution varies depending on the current process. Process 0 writes its own message first, then receives and writes the others in order by process ID. All other processes simply send their message to process 0. The fprintf function is used to write one line for each MPI process to each of the two output files.

This program is compiled for use with MPI using the following command:

To see how to run your parallel executable of the nodesused program on the cluster, use the same slurm script as above and just replace the jobname and executable name by nodesused, see https://umbc.atlassian.net/wiki/spaces/faq/pages/edit-v2/1325957222#Parallel-Nodesused-Runs.