Page Contents

• Example 1

• Example 2

• Identifiers for GPU cards on the '--constraint' flag

• Common Errors

  • Unavailable GPU cards requested

  • Excessive GPU cards requested

  • Excessive CPU memory requested

Related Pages

• Common Slurm Errors

In a sbatch, srun, or salloc script, the --mem flag refers to the amount of memory requested (and hence allocated) on the CPU of a node, whether the node contains a GPU or not. In other words, specifying the memory using the --mem flag always fetches the CPU memory and never the GPU memory.

If the user has a requirement for the GPU memory, that can be specified and requested in the examples shown below.

Example 1

Suppose an user needs 240 GiB of GPU memory and 500GiB of CPU memory. A quick glance at the Cluster Specifications page suggests that this requirement can be met using a GPU node containing the RTX 8000 GPU cards. There are two such nodes. They can specify the number of GPU cards they need using the --gres flag and the type of GPU card they need (which is available on certain nodes) using the --constraint flag as follows:

in addition to the other flags for running a sbatch job on the chip-gpu.

The srun command for the same could look like

srun --cluster=chip-gpu --account=pi_doit --mem=500G --time=1:00:00 --gres=gpu:5 --constraint='RTX_8000' --pty $SHELL

This will allow slurm to allocate 5 RTX 8000 cards (out of the 8 available) from one of the two nodes containing those cards. Note that the --constraint flag tells slurm that the user needs a node that has the RTX 8000 GPU cards.

The snippet below shows the output of the srun command which verifies that the requested GPU(s) and GPU memory was allocated to the user:

Example 2

Suppose an user needs 96 GiB of GPU memory and 200GiB of CPU memory. A quick glance at the Cluster Specifications page suggests that this requirement can be met using a GPU node containing the RTX 8000 GPU cards or a GPU node containing the L40S GPU cards. There are two nodes with RTX 8000 GPU cards and 10 nodes with L40S cards. They can specify the number of GPU cards they need using the --gres flag and the type of GPU card they need (which is available on certain nodes) using the --constraint flag as follows:

The srun command for the same could look like

srun --cluster=chip-gpu --account=pi_doit --mem=200G --time=1:00:00 --gres=gpu:2 --constraint='RTX_8000' --pty $SHELL

Here, the | in 'RTX_8000|L40S' means that either an RTX_8000 GPU card or a L40S GPU card would suffice for the user, as both of them have 48 GiB of GPU memory.

Note: To specify a GPU card that starts with ‘RTX', use an ‘_' to separate the alphabetical and the numeric part of the GPU card name, i.e. ‘RTX_8000’, or ‘RTX_6000’, or 'RTX_2080TI’

The snippet below shows the output of the srun command which verifies that the requested GPU(s) and GPU memory was allocated to the user:

Identifiers for GPU cards

The table below specifies the identifiers the user should provide with the constraint flag to request a specific GPU:

Common Errors

Listed below are a few common errors when requesting GPU nodes and memory from chip-gpu. Refer to Common Slurm Errors for more.

Unavailable GPU card requested

If an incorrect identifier is provided by the user, slurm will generate a message stating the same, e.g. when the identifier for the GPU card is 'RTX_3000', which is still a GPU card, but it is not available on the chip cluster. The message would look like the following:

Excessive GPU cards requested

In this example, the user is requesting 12 GPU cards from one GPU node, which is an invalid request as the maximum number of GPU cards available on a single node on chip is 8 (i.e. the nodes with the RTX 2080 Ti, RTX 6000, and the RTX 8000 GPU cards).

1. Common Slurm Errors

2. More about the srun, sbatch, and salloc commands

3. Getting Started on chip