/
Common Slurm Errors

Common Slurm Errors

Page Contents

Related Pages

The examples below document common errors that an user would encounter when trying to submit interactive slurm jobs to the chip cluster using the srun command. Note that the same errors would be encountered when the erroneous flags and/or their erroneous values are provided in an sbatch file or an salloc command (refer to the More about the srun, sbatch, and salloc commands page to learn more).

Invalid Values or Formats for flags

Error: The account flag value is invalid (ex: pi_abcd is not a valid PI group on chip)

[user1@chip ~]$ srun --cluster=chip-cpu --account=pi_abcd --partition=general --qos=normal --time=1:00:00 --mem=50000M --pty $SHELL srun: error: Unable to allocate resources: Invalid account or account/partition combination specified

Resolution: Provide a valid name for the --account flag. To check which PI group(s) an user belongs to, they can run the groups command once SSHed into chip.

Providing the incorrect value for the --partition flag can also raise the same error. Refer to the chip Partitions and Usage for more details on allowed values for the --partition flag. See the example below.

 

Error: Invalid combination of values for the --account and --partition flags.

[user1@chip ~]$ srun --cluster=chip-cpu --account=pi_group2 --partition=2024 --qos=normal --time=01:00:00 --mem=5000 --pty $SHELL srun: error: Unable to allocate resources: Invalid account or account/partition combination specified

Resolution: Since pi_group2 isn’t a contributor, they cannot access the 2024 partition, and it is an invalid combination of values for the --account and --partition flags. To access the 2024 partition, the --account flag should be set to a value corresponding to PI who is a contributor.

 

Error: Invalid Partition Specification

[user1@chip ~]$ srun --cluster=chip-cpu --account=pi_group --partition=new --qos=normal --cpus-per-task=4 --time=1: 00:00 --mem=50000M --pty $SHELL srun: error: invalid partition specified: new srun: error: Unable to allocate resources: Invalid partition name specified

Resolution: Provide the allowed values for the --partition flag. Refer to the chip Partitions and Usage to learn more.

Error: The time requested exceeds the allowed limit.

[user1@chip ~]$ srun --cluster=chip-cpu --account=pi_group --partition=general --qos=normal --time=1111:00:00 --mem=50000M --pty $SHELL srun: error: QOSMaxWallDurationPerJobLimit srun: error: Unable to allocate resources: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)

Resolution: Provide an amount of time based on the value for the --qos flag, which can be found in the chip-cpu QOS Restrictions page, e.g. when the value for the --qos flag is normal, the upper bound on the amount of time requested is 4 hours or --time=04:00:00

 

Error: Invalid Time Specification

[user1@chip ~]$ srun --cluster=chip-cpu --account=pi_group --partition=general --qos=normal --time=1h30m --mem=50000M --pty $SHELL srun: error: Invalid --time specification

Resolution: Provide the amount of time in either:

  1. HH:MM:SS: --time=01:30:00 - To request 1 hour, 30 minutes, or

  2. Minutes: --time=90 - To request 90 minutes, or

  3. Days-Hours: --time=1-12:00:00 - To request 1 day, 12 hours.

 

Error: Invalid Memory Specification

[user1@chip ~]$ srun --cluster=chip-cpu --account=pi_group --partition=general --mem=8GB --time=01:00:00 --pty $SHELL srun: error: Invalid memory specification

Resolution: Provide units for memory in either:

  1. M - For megabytes, --mem=8000M

  2. G - For gigabytes, --mem=8G

  3. No units - Defaults to megabytes, --mem=8000

 

Error: Invalid Cluster Specification

[user1@chip ~]$ srun --cluster=chip-cpu-gpu --account=pi_group --partition=general --qos=normal --cpus-per-task=4 --time=1:00:00 --mem=50000M --pty $SHELL srun: error: No cluster 'chip-cpu-gpu' known by database. srun: error: 'chip-cpu-gpu' can't be reached now, or it is an invalid entry for --cluster. Use 'sacctmgr list clusters' to see available clusters.

Resolution: Provide the allowed values for the --cluster flag, which are

  1. chip-cpu, or

  2. chip-gpu

Formatting Error when writing flags and their values

Error: An extra space was provided on either side of the equal to sign between the name of the flag and the value of the flag for the cluster flag. This is true for all flags.

[user1@chip ~]$ srun --cluster = chip-cpu --account=pi_group --partition=general --qos=normal --time=01:00:00 --mem=50000M --pty $SHELL srun: error: No cluster '=' known by database. srun: error: '=' can't be reached now, or it is an invalid entry for --cluster. Use 'sacctmgr list clusters' to see available clusters.

Resolution: Remove the extra spaces between the name of the flag, i.e. --cluster, the equal to sign, and the value of the flag, i.e. chip-cpu. After removal, it should look like --cluster=chip-cpu.

Invalid Combination of flags

Error: The --gres flag was provided for requesting CPU resources, which is an invalid request.

[user1@chip ~]$ srun --cluster=chip-cpu --account=pi_group --partition=general --gres=gpu:1 --qos=normal --time=01:00:00 --mem=50000M --pty $SHELL srun: error: Unable to allocate resources: Invalid generic resource (gres) specification

Resolution: Remove the --gres flag and its value when requesting CPU resources on chip, i.e. when --cluster=chip-cpu. The --gres flag should be provided only when requesting a GPU resource, i.e. when the value for the --cluster flag is chip-gpu.

 

Error: The --partition flag or the --qos flag or both those flags were provided for requesting GPU resources, and this is an invalid request. The --partition and --qos flags should only be provided when requesting a CPU resource, i.e. when the value for the --cluster flag is chip-cpu.

[user1@chip ~]$ srun --cluster=chip-gpu --account=pi_group --partition=general --gres=gpu:1 --time=01:00:00 --mem=50000M --pty $SHELL srun: error: invalid partition specified: general srun: error: Unable to allocate resources: Invalid partition name specified
[user1@chip ~]$ srun --cluster=chip-gpu --account=pi_group --gres=gpu:1 --qos=normal --time=01:00:00 --mem=50000M --pty $SHELL srun: error: Unable to allocate resources: Invalid qos specification

Resolution: Remove --partition=general flag and its value from the first example, and remove --qos=normal flag and its value from the second example.

Invalid Node Configurations

It is to be noted that there are a large number of ways the below error (srun: error: Unable to allocate resources: Requested node configuration is not available) can be generated, and it depends on the combinations of values for the --mem, --cpus-per-task, --gres, --nodes, and --constraint flags.

Error: The CPU memory requested is beyond the value permitted.

[user1@chip ~]$ srun --cluster=chip-cpu --account=pi_group --partition=general --qos=normal --time=1:00:00 --mem=50000G --pty $SHELL srun: error: Memory specification can not be satisfied srun: error: Unable to allocate resources: Requested node configuration is not available

Resolution: Provide a value for the CPU memory that is within the limits of what is available for a given worker node. Refer to the Cluster Specifications to learn more about the allowed values for CPU memory.

 

Error: The number of CPU cores requested is not available on any node (it is too large).

[user1@chip ~]$ srun --cluster=chip-cpu --account=pi_group --partition=general --qos=normal --cpus-per-task=999 --time=1:00:00 --mem=50000M --pty $SHELL srun: error: Unable to allocate resources: Requested node configuration is not available

Resolution: Provide a value for the number of CPU cores that is within the limits of what is available for the chip cluster. Refer to the Cluster Specifications to learn more about the allowed values for the number of CPU cores.

 

Error: The number of GPU cards requested is not available on any node (it is too large).

[user1@chip ~]$ srun --cluster=chip-gpu --account=pi_group --cpus-per-task=4 --gres=gpu:14 --time=1:00:00 --mem=50000M --pty $SHELL srun: error: Unable to allocate resources: Requested node configuration is not available

Resolution: Provide a value for the number of GPU cards that is within the limits of what is available for the chip cluster. Refer to the Cluster Specifications to learn more about the allowed values for the number of GPU cards.

 

Error: The number of GPU cards requested is not available on the type of node requested (identified by the type of GPU card requested).

[user1@chip ~]$ srun --cluster=chip-gpu --account=pi_group --cpus-per-task=4 --gres=gpu:5 --constraint=L40S --time=1:00:00 --mem=50000M --pty $SHELL srun: error: Unable to allocate resources: Requested node configuration is not available

Resolution: Provide a value for the number of GPU cards that is compatible with the type of node specified by the GPU card. In this case, for all nodes containing L40S GPU cards, which the user requested, there are only 4 GPU cards available per node. Hence the value of the --gres flag should be no more than gpu:4. Nodes with different GPU architectures can have different amounts of GPUs per node. Refer to the Cluster Specifications to verify the maximum number of GPUs for your target hardware.