Timeouts and possible causes
SSHing into the cluster
There are no timeouts for SSHing into a login node in the cluster. Assuming a stable internet connection, once a user has SSHed into the cluster, the user should be able to stay there.
Note: If the user’s machine is put to sleep, the SSH may timeout.
srun, sbatch and salloc commands
When using any of the above commands (i.e. srun, sbatch, and salloc) with the corresponding flags to switch to a worker/compute node from a login node, the --time flag specifies the amount of time allocated to the user for that job on the worker node, and the job will be killed after reaching the specified amount of time, and hence the user will also be disconnected from the worker node.
When scheduling jobs on the cluster, the user should request sufficient time in their slurm allocation for your job’s execution. Different jobs will require different amount of time to run. As an example, if the user predicts and requests 1 hour (--time=01:00:00) for a job, and the job does not complete (it is killed) when the requested amount of time has elapsed, the user should request 1 hour and 30 minutes the next time (--time=01:30:00) for the same job. In another example, an user might predict 2 hours - 4 hours to complete a job, and hence they might request 6 hours (--time=06:00:00) for the same job in the slurm file (srun, sbatch, or salloc).
The snippet below shows an example of the message slurm would generate if a job is killed due to exceeding the time limit.
[user1@c21-01 ~]$ srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
slurmstepd: error: *** STEP 224957.0 ON c21-01 CANCELLED AT 2025-07-08T10:43:36 DUE TO TIME LIMIT *** Jupyter Notebooks
Similar to the policy for srun, sbatch and salloc commands, a job running on a Jupyter Notebook (that was started on the cluster via a sbatch slurm job) will be killed and the user will be timed out of the Jupyter Notebook when the time period specified for the sbatch job has elapsed.
Unstable Internet Connection
Remote access to the chip cluster (via SSH) is dependent on a stable internet connection. Please note that a drop in the connection (even for a fraction of a second) will timeout the user’s connection to the cluster, but will not affect the user’s job that is already running on a worker node on the cluster. If the user has started a job on the cluster, it should still be running smoothly even if the user is timed out due to internet issues. Once the user logs back in, they should be able to view the running job and logs of the job on the node it is running on.
It is to be noted that sbatch/salloc tasks will persist, but interactive tasks (i.e. srun) will not when exposed to an unstable internet connection.
Scheduled Maintenance
When a maintenance is scheduled for the cluster, a notification (via email) will be sent out to all users well in advance so that users can prepare for the down-time. Under such circumstances, the cluster will be shut-down at the proposed time, and any user who is logged in to the cluster, will be timed out at the same time. The MoTD (Message of the Day), displayed on the command line when the user SSHes into the cluster, will communicate the proposed shutdown under such circumstances.