Best Practices: Environment Setup in Slurm sbatch Scripts vs .bashrc
Overview
When running jobs on-chip using Slurm, users often configure software environments via shell startup files such as .bashrc. While this works for interactive terminal sessions, it is generally not recommended for batch jobs submitted through sbatch.
A common confusion is that Slurm batch jobs do not always source .bashrc automatically unless the job explicitly launches an interactive/login shell or manually sources the file. As a result, jobs that depend on settings hidden inside .bashrc may fail, behave inconsistently, or become difficult to debug.
For reproducibility and maintainability, the recommended practice is:
Keep
.bashrcminimal and place job-specific setup directly inside thesbatchscript.
Understanding the Difference Between .bashrc and sbatch
.bashrc
The .bashrc file is primarily intended for interactive shell configuration. It is executed when opening a new interactive Bash session.
Typical uses include the following:
Shell aliases
Prompt customization
Convenience functions
Terminal appearance settings
Lightweight environment preferences
Example:
alias ll='ls -lah'
export EDITOR=nanoThese settings improve interactive usability but are not typically required for running compute jobs.
sbatch Scripts
An sbatch script defines the environment and commands required for a specific Slurm job.
Because batch jobs should be reproducible and self-contained, all required software setup should be explicitly included inside the job script itself.
Typical items include the following:
module loadcommandsConda or virtual environment activation
CUDA configuration
Hugging Face cache settings
Threading environment variables
Project-specific paths
Software version selection
Recommended Best Practice
Put Job-Specific Setup Inside the sbatch Script
Example:
#!/bin/bash
#SBATCH --job-name=train_llm
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --mem=32G
#SBATCH --time=02:00:00
module load Python/3.12.3-GCCcore-13.3.0
source /umbc/rs/mygroup/users/$USER/myenv/bin/activate
export HF_HOME=/umbc/rs/mygroup/users/$USER/hf_cache
export TRANSFORMERS_CACHE=$HF_HOME
export OMP_NUM_THREADS=4
python train.py
This ensures:
The job contains all required dependencies
Another user can reproduce the workflow
The environment does not depend on the hidden shell state
The same script behaves consistently across sessions and nodes
Why Relying on .bashrc Can Cause Problems
1. Slurm Jobs May Not Source .bashrc
Many users assume that environment variables or modules defined in .bashrc are automatically available inside batch jobs. This is not guaranteed.
As a result:
pythonmay point to the wrong versionVirtual environments may not activate
Environment variables may be missing
GPU libraries may fail to load
These issues can be difficult to diagnose because the configuration is hidden outside the job script.
2. Large .bashrc Files Become Difficult to Maintain
Over time, users often accumulate:
Multiple Conda auto-activations
Many
module loadcommandsProject-specific exports
Aliases for old workflows
Hardcoded paths
This can create conflicts between projects and produce inconsistent behavior between login sessions and compute jobs.
A cleaner approach is:
Keep
.bashrclightweightConfigure each project independently inside its own batch script
3. Reproducibility and Collaboration
Self-contained batch scripts are easier to:
Share with collaborators
Reproduce for publications
Debug later
Move between systems or research groups
If all required setup is inside the script, users do not need access to another person’s .bashrc configuration to run the workflow.
What Should Go in .bashrc?necessary.
In general, .bashrc should remain lightweight and focused on improving the interactive shell experience rather than configuring research workflows or job environments.
A minimal .bashrc is easier to maintain and helps avoid conflicts between projects, software stacks, and batch jobs. Keeping it simple also reduces confusion when debugging Slurm jobs, since fewer hidden environment changes are applied automatically during login sessions.
Typical .bashrc usage includes small quality-of-life customizations such as aliases, prompt settings, editor preferences, or simple convenience functions that are broadly useful across all terminal sessions.
Example:
alias ll='ls -lah'
alias gpuq='squeue --me'
export EDITOR=vimThese settings are typically safe because they are lightweight, broadly useful, and not tied to a specific research workflow or job configuration.
What Should NOT Go in .bashrc?
Avoid placing these in .bashrc unless necessary:
Heavy
module loadcommandsConda auto-activation
GPU-specific setup
Project-specific cache paths
should be necessary to goResearch workflow configuration
Anything required only for one project
Examples to avoid:
module load CUDA/12.1
source ~/myproject_env/bin/activate
export HF_HOME=/large/project/cacheThese are better placed directly inside the job script.
Common Rule of Thumb
A useful guideline is the following:
If a setting is required for a specific job to run correctly, place it in the
sbatchscript.
If a setting only improves your interactive terminal experience, place it in
.bashrc.
Notes
Explicit environment setup improves debugging and reduces hidden dependencies.
Job scripts should be portable between systems whenever possible.
Environment activation inside
sbatchscripts is recommended even if similar settings exist in.bashrc.Users should avoid assuming that the login-node shell state automatically transfers into compute jobs.
Keeping environments localized to individual workflows helps prevent conflicts between projects.
Moreover, users should source as few things as possible in .bashrc. Slurm jobs should explicitly define their own environments within the sbatch script itself. This keeps workflows reproducible, easier to debug, easier to share, and more reliable across compute environments.