What to do if I run out of space in my chip home directory?

What to do if I run out of space in my chip home directory?

Common Errors related to home directory storage

[$USER@c18-01 ~]$ pip install torch Defaulting to user installation because normal site-packages is not writeable Collecting torch Downloading torch-2.6.0-cp39-cp39-manylinux1_x86_64.whl (766.7 MB) |████████████████████████████████| 766.7 MB 2.6 kB/s ... Collecting mpmath<1.4,>=1.1.0 Downloading mpmath-1.3.0-py3-none-any.whl (536 kB) |████████████████████████████████| 536 kB 132.4 MB/s WARNING: Additional context: user = True home = None root = None prefix = None ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device: '/home/$USER/.local/lib/python3.9/site-packages/nvidia' [$USER@c18-01 ~]$
[$USER@c18-01 ~]$ mkdir newDirectory mkdir: cannot create directory ‘newDirectory’: No space left on device [$USER@c18-01 ~]$
[$USER@c18-01 ~]$ touch aNewFile.txt touch: cannot touch 'aNewFile.txt': No space left on device [$USER@c18-01 ~]$

Identify what is taking up space

By running the following BASH command in your home directory, you’ll be given an estimate of the storage used by your home directory.

[$USER@c18-01 ~]$ df -h . Filesystem                         Size  Used Avail Use% Mounted on nfs.iss:/ifs/data/chip/home/$USER  500M  384K  500M   1% /home/$USER [$USER@c18-01 ~]$ 

This home directory is not full – it’s using about 1% of its allocated space: 500M.

 

By running the following BASH command in your home directory, you’ll be given a listing of files and directories along with their size. In the example below,

[$USER@c18-01 ~]$ du -sh ~/* .[^.]* | sort -h 24K .bash_history 24K .bash_logout 24K .bash_profile 24K .bashrc 24K .emacs ... 24K .viminfo 104K .ssh

In the same home directory, you can see the sizes of each of the files. None of these are large enough to be causing a storage issue within this home directory.

If you find that you are approaching the 500M limit, or if you are receiving errors like the ones shown above, there are a few things you can do.

 

Things you can do

1. Configure the software you’re using to write data to your research volume

By default Conda sets up your environment and installs packages in /home/$USER/.conda. We have set envs_dirs to /umbc/rs/$group/users/$USER/conda_envs to store all of the environments and pkgs_dirs to /umbc/rs/$group/users/$USER/conda_pkgs to store all of the packages that conda downloads.

If .conda is taking the highest storage, by running the command you can clear the directory

[$USER@chip ~]$ mv .conda /umbc/rs/$group/users/$USER/conda_envs [$USER@c18-01 ~]$ rm -rf .conda

If .vscode-server is the directory occupying a high amount of storage. Please refer to this documentation:

How to connect to chip with VSCode?

By default, pip installs packages to /home/$USER/.local/lib/python3.xx/site-packages. Your home directory has a limited storage thus, the packages installed in .local will fill it up. You can change the install directory by following the instruction here : How to change where pip installs packages?

If Ollama is creating large files or directories in your home directory that are contributing to space issues, consider these general strategies:

Use Symbolic Links: Identify the specific directory that Ollama is using for data storage in your home directory (e.g., ~/.ollama or similar). You can then move this directory to your research volume and create a symbolic link from its original location in your home directory to the new location in your research volume.

Utilize Scratch Space for Temporary Data: For any large, temporary data generated by Ollama during job execution, consider directing it to the /scratch directory on the compute node. Remember that data in /scratch is temporary and will be deleted after your job completes.

If MATLAB's hidden directories or generated files are consuming your home directory space, consider these general strategies:

Configure MATLAB Preferences: While specific commands for MATLAB are not provided in the sources, many applications allow you to configure default save locations or temporary file directories within their settings or preferences. If MATLAB has such an option, direct its output or temporary files to your research volume (/umbc/rs/$GROUP/users/$USER).

Use Symbolic Links: If direct configuration within MATLAB isn't feasible or sufficient, you can use the symbolic link method described for Ollama. Identify the large directory MATLAB is creating (e.g., ~/.matlab, ~/.MATLAB or similar) in your home directory. Move this directory to your research volume and then create a symbolic link back to its original path in your home directory. This will allow MATLAB to continue referencing the expected path while the actual data resides in your larger research volume.

Utilize Scratch Space for Temporary Data: For intermediate computations or large temporary data sets generated by MATLAB during a job, consider directing them to the /scratch directory. Remember to move any necessary output from /scratch to your permanent research volume before the job terminates.

For Other Common Storage Errors: Common Storage Errors

2. Use Symbolic Links to redirect data to your research volume

Once you’ve identified the directory (or hidden directory) that’s taking up space in your home directory, you can use the following steps to free-up space in your home directory while still enabling the related software to connect with that directory.

  1. Move the directory to your research volume

Moving directories can be destructive. Make sure that the mv command shown below won’t overwrite important data at the target location

[username@chip ~] mv ~/.largeDirectory /umbc/rs/${GROUPNAME}/users/${USER}/.largeDirectory
  1. Create a symbolic link from your home directory to the recently moved directory

[username@chip ~] ln -s /umbc/rs/${GROUPNAME}/users/${USER}/.largeDirectory ~/.largeDirectory
  1. Test

[username@chip ~] cd ~/.largeDirectory [username@chip .largeDirectory] pwd -P /umbc/rs/${GROUPNAME}/users/${USER}/.largeDirectory

Example:

# Identify full home directory [$USER@chip ~]$ df -h . Filesystem                         Size  Used Avail Use% Mounted on nfs.iss:/ifs/data/chip/home/$USER  500M  500M  384K 100% /home/$USER # Identify large subdirectories [$USER@chip ~]$ du -sh ~/* .[^.]* | sort -h 24K .bash_history 24K .bash_logout 24K .bash_profile 24K .bashrc 104K .ssh 497M .cache # .cache is the large subdirectory in this example # Move large subdirectory to research volume [$USER@chip ~]$ mv ~/.cache /umbc/rs/$GROUP/users/$USER/.cache # Create symbolic link to point to new location [$USER@chip ~]$ ln -s /umbc/rs/$GROUP/users/$USER/.cache ~/.cache

3. Delete data

This option stands to be the most destructive/disruptive if you make a mistake. DoIT Research Computing staff will never delete data from your home or research volume directories. Be careful when removing, copying, and moving data. Posture on Sensitive Data