What is Ollama?

Ollama is a tool that lets users download, run, and interact with large language models (LLMs) directly on their own machine or server, without needing to call an external API like OpenAI’s. It can be used to run AI models locally and it works with various open-source LLMs (like Meta’s LLaMA family, Mistral, Gemma, etc.).

AI models require GPU acceleration. Therefore, to use ollama, users must use the chip-gpu cluster. Ollama can be used with either srun or sbatch, for both interactive and batch jobs, and the steps below will work for both.

Loading Ollama

To identify what versions of ollama are currently installed on chip, run module avail ollama:

(base) [user1@g24-07 ~]$ module avail ollama

--------------------------------------------------------------------------------------------- Core & Custom Modules ---------------------------------------------------------------------------------------------
   ollama/0.6.5    ollama/0.8.0    ollama/0.9.2    ollama/0.10.1    ollama/0.11.4    ollama/0.11.10    ollama/0.13.5 (D)

  Where:
   D:  Default Module

After identifying your desired version, load it using module load ollama/x.x.x

[user1@g20-06 ~]$ module load ollama/0.13.5

Running the Ollama server

Ollama uses a client-server architecture. The ollama server holds LLM models loaded into memory, and manages system resources. To interact with the server, you use a client to query the server, which then processes your inference and returns an output. This is the basis of client-server architecture

However, when running the ollama serve command without any modifications, the ollama server process will be ran in the foreground. This means it is running in your active terminal window, which prevents you from being able to interact with the terminal while the process is running. Foreground processes are canceled when you hit CTRL + C to cancel. However, this is not ideal.

To continue to utilize the systems resources while running an ollama server, you must run the process in the background. Background processes are independent of your terminal, and allow you to continue to interact with the cluster while running the ollama server.

To start an ollama server as a background process, run:

(base) [user1@g20-06 ~]$ ollama serve > ollama_server.log 2>&1 & 
[1] 624999

Command Breakdown

ollama serve - Starts the ollama server
> ollama_server.log - Redirects the servers output (STOUT) to a log file named ollama_server.log
2>&1 - Captures the servers error output (STERR) and redirects it to the log file
& - Runs the command/process as a background process (without this, the server will start in the foreground)
Output - [1] 624999 - The process information for the ollama server background process
- [1] - the job number assigned by the shell. This is just an internal label so the user can refer to the job with commands like fg %1 or kill %1
- 624999 - the process ID (PID) of the command running in the background. The user can use it with commands like kill 624999 or ps -p 624999

Pulling Models

By default, ollama saves pulled LLM models in the .ollama hidden folder in the users home directory. On chip, users only have 500M of storage space in their home directory. This is not enough space for LLM models, so we need to move the .ollama folder to your research volume, then create a symbolic link pointing to the new location.

(base) [user1@g20-06 ~]$ mv ~/.ollama /umbc/rs/pi_group/users/user1/.ollama
(base) [user1@g20-06 ~]$ ln -s /umbc/rs/pi_group/users/user1/.ollama ~/.ollama

Now, you can pull your desired model with ollama pull $MODEL_NAME, llama3.1 is used in this example:

(base) [user1@g20-06 ~]$ ollama pull llama3.1
pulling manifest 
pulling 667b0c1932bc: 100% ▕████████████████████████████████████████▏ 4.9 GB                         
pulling 948af2743fc7: 100% ▕████████████████████████████████████████▏ 1.5 KB                         
pulling 0ba8f0e314b4: 100% ▕████████████████████████████████████████▏  12 KB                         
pulling 56bb8bd477a5: 100% ▕████████████████████████████████████████▏   96 B                         
pulling 455f34728c9b: 100% ▕████████████████████████████████████████▏  487 B                         
verifying sha256 digest 
writing manifest 
success

Running Models

After starting the ollama server, there are various methods available to interact with the LLM.

Command: `ollama run`

To interact directly with a LLM using ollama itself, run ollama run $MODEL_NAME. This loads the desired model into memory, and opens an interactive chat where you can type prompts for the model.

[user1@g20-06 ~]$ ollama run llama3.1
>>> Send a message (/? for help)

Alternatively, you can pass a prompt to the ollama run command to get a single response instead of an interactive prompt.

[user1@g20-06 ~]$ ollama run llama3.1 "Write me a haiku about GPUs."
Silicon wings rise
Numbers dance on the screen's dark
Computing storm born
[user1@g20-06 ~]$

Curl

Instead of using ollama run, you can use curl to interact with the ollama API in order to submit prompts

[user1@g20-06 ~]$ curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Write me a haiku about GPUs.",
  "stream": false
}'
{"model":"llama3.1","created_at":"2025-08-14T20:45:06.416757511Z","response":"Silicon fire\nCores burn with frantic dance\nPixels, swift delight","done":true,"done_reason":"stop","context":[128006,882,128007,271,8144,757,264,6520,39342,922,71503,13,128009,128006,78191,128007,271,28671,1965,4027,198,34,4692,8395,449,89706,15612,198,39629,11,30462,18454],"total_duration":26393238590,"load_duration":24758767886,"prompt_eval_count":18,"prompt_eval_duration":1424091055,"eval_count":16,"eval_duration":209823359}

Ollama with sbatch

The steps to use ollama listed above for srun are nearly identical when using sbatch. Below is a sample sbatch script that starts the ollama server, and submits a prompt.

#!/bin/bash
#SBATCH --job-name=ollama_llama3.1
#SBATCH --cluster=chip-gpu
#SBATCH --account=pi_group
#SBATCH --gres=gpu:1
#SBATCH --time=1:00:00
#SBATCH --mem=50G
#SBATCH --output=ollama-%j.out
#SBATCH --error=ollama-%j.err

# Load your environment
module load ollama/0.11.4

# Start Ollama server in background
ollama serve > ollama_server.log 2>&1 &

# Wait a bit to make sure the server is running before connecting
sleep 5

# Pull and run llama3.1 with a prompt
ollama run llama3.1 "Write me a haiku about GPUs."

Running an LLM using Ollama on chip