Installing Ollama on your server

Ollama is the shortest way to run a model at home: one command to install it, one to pull a model, and it exposes an API the rest of your tools already know how to call.

This guide starts from an Ubuntu Server with Docker on it — see Installing Ubuntu Server and Installing Docker if that is not done yet. Allow twenty minutes, fifteen of which are a download.

First: does it even fit in your card?

That is the question that decides, and it is answered before you install anything. A model that does not fit in your card's memory spills into system memory and the speed collapses — you will not learn that from an error message, but from an answer that takes two minutes to arrive.

What your hardware can run gives the figure for your card, and the same section answers the other way round: what a given model demands.

Without a graphics card

It works, on the small sizes, and it is slow. Better to know that now than to discover it after the install.

docker run -d -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 \
  --name ollama ollama/ollama

Note the 127.0.0.1: in front of the port. The command you will find everywhere says -p 11434:11434, which opens the service to your whole network. We come back to it below, because it is the real trap in this guide.

With an NVIDIA card

Docker does not see your card by default. You have to give it the NVIDIA Container Toolkit, in five steps — this is where everyone gets stuck.

NVIDIA=https://nvidia.github.io/libnvidia-container
KEY=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -fsSL $NVIDIA/gpgkey | sudo gpg --dearmor -o $KEY

curl -s -L $NVIDIA/stable/deb/nvidia-container-toolkit.list \
  | sed "s#deb https://#deb [signed-by=$KEY] https://#g" \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

That sed line looks like copy-paste noise. It rewrites the repository so it is tied to the key you just installed, and without it apt update rejects the repository as unsigned. It is the step half the tutorials online leave out.

Then the container, with the card:

docker run -d --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 \
  --name ollama ollama/ollama

Check that it actually sees it:

docker exec ollama nvidia-smi

If the card does not show up, the container still runs — on the processor, slowly, without saying a word. That is the silent failure of this guide, and this command is the only way to see it.

The real trap: Ollama has no password

There is no account, no token, no authentication of any kind. Whoever reaches port 11434 drives the machine: they can pull models, delete them, and spend your card on their own work.

That is why every command here says -p 127.0.0.1:11434:11434 and not -p 11434:11434. The first listens on the machine itself; the second opens the service to your whole local network — and if your router forwards that port, to the whole internet.

To reach it from another device, two routes, and neither of them is "open the port":

• Tailscale links your devices together without exposing anything;
• a reverse proxy with authentication, if you want to reach it from a browser.

Pull a model and talk to it

docker exec -it ollama ollama pull qwen3.5:9b
docker exec -it ollama ollama run qwen3.5:9b

The first pull is several gigabytes. The second start is instant: the model stays in the ollama volume.

To check the API answers — which is what your other tools will need:

curl http://127.0.0.1:11434/api/tags

What this guide will not tell you

How many tokens per second you will get. It depends on your card's memory bandwidth, on the engine, on the context length and on how long the answers are — that is a benchmark, not a fact, and nobody can promise it for your exact setup.

What we can tell you is what fits and what it costs to run. That is computed, one page per piece of hardware, in the models section.