Error: could not connect to ollama server, run 'ollama serve' to start it from docker run --rm ollama/ollama run qwen2.5:0.5b means the container ran the Ollama client with no server inside it. Start a server container first with docker run -d, then run the model in it with docker exec <container> ollama run <model>.

The image's entrypoint is /bin/ollama and its default command is serve. In this test the words after the image name took the place of serve, so the container ran the client and never started a server.

The error

PS> cmd /c "docker run --rm --name fix9-old-run1 ollama/ollama run qwen2.5:0.5b 2>&1"; echo exit=$LASTEXITCODE
Error: could not connect to ollama server, run 'ollama serve' to start it
exit=1

Other client commands gave the same answer in a fresh container:

PS> cmd /c "docker run --rm --name fix9-old-run2 ollama/ollama list 2>&1"; echo exit=$LASTEXITCODE
Error: could not connect to ollama server, run 'ollama serve' to start it
exit=1

Why it happens

PS> docker image inspect ollama/ollama --format "{{json .Config.Entrypoint}} {{json .Config.Cmd}}"
["/bin/ollama"] ["serve"]

With no arguments the container runs /bin/ollama serve and is a server. With run qwen2.5:0.5b after the image name it runs /bin/ollama run qwen2.5:0.5b, the client, and that container has no server process for the client to talk to. The client gave up with the message and exit code 1, and --rm removed the container.

The message reads the same as on a Linux or WSL machine where the service is stopped, but the cause here is the way the container was started: the same image, started with no arguments, ran a server that answered ollama list below.

The fix

PS> docker run -d --name fix9-old-srv --gpus all -v "...\fixes9\ollama-docker\models:/root/.ollama/models" -p 127.0.0.1:15976:11434 ollama/ollama
26f792b615d8aa2e63e3072e7dd653b128ce9ec0ae968761756494f62e6fe5c1
PS> docker exec fix9-old-srv ollama list
NAME                 ID              SIZE      MODIFIED
qwen3:0.6b           7df6b6e09427    522 MB    3 days ago
...
qwen2.5:0.5b         a8b0c5157701    397 MB    3 days ago
all-minilm:latest    1b226e2802db    45 MB     3 days ago
PS> docker exec fix9-old-srv ollama run qwen2.5:0.5b "Name one planet. One word."
Earth.
PS> docker exec fix9-old-srv ollama ps
NAME            ID              SIZE      PROCESSOR    CONTEXT    RUNNER      UNTIL
qwen2.5:0.5b    a8b0c5157701    483 MB    100% GPU     4096       llamacpp    4 minutes from now

The first container is the server, started in the background with its default command. docker exec runs the client inside that same container, where it found the server and the models. The models folder here is a copy mounted from the host, so nothing was pulled; the path is shortened. The --gpus all flag is there so the model ran on the card, as ollama ps shows.

How it was reproduced

The cached ollama/ollama:latest image (client 0.40.8) on Docker Desktop with engine 29.9.3 and the WSL 2 backend, Windows 11. The failing commands were docker run --rm ... ollama/ollama run qwen2.5:0.5b and ... ollama/ollama list; the working pattern used a server container with a copied models folder holding qwen2.5:0.5b. The container names are the test's own; the model was stopped with ollama stop and the container removed with docker rm -f -v.

Frequently asked

Why does docker run ollama/ollama run model say could not connect to ollama server?
In this test the arguments after the image name took the place of the image's default command, serve. The container ran only the client, which found no server and exited with code 1.
How do I run a model with the Ollama Docker image?
Start a server container with docker run -d and the ollama/ollama image, then run docker exec with that container's name followed by ollama run and the model name.
Do I need to run ollama serve inside the Docker container?
Not by hand. With no arguments after the image name the container runs /bin/ollama serve, as docker image inspect shows; run client commands in it with docker exec.

For the same message on Linux or WSL with no container involved, see could not connect to ollama server on Linux and WSL. More decoded errors in the Fixes category.