The assistant works. Now somebody at the clinic has to keep it working, and that somebody is not a machine-learning engineer. This part is the runbook: what happens daily (nothing), how to update the runtime, a model and the knowledge, what to back up and what not to, what the logs look like when something is wrong, and what a month costs in electricity against the same month on a hosted API. Every command in it was run once on the server and its real output pasted, which is how the warm-up timer's first bug was found.

A console window titled 'coder000 - private AI, operations (Ubuntu in WSL2)'. systemctl list-timers shows ollama-warm.timer with NEXT in 58 s and no LAST yet; systemctl status shows ollama-warm.service loaded, static, inactive (dead), triggered by ollama-warm.timer, with a journal line from an earlier run listing nomic-embed-text at 100% GPU; ollama ps shows an empty table; journalctl shows the runtime starting and answering two requests.
Caught by accident at the right moment: the server had just restarted, the resident model list is empty, and the warm-up timer is 58 seconds from its first fire. Ninety seconds later both models are on the GPU without anyone asking. The first line of each block is a typed label; the output under it is verbatim.

Daily: nothing, because a timer does it

Part 4 added a warm-up script so the first question after a restart is not the slow one. Part 11 makes it automatic: a systemd oneshot service that runs the script, and a timer that fires it 60 seconds after boot and every six hours after that. Installed with a four-line script, verified with systemctl list-timers, and on the WSL server it fired 56 seconds after a boot and loaded both models onto the GPU before anyone asked.

Its first run failed, with both models already loaded. The unit reported an error exit because the last line of the warm-up script calls ollama ps, which resolves the model directory from the HOME variable, and systemd does not set HOME for a service. The panic came after the useful work was done, so the timer looked broken while doing its job. One Environment="HOME=/usr/share/ollama" line in the unit fixed it, with a comment saying why, because the next person to see "panic: $HOME is not defined" in a journal deserves the short version.

Updating things, in order of how often

  • The knowledge, most often: edit the markdown files, run the ingest command, read the table. Unchanged documents are skipped, changed ones re-embedded. Only a change of embedding model forces everything to re-embed.
  • A model, occasionally: ollama pull of the same tag fetches a newer build. Tag the current one first with ollama cp so it can be restored, and after any model change run the evaluation from Part 10 before staff touch it. A model that scores lower on the fifty questions is not an upgrade.
  • The runtime, rarely: re-run the installer; it upgrades in place, ends with "Install complete", leaves the models alone. Then the warm-up.

Backups: what is data and what is a download

The knowledge tables live in the same PostgreSQL as the appointments, so the clinic's existing database backup already covers them; a targeted pg_dump of the two tables is 280 KB for five documents. The markdown files in the repo are the true source and can be re-ingested at any time. Model files are large and re-downloadable, so they are not backed up; the runbook lists the exact tags to pull after a rebuild. The rule the owner needs: the database and the repo are backed up, everything else is reinstalled.

Logs, and a deliberate error

journalctl -u ollama shows one line per request with the status and the time taken. To show what wrong looks like, the runbook requests a model that does not exist and pastes the 404 the runtime answers with. Out-of-memory is the other line worth recognising, and the health check covers its usual cause: ollama ps should show both models at 100% GPU; a model on CPU means something else took the video memory.

What a month costs

From Part 3's measurements: about 35 W idle around the clock, about 120 W above idle while answering. For a clinic asking five hundred questions a day at five seconds each, that is 25.2 kWh of idling and 2.5 kWh of answering, 27.7 kWh a month, about $4 at a placeholder $0.15 per kWh, on top of a server the clinic already runs. The same fifteen thousand questions a month at roughly 800 input and 200 output tokens each, at a placeholder hosted price of $3 and $15 per million tokens, come to about $81 a month, and rise with every new user. Both figures are placeholders that the reader must replace with current prices; the shape of the comparison is the point, and it does not change.

What to tell the clinic owner

  • Nothing about a patient leaves the building; the assistant's port is not reachable from the network.
  • It answers only from the clinic's own documents and shows which one; when they do not say, it says it does not know.
  • Changing what it knows means editing a document and running one command.
  • It costs a few dollars of electricity a month and no subscription.
  • Someone owns the box: a few hours a quarter for updates, and the runbook tells them what to type.

Do it yourself

The runbook is deploy/ollama/OPERATIONS.md at tag private-11; these are its commands in the order a clinic will need them.

  1. Install the warm-up timer once. Two units and a script; the HOME line is the one that took a morning:
    # ollama-warm.service
[Unit]
Description=Warm the Ollama chat and embedding models
After=ollama.service
Requires=ollama.service

[Service]
Type=oneshot
Environment="HOME=/usr/share/ollama"
ExecStart=/usr/local/bin/ollama-warm.sh
TimeoutStartSec=300

# ollama-warm.timer
[Timer]
OnBootSec=60s
OnUnitActiveSec=6h
Unit=ollama-warm.service

[Install]
WantedBy=timers.target
    sudo bash deploy/ollama/install-warm-timer.sh
systemctl list-timers --all | grep warm
  2. Update the knowledge, most often: edit the markdown, run the ingest, read the table. Unchanged documents say skipped.
  3. Update a model, occasionally: keep the current build under a dated tag first, pull, then re-run Part 10 before staff use it:
    ollama cp qwen2.5:14b qwen2.5:14b-2026-09
ollama pull qwen2.5:14b
python tools/private-bench/eval.py qwen2.5:14b
  4. Update the runtime, rarely: re-run the installer, which upgrades in place and leaves the models alone; then ollama --version and the warm-up.
  5. Back up the two knowledge tables with the clinic database, and nothing else; the markdown is the source and the models are downloads:
    docker compose exec db pg_dump -U cliniclive -d cliniclive -t knowledge_document -t knowledge_chunk --data-only | head
  6. Know what the logs look like before you need them. One healthy line, and one provoked error:
    journalctl -u ollama -n 20 --no-pager
curl -s http://127.0.0.1:11434/api/generate -d '{"model":"nope:latest","prompt":"x"}'
# journal: ... | 404 | ... | POST "/api/generate"
  7. Health check, weekly or when someone complains:
    curl -s http://127.0.0.1:11434/api/version
ollama ps        # both models, 100% GPU; a model on CPU means something else took the video memory
  8. Work out the month from your own Part 3 numbers: idle watts times 720 hours, plus answering watts above idle times questions per day times seconds per question times thirty; then the same questions at a hosted per-token price with today's figures, never this post's.

What you should see: a timer that fires a minute after boot with both models on the GPU before anyone asks, a runbook the clinic's IT contact can follow without you, and a cost line with two numbers on it.

The brief that built it, trimmed to the lines that decided the result. Season one put every prompt in a commit message; this season's briefs were written to a coding assistant before each part, and this is what they said.

deploy/ollama/OPERATIONS.md for the clinic: the warm-up timer (oneshot service + timer, 60 s after boot and
every 6 h, an install script), runtime and model updates with the rule to re-run the evaluation after any
model change, knowledge updates, backups (pg_dump of the two tables; markdown is the source; models are not
backed up), logs with a provoked 404, disk, the health check, the monthly cost as a formula with a worked
example against a hosted-token example, and five plain-language points for the owner. Run every command once
on the server and paste the real output.

Perishable facts, as of September 2026: both cost columns use placeholder prices; the electricity and token counts are measured, the prices are not. Ollama 0.34.2; the HOME requirement of ollama ps under systemd is current behaviour and may change.

Model pick: the runbook's structure and the cost model were set in the brief at high effort; the cheaper model executed every command and pasted the outputs, and caught the HOME panic by reading the journal rather than trusting the timer's green status.

What the AI got wrong: the warm-up unit without HOME. And a near-miss worth recording: ollama ps came back empty in the middle of the session and the assistant nearly wrote that removing a tagged copy evicts the running model. The journal showed the real cause, WSL idling and restarting, and the false sentence never reached the document. Check the log before you write the rule.

The meter: the running-cost reading is now complete: 27.7 kWh a month for the worked example, against a per-token bill that only grows. The build meter closes in the retro.

Checkpoint: tag private-11 in the repo: deploy/ollama/OPERATIONS.md, the timer units and the install script. Install the timer, reboot the server, and check that both models are on the GPU within two minutes without anyone asking a question. Next, and last: the retro.