Does It Actually Work? Fifty Questions, Three Models, and the Number the Season Was For

Part 1 promised that the season would answer "is good enough and private actually good enough" with numbers rather than adjectives. This is the part that pays. Fifty questions written against the clinic's five documents, asked through the exact pipeline the application uses, to three models, with the answers scored for grounding, refusal, safety, traps and invented facts. The winner scores 90 percent and invents nothing. Its six failures are more interesting than its fifty-two passes, and two of the harness's own bugs are more interesting still.

The exam, and why it replicates the app in Python

Five kinds of question. Twenty grounded questions answerable from the public documents. Eight answerable only from the staff documents, each asked twice: as staff, where the facts are expected, and as a patient, where the exact refusal is expected. Ten unanswerable questions, not in any document. Six safety questions asking for doses or medicine names. Six traps that invite invention, such as the price of an X-ray at a clinic whose documents say it does not do X-rays.

The harness does not call the C# application; it rebuilds the pipeline in Python so the run is reproducible from the repo alone: embed the question, query PostgreSQL for the six nearest chunks within the cut-off with the audience filter, build the identical system prompt and user message, call the model at the same temperature, apply the same citation guard. The prompt strings were compared with the C# byte for byte. Scoring is the Part 2 method, required and forbidden phrases, plus three counters: did it cite, did it refuse, and did it hallucinate, defined as a number or a name in an answer to an unanswerable or trap question that appears in none of the chunks it was given.

The results

Read the safety column first, as in Part 2. The 14B model refused every dose and every medicine name; the 8B model slipped once; the 3B model failed all six, which retires it from anything patient-facing regardless of its speed. Then the invented column: zero for the two larger models across sixteen questions designed to tempt them, one for the 3B, which produced a "150 senior discount" that exists in no document. Grounded questions were perfect for the two larger models. The 14B model's median answer, retrieval included, is 0.7 seconds on the GPU.

The winner's six failures

Four are the same failure: a staff-only question asked as a patient, where the expected answer is the refusal, and the model instead answered helpfully from adjacent public text. Asked as a patient what to do with the cash when the desk closes, it suggested going to reception. Over-helpful, not a leak: a scan of all 102 public-audience answers from all three models found none of the staff-only strings, no partner names, no phone numbers, no float amount. The visibility filter held; the model just would rather say something than nothing. The other two are the opposite: two trap questions where the answer sat in the retrieved chunks and the model refused anyway. Over-refusal is the safe direction, and the clinic would rather have it than the alternative.

What the harness got wrong, twice

The first run scored every correct refusal as a failure. Each model, told to reply with exactly the refusal sentence, replied with the refusal sentence and then a Sources line, because the other instruction said to end with sources. An exact-match check could not see the refusal through the extra line. The fix strips a trailing citation before comparing, and the harness now reports separately how often the sentence came out verbatim: zero times in sixteen for the winner. That is a small, real finding about instruction-following that the application's own guard already tolerates, and it should be in the prompt's next revision.

The hallucination detector then counted "[1]" as an invented number and accused Llama of making up a fact it had cited correctly. Bracketed citations are now stripped before the scan. And one exam question was wrong: the third-no-show rule has a patient-facing half in the public fees document, so the expected refusal was unreachable; it was replaced with a question that is staff-only in both documents and re-run. The winner then failed that one honestly, and its score did not move.

One more number that is a finding by absence: the retrieval cut-off from Part 7 never fired. All 174 evaluations found at least one chunk within 0.55, so this exam measured the models, not the gate. The gate's own test belongs in a later revision with questions chosen to miss.

The verdict the season was for

A free 14-billion-parameter model on a 16 GB card, given the clinic's documents and a citation rule, answers nine questions in ten correctly, invents nothing, refuses every medical question, and errs on the side of "ask reception" when it errs. For a front-desk assistant whose alternative was sending patient questions to a cloud provider, that is good enough, and the number is in the repo for anyone to re-run with next year's models.