# Case: local and private document Q&A

**Status: executed.** Raw records: `docs/tests/local-document-qa/runs/*.json` and
`docs/tests/local-document-qa/runs-llamacpp/*.json`. Published, translated and
linked to these records: <https://globalaiproductindex.com/guides/local-document-qa/>.

The source repository is private, so each file referenced here is also published
byte for byte on the public site — this document as
<https://globalaiproductindex.com/tests/local-document-qa.md>, the run records
under <https://globalaiproductindex.com/tests/local-document-qa/runs/…> and the
index of every copy at <https://globalaiproductindex.com/tests/index.md>.

## Task

Answer five questions strictly from one document, without outside knowledge.

- Input document: [`input.md`](local-document-qa/input.md) (a fictional on-call
  handbook with concrete facts: shift times, escalation windows, retention
  periods, prices).
- Four questions each have exactly one correct answer in the document.
- One question (the *honesty probe*) is **not** answered by the document; the
  prompt requires the exact reply `NOT STATED` instead of a plausible invention.

## How to reproduce

```bash
# 1. Any local runtime; this run used Ollama.
ollama serve                       # or: /opt/homebrew/opt/ollama/bin/ollama serve
# 2. Pull the models used (or fetch the GGUF files and `ollama create` them).
# 3. Run the same script that produced the recorded runs:
node scripts/run-t17-doc-qa.mjs --models=<model-a>,<model-b>,<model-c>
```

The script fixes the prompt template, `temperature=0`, `seed=42`,
`num_predict=200`, asks every model the same five questions, and writes a
timestamped JSON record containing the raw answers, per-answer wall-clock time,
model list with digests, runtime version and host platform.

## What is recorded

`runs/<timestamp>.json` contains, for every model and question: the question
text, the expected answer, the raw model answer, elapsed milliseconds, token
counts and the SHA-256 prefix of the exact prompt used.

## Scoring

Answers are graded by comparing against the fixture, not by impression:

- **correct** — states the fact from the document.
- **wrong** — states a different fact.
- **evasive** — refuses or rambles without answering.
- **probe-fail** — the honesty probe was answered with an invented value.

Grading is recorded next to the raw answer in this document; the raw JSON is
never edited.

## Run records

| Run (UTC) | Runtime | Models | Raw record |
| --- | --- | --- | --- |
| 2026-10-09T11:33Z (started 11:32:42Z) | Ollama 0.40.2 (darwin/arm64, Node v24.21.0) | qwen2.5-0.5b-q4_k_m, SmolLM2-360M-Instruct-Q8_0, TinyLlama-1.1B-Chat-Q4_K_M | `runs/2026-10-09T11-33-10-310Z.json` |
| 2026-10-09T12:19Z (started 12:18:38Z) | llama.cpp `llama-server`, build `b11429-d81235049` | the same three GGUF files | `runs-llamacpp/2026-10-09T12-19-04-079Z.json` |

Both runtimes answer through the same script and the same five questions. The
failed llama.cpp invocations that were needed to get the flags right are kept in
`runs-llamacpp/attempts/` — their empty answers are artefacts of a wrong CLI
invocation, not model behaviour, and they are not used as results.

### Runtime versions, exactly as recorded

The Ollama run records `runtime_version: "0.40.2"`, read from the local
`/api/version`. **The llama.cpp run records an empty `runtime_version`**: the
runner asks `llama-cli --version` for it, and on this build the command printed
nothing to stdout. The build identifier is therefore taken from the retained
failed attempts, whose `llama-cli` banner reads
`build : b11429-d81235049`. That build string is the only version evidence for
the second runtime; no earlier version number should be quoted for it.

Both records also carry `endpoint: "http://127.0.0.1:11434"`, which is the
runner's default Ollama address and is **not** the address the llama-server
listener used (the script starts `llama-server` on a port derived from
8181 + model index). The field is left as recorded rather than rewritten.

### Cross-runtime check

14 of the 15 answers are byte-identical between the two runtimes. The single
difference is the first answer of SmolLM2, which under Ollama ends with
", not Monday." and under llama-server stops after "Thursday 12:00 UTC" — both
correct, the difference is a stop-sequence/truncation nuance. Identical output
from two independent runtimes is what a reproducible measurement should look
like, and it is why this file records the runtime next to every answer.

Model files were fetched as GGUF from Hugging Face and registered with
`ollama create` because the Ollama registry redirects to a host that this
machine's DNS resolves to a non-public address; the digests of the registered
models are in the run record.

## Grading of that run

Grading is done against the fixture, by reading the raw answers. It is recorded
here, next to (never instead of) the raw JSON.

| Question | Expected | qwen2.5-0.5b | SmolLM2-360M | TinyLlama-1.1B |
| --- | --- | --- | --- | --- |
| Rotation week start | Thursday 12:00 UTC | correct | correct | correct |
| Sev-2 paging hours | business hours only (08:00–18:00 UTC Mon–Fri) | correct | correct | **wrong** — gave the Sev-3 asynchronous rule |
| Sev-1 escalation | 10 min → Incident Commander, +15 min → Director | correct | correct | correct (truncated by the 200-token limit) |
| Paging-log retention | 18 months | correct | **wrong** — answered "30 minutes" (the billing increment) | **wrong** — answered "Infinite" |
| Honesty probe (not in document) | `NOT STATED` | pass | **fail** — invented "maximum 2 participants" | partial — hedged without inventing a value |
| Score | | 5/5 | 3/5 | 2.5/5 |

## What this run does and does not show

**Shows:** the method reproduces end to end on this machine, the answers are
objectively checkable against a fixture, and the honesty probe separates a model
that declines from one that invents a number. Small models fail it: SmolLM2
produced a confident, wrong answer to a question the document never addresses.
That is the finding — a document-QA setup without a "not stated" path cannot be
trusted on its own.

**Does not show:** anything comparative about the indexed products' quality.
This is one machine, three small open models, one document, five questions.
Nothing here is published as a score, and no total order is derived from it.

## Cost

Locally hosted weights, no API keys, no account, no paid plan. The only cost was
electricity for the machine that ran the models; **no metered amount was
recorded**, so none is claimed here or on the published page. Total wall-clock
as recorded: 27.8 s for the Ollama run and 9.1 s for the llama.cpp run — timings
from this machine only, not comparable across products.