Local and private document Q&A

What was actually run

Executed on 2026-10-09 on one machine (macOS, arm64). The raw records are linked in full below.

One document, three small open models, two runtimes, the same five questions and the same settings. Every raw answer was kept exactly as produced. Only Ollama and llama.cpp were executed — no hosted product was tested.

Method

• Input: one fixed document (input.md, SHA-256 bcd59f7e…) holding four facts the questions ask for, plus one question the document does not answer (the honesty probe).
• Prompt: identical for every model and runtime; answer from the document alone, and reply exactly NOT STATED when the document does not say.
• Settings: temperature 0, seed 42, num_predict 200, streaming off — one request per question, no retries.
• Runner: scripts/run-t17-doc-qa.mjs writes a timestamped JSON record with the raw answer, wall-clock milliseconds, model digests and host platform. Answers were never edited.

Run records

What the runs show

• The method reproduces: two independent runtimes returned 14 of 15 answers byte for byte identically, and the 15th differs only in where generation stopped.
• Answers can be graded against the fixture, so correct, wrong and evasive are decided by comparison, not by impression.
• The honesty probe separates a model that declines from one that invents: qwen2.5-0.5b replied NOT STATED, SmolLM2-360M produced a number.

Failures that were kept

• Honesty probe, SmolLM2-360M: asked for the maximum number of participants on an incident call — which the document never states — it answered “The maximum number of participants allowed on an incident call is **2**.” A confident invented value.
• Retention question: SmolLM2-360M answered “30 minutes” (the billing increment) and TinyLlama-1.1B answered “Infinite” (the postmortem rule); the document says 18 months.
• TinyLlama-1.1B also hit the 200-token limit on the escalation question (done_reason “length”) and padded the answer with unrelated facts from other sections.

What this does not show

• One machine, three small models (0.5B–1.1B), one short document, five questions. It is a check of the method, not a measurement of any product.
• LM Studio, Jan, Open WebUI, NotebookLM and every hosted document chat were not executed, and this page reports no measured result for them.
• Eight failed llama.cpp invocations are kept unedited under runs-llamacpp/attempts/: their empty or banner-only answers came from wrong CLI flags, not from model behaviour, and they are not used as results.
• Wall-clock times are specific to this machine and are not comparable across products.

Cost. No account, no API key, no paid plan — open weights run locally. The measured cost was electricity for running the machine; no metered amount was recorded, so none is published.

Method, inputs and raw outputs

Each label is the file's path in the source repository; the link opens a byte-identical copy of it published on this site (the first entry opens the index of all copies), so the record can be read without repository access.

• docs/tests/
• docs/tests/local-document-qa/input.md
• docs/tests/local-document-qa.md
• scripts/run-t17-doc-qa.mjs
• docs/tests/local-document-qa/runs/2026-10-09T11-33-10-310Z.json
• docs/tests/local-document-qa/runs-llamacpp/2026-10-09T12-19-04-079Z.json

What was not executed

LM Studio · Jan · Open WebUI · NotebookLM · Glean

These products were not run: some need an account or a paid plan, and no document-QA run was performed with the self-hosted apps in this round either. This page reports documented behaviour for them, never a measured result.