# Reproducible run records (published copies) These are the raw records behind the two executed launch themes on this site: - [Local and private document Q&A](https://globalaiproductindex.com/guides/local-document-qa/) - [Multi-file edits in an existing repository](https://globalaiproductindex.com/guides/existing-repo-multi-file-edits/) The research-with-citations theme has **no records**: no answer engine was executed, and publishes method and limits only. ## What these files are Every file below is a **byte-identical copy** of a file in this project's source repository under the path shown in its heading (the runners live in `scripts/`, everything else in `docs/tests/`). The copies are published here so the records can be read and cited without repository access. Nothing was edited, shortened, reordered or prettified for publication, and the SHA-256 of each published copy is listed so you can check it. Host paths inside the records (the scratch directories and the local GGUF file paths) are exactly as the runners recorded them; keeping the copy byte-for-byte is what makes the digest check meaningful, so no path was tidied. Two kinds of shell are deliberately kept apart: - **Results** — runs whose answers were used in the published findings. - **Attempts / calibration** — runs kept because deleting them would hide how the working invocation was found. They are labelled as not-results both here and on the guide pages, and no finding is drawn from them. ## The records ### `docs/tests/existing-repo-coding/fixture.md` The three-file scratch repository, the task given to every model, why the fixture is a fair test, and the condition each indexed product needs (none was executed). - Published copy: - Size: 5219 bytes - SHA-256: `2291c6ee27bf5fb100e0cbf5d0944c5f53e73b06c15226a081de17ab0cc568d5` ### `docs/tests/existing-repo-coding/runs/2026-10-09T12-12-26-217Z.json` Calibration attempt, NOT a result: the harness baseline command was wrong (`Cannot find module …/test`). - Published copy: - Size: 7527 bytes - SHA-256: `504e0210dda0a34225c1f6460bf64088787b855dbdd6c29c7497a8cead570180` ### `docs/tests/existing-repo-coding/runs/2026-10-09T12-12-52-783Z.json` Calibration attempt, NOT a result: same baseline defect; the fixture counts read 1 fail / 0 pass. - Published copy: - Size: 7655 bytes - SHA-256: `32f76b68a143208c853b2161456ec5d68709c60ff7744737f48dd2ade1059bcc` ### `docs/tests/existing-repo-coding/runs/2026-10-09T12-13-58-519Z.json` The canonical multi-file run. Baseline 2 pass / 3 fail; all three models failed the task. Raw output, diff stat and test tail per model. - Published copy: - Size: 6902 bytes - SHA-256: `552100a90d721450d38e82463b1a3a9d2eb2ed10f96a4dd1ce0d6db93f234a2e` ### `docs/tests/local-document-qa.md` Case document for the document-Q&A runs: task, reproduction steps, the grading table and what the run does and does not show. - Published copy: - Size: 6810 bytes - SHA-256: `f8f12cb3b8b7bc65229d530bef7a07040228ea5c22a6bd6bd39b9806514cdc84` ### `docs/tests/local-document-qa/input.md` The single input document every document-Q&A model received (a fictional on-call handbook). Its SHA-256 is recorded in the run JSON. - Published copy: - Size: 2090 bytes - SHA-256: `bcd59f7e7294d67e336b2b1a7b5f4878fab453eea34101396bb1199523435daf` ### `docs/tests/local-document-qa/runs-llamacpp/2026-10-09T12-19-04-079Z.json` The same three models through the independent llama.cpp `llama-server` runtime. 14 of the 15 answers are byte-identical to the Ollama run. - Published copy: - Size: 8889 bytes - SHA-256: `329284a0770024c7c531cd812b2c350f363d6e55a97d11605bfa049df3646390` ### `docs/tests/local-document-qa/runs-llamacpp/attempts/2026-10-09T12-14-20-549Z.json` Failed llama.cpp invocation kept unedited. Its answer is empty, or the interactive banner echoed back, or the prompt itself — artefacts of a wrong CLI invocation, not model behaviour. NOT a result. - Published copy: - Size: 2877 bytes - SHA-256: `cf46d5b6e10f7d561021bd743e4f96614f921f9368317e1c7bdd13af1fb162d2` ### `docs/tests/local-document-qa/runs-llamacpp/attempts/2026-10-09T12-14-46-114Z.json` Failed llama.cpp invocation kept unedited. Its answer is empty, or the interactive banner echoed back, or the prompt itself — artefacts of a wrong CLI invocation, not model behaviour. NOT a result. - Published copy: - Size: 10710 bytes - SHA-256: `68391081d80bc5b6da01a0107a8a61596b5617075ef6b2ea721822966e59a902` ### `docs/tests/local-document-qa/runs-llamacpp/attempts/2026-10-09T12-15-20-292Z.json` Failed llama.cpp invocation kept unedited. Its answer is empty, or the interactive banner echoed back, or the prompt itself — artefacts of a wrong CLI invocation, not model behaviour. NOT a result. - Published copy: - Size: 35961 bytes - SHA-256: `e83db3206c9bf21be11b1dc8b545826e9ea2c6f008162dc5f0283841530ef9c1` ### `docs/tests/local-document-qa/runs-llamacpp/attempts/2026-10-09T12-16-14-564Z.json` Failed llama.cpp invocation kept unedited. Its answer is empty, or the interactive banner echoed back, or the prompt itself — artefacts of a wrong CLI invocation, not model behaviour. NOT a result. - Published copy: - Size: 35494 bytes - SHA-256: `6e0122ad9df75715c9c94ff451532037c6ddb03e0afa7310ed1be2fcec107736` ### `docs/tests/local-document-qa/runs-llamacpp/attempts/2026-10-09T12-17-16-210Z.json` Failed llama.cpp invocation kept unedited. Its answer is empty, or the interactive banner echoed back, or the prompt itself — artefacts of a wrong CLI invocation, not model behaviour. NOT a result. - Published copy: - Size: 2779 bytes - SHA-256: `45aae5aa4bb7a8157539d33744427eae7da13e7bb7b8525a2008f6fa076d373a` ### `docs/tests/local-document-qa/runs-llamacpp/attempts/2026-10-09T12-17-20-860Z.json` Failed llama.cpp invocation kept unedited. Its answer is empty, or the interactive banner echoed back, or the prompt itself — artefacts of a wrong CLI invocation, not model behaviour. NOT a result. - Published copy: - Size: 1447 bytes - SHA-256: `9f7b85c4db9bf1664d43b6bd2c9e2ae5523b2088d4cb87e46702078470df0d9e` ### `docs/tests/local-document-qa/runs-llamacpp/attempts/2026-10-09T12-17-46-837Z.json` Failed llama.cpp invocation kept unedited. Its answer is empty, or the interactive banner echoed back, or the prompt itself — artefacts of a wrong CLI invocation, not model behaviour. NOT a result. - Published copy: - Size: 5982 bytes - SHA-256: `a4363b3ee69a69f169d32523f57200f9f0fadca8a93eece59cab219f750266d0` ### `docs/tests/local-document-qa/runs-llamacpp/attempts/2026-10-09T12-18-21-427Z.json` Failed llama.cpp invocation kept unedited. Its answer is empty, or the interactive banner echoed back, or the prompt itself — artefacts of a wrong CLI invocation, not model behaviour. NOT a result. - Published copy: - Size: 8888 bytes - SHA-256: `e4b9fce6c1f565cd7d4abce79d8e839e3b4ae6373254f00fa0d1807857cc23c4` ### `docs/tests/local-document-qa/runs/2026-10-09T11-33-10-310Z.json` The canonical document-Q&A run (Ollama 0.40.2, three models, five questions): raw answers, per-answer milliseconds, model digests and settings. - Published copy: - Size: 9481 bytes - SHA-256: `b318715a5fbbf92995c1365ce878382f31280a855404adde5ab9b351901c7bdc` ### `docs/tests/README.md` Method rules for the whole `docs/tests/` tree: what counts as a real run, what is a fixture, and the inventory of every record. - Published copy: - Size: 7392 bytes - SHA-256: `016691802a4cfb2e0b864883b35bfa04aaa51edaf0504a9a946eca14063cce3e` ### `scripts/run-t17-coding.mjs` The runner that produced the multi-file records: the exact prompt template, the src/-only write guard and the test command. - Published copy: - Size: 6771 bytes - SHA-256: `f8cd550b55a1b50d20b56e1ff8e56ad6412186cc6fe56a1982077707234102fc` ### `scripts/run-t17-doc-qa.mjs` The runner that produced the document-Q&A records: the exact prompt template, questions, settings and record format. - Published copy: - Size: 9485 bytes - SHA-256: `0fce321feb0d8ccd1da8d55ccb54acbe8bd8da5618a269788bdf8e831239d358` ### `docs/tests/source-backed-research-qa.md` The research-citations case. Status: NOT executed. It is the plan, the condition each engine needs (verified against the product records), and states that no run exists. - Published copy: - Size: 4523 bytes - SHA-256: `e93bbd744005565a5bd75b87a9c94e440996d248f1f5807d745bae8c3538b735` ## How to reproduce ```bash # Document Q&A (any local runtime; the recorded runs used Ollama, then llama-server) node scripts/run-t17-doc-qa.mjs --models=,, node scripts/run-t17-doc-qa.mjs --runtime=llamacpp --models=,, # Multi-file edits (Ollama; creates a scratch git repository under /tmp) node scripts/run-t17-coding.mjs --models=,, ``` Grading is not a script: answers are compared against the fixture document and the recorded expectation, and the result is written next to the raw answer in [local-document-qa.md](https://globalaiproductindex.com/tests/local-document-qa.md) and the [method rules](https://globalaiproductindex.com/tests/method.md). ## Limits These records describe one machine, a handful of small open models and one fixture each. They show that the *method* reproduces and that the answers are checkable; they are not a benchmark of any product, and no score or ranking is derived from them. The hosted products indexed on this site were not executed — see each guide's "What was not executed" section.