Multi-file edits in an existing repository

What was actually run

Executed on 2026-10-09 on one machine (macOS, arm64). The run record is linked in full below.

The runner created the scratch repository described in the fixture, asked three small local models through Ollama to fix it, applied whatever they returned and re-ran the tests. All three failed. The hosted coding agents on this page were not executed.

Method

• Fixture: a real git repository with src/calc.js (wrong divisor), src/format.js (missing empty-list branch) and a fixed test file. node --test reports 2 pass / 3 fail before any model runs, and the tests may not be changed.
• Prompt: one request per model — temperature 0, seed 42, num_predict 1200. It asks for the failing repository to be fixed and for each changed file to be returned in full in a fixed FILE marker format. The exact template is in scripts/run-t17-coding.mjs.
• The runner writes only files under src/, then runs node --test and records the raw output, the git diff --stat and whether the suite passes. Nothing is retried or edited.
• Every model received the identical fixture and prompt; only the model name differed between the three attempts.

Run records

What the run shows

• The fixture is a real, checkable task with a fixed baseline and is not passable by accident: the fix needs an edit in calc.js and a branch in format.js.
• The runner can only write files under src/, so a model cannot pass by rewriting the tests — the guard was not needed in this run, but it is part of the method.
• At this parameter count the three local models are not a substitute for the hosted coding agents indexed on this page. That is a finding about these three models, not a quality judgement on any product.

Failures that were kept

• qwen2.5-0.5b followed the output format but emitted the prompt's own placeholder text as file content: the applied src/calc.js begins with the literal text of the placeholder token, so node --test died with SyntaxError: Unexpected token '<'.
• SmolLM2-360M answered with a numbered “open your text editor” walkthrough, so the parser found no FILE markers and wrote no file; the suite stayed at its 2 pass / 3 fail baseline.
• TinyLlama-1.1B returned a shorter prose walkthrough with the same outcome: nothing applied, 2 pass / 3 fail. None of the three fixed the bug.

What this does not show

• Three small local models, one three-file repository, one attempt each, one machine. Cursor, Claude Code, GitHub Copilot, Windsurf, Replit Agent, Aider, Cline and Tabnine were not executed.
• Two earlier records from the same day (runs/2026-10-09T12-12-26-217Z.json and runs/2026-10-09T12-12-52-783Z.json) are kept unedited: in them the harness's baseline command was wrong (Cannot find module …/test), so the fixture counts read “?” and 1/0. They are calibration attempts, not results.
• The record stores the raw output, the diff stat and the first lines of the test output, not a token-level log; wall-clock is specific to this machine.
• No score and no ranking is derived from one repository and one attempt per model.

Cost. No account, no API key, no paid plan — the models ran locally through Ollama. The measured cost was electricity for running the machine; no metered amount was recorded, so none is published.

Method, fixture and raw outputs

Each label is the file's path in the source repository; the link opens a byte-identical copy of it published on this site (the first entry opens the index of all copies), so the record can be read without repository access.

• docs/tests/
• docs/tests/existing-repo-coding/fixture.md
• scripts/run-t17-coding.mjs
• docs/tests/existing-repo-coding/runs/2026-10-09T12-13-58-519Z.json
• docs/tests/existing-repo-coding/runs/2026-10-09T12-12-26-217Z.json
• docs/tests/existing-repo-coding/runs/2026-10-09T12-12-52-783Z.json

What was not executed

Cursor · Claude Code · GitHub Copilot · Windsurf · Replit · Aider · Cline · Tabnine

This round did not execute these applications, so no measured result is reported for any of them. Some of them need an account or a particular plan before they can be run: Claude Code needs an available access authorisation, and the documented paths are a Claude subscription (Pro, Max, Team or Enterprise), a Claude Console account, or access through a supported cloud provider — so a paid Claude subscription is not the only route — while the hosted agents need a signed-in account. That is not true of every product listed — Aider and Cline are open source and can connect to a local model, so neither needs a paid plan. No further reason was recorded for this round, so where a condition was not confirmed it is stated as unconfirmed rather than assumed. Their pages and comparisons report documented behaviour with a source and a check date.