# Fixture: a three-file bug fix in an existing repository

The repository below is created from scratch by the runner, so it exists exactly
the same way for every tool.

```
repo/
├── src/calc.js      # add() and average() — average() is wrong: it divides by length + 1
├── src/format.js    # formatSummary() — prints "avg: NaN" when the list is empty
└── test/calc.test.mjs  # five assertions, three of which currently fail
```

Task given to every tool, verbatim:

(the repository's `npm test` is `node --test`)

> In this repository, `npm test` fails. Find the cause and fix it. You may change
> any file under `src/`. Do not change the tests. Output each changed file in
> full, using this exact format:
>
> ```
> === FILE: src/calc.js ===
> <full new contents>
> === FILE: src/format.js ===
> <full new contents>
> ```

The runner applies whatever the tool returns, then runs `npm test` in the
repository and records: files actually changed (diff), the test output, and
whether the suite passes. No answer is edited or retried after being recorded.

## Why this is a fair test of "multi-file edits in an existing repository"

- The failing behaviour must be traced across two files (a wrong divisor in
  `calc.js` and an empty-list guard in `format.js`), so a single-file guess does
  not pass.
- The tests are fixed input; changing them cannot help.
- The repository is a real git repository with a commit, so the runner records
  the diff of what actually changed.

## Run records

| Run (UTC) | Runtime | Models | Baseline | Result | Raw record |
| --- | --- | --- | --- | --- | --- |
| 2026-10-09T12:13Z | Ollama 0.40.2 (darwin/arm64) | qwen2.5-0.5b, SmolLM2-360M, TinyLlama-1.1B | 2 pass / 3 fail | **all three failed the task** | `runs/2026-10-09T12-13-58-519Z.json` |

### What each tool did

| Model | Files returned | Files written (inside `src/` only) | Diff | Final `node --test` |
| --- | --- | --- | --- | --- |
| qwen2.5-0.5b | 2 | `src/calc.js`, `src/format.js` | −16/+2 lines — both files were truncated, so the exports were lost | fail |
| SmolLM2-360M | 0 | none | none | fail (unchanged) |
| TinyLlama-1.1B | 0 | none | none | fail (unchanged) |

SmolLM2 and TinyLlama answered with prose instructions ("open your text editor…")
instead of the requested file format, so nothing could be applied. qwen2.5
followed the format but emitted incomplete files. The raw outputs are stored
unedited in the run JSON.

## What this run shows

- The fixture is a real, checkable task with a fixed baseline (2/5 passing), and
  it is **not** passable by accident: the fix requires changing the divisor in
  `calc.js` *and* adding an empty-list branch in `format.js`.
- Small local models are not a substitute for the hosted coding agents indexed
  on the site, at least not at this parameter count. That is a finding about
  these three models, not about a product's quality.
- The runner only writes files matching `src/*`, so a model cannot reward itself
  by editing the tests. In this run the guard was not needed, but it is part of
  the method.

## What was not run, and why

The indexed products in this theme — Cursor, Claude Code, GitHub Copilot,
Windsurf, Replit Agent, Aider, Cline, Tabnine — were **not executed in this
round**, so this document reports no measured performance for any of them. The
table states what each one actually needs, checked against this site's product
records, rather than assuming every one of them requires a paid plan.

| Product | Executed this round | Condition recorded |
| --- | --- | --- |
| Cursor | No | An account; a free Hobby tier exists and paid plans add higher limits. |
| Claude Code | No | An access authorisation; a Claude subscription (Pro, Max, Team or Enterprise), a Claude Console account or a supported cloud provider all count, so a paid subscription is not the only route. |
| GitHub Copilot | No | An account; a free tier exists and Pro and higher plans are paid. |
| Windsurf | No | An account; a free tier exists and paid plans add limits. |
| Replit Agent | No | An account; a free Starter plan exists and agent usage is metered or paid. |
| Aider | No | Open source and free; it can connect to a local model, so no paid plan is required. |
| Cline | No | Open source and free for individuals; it can use your own API keys or a local model, so no paid plan is required. |
| Tabnine | No | An account; a free tier exists and paid Dev and Enterprise plans add features. |

None of these applications was executed this round, so no measured performance
is reported. Some need an account or a particular plan (Claude Code requires an
access authorisation: a Claude subscription (Pro, Max, Team or Enterprise), a
Claude Console account or a supported cloud provider all count, so a paid
subscription is not the only route; the hosted agents need a signed-in account),
but that is not true of every product listed: Aider and Cline are open source
and can connect to a local model. No further reason was recorded for this round,
so where a condition was not confirmed it is stated as unconfirmed rather than
assumed.
Their comparison pages report documented behaviour, each cell carrying its
source and check date.