Pilot results
This is a pilot, not a leaderboard. One agent harness (opencode 1.19.3), two free models, three runs per task. It does not rank any vendor or paid agent, and no Claude Code, Codex, Cursor or Devin results are included.
What we ran
Three harder coding tasks, 18 runs in total (3 tasks × 2 models × 3 runs), all one-shot with no human steering. Date: 2026-10-16. No money was spent.
| Task | What it tests |
|---|---|
| A | Multi-file bug with a misleading symptom: the failing test shows a cart subtotal off by one cent, but the root cause is floor division in a shared money helper. |
| B | Behavior-preserving refactor, checked against the frozen original with fixed edge cases and 300 random inputs. |
| C | Ambiguous spec (e.g. "1h30", garbage input) in non-interactive mode: choose, then document assumptions. |
Results (n=3 per row)
| Task | Description | Model | Runs passing all checks | Mean score /100 | Min–max | Std dev | Median wall (s) |
|---|---|---|---|---|---|---|---|
| A | Multi-file bug fix | big-pickle | 1/3 | 72.1 | 61.8–92.6 | 14.5 | 27 |
| A | Multi-file bug fix | deepseek-v4.1-flash-free | 3/3 | 96.2 | 94.9–97.0 | 0.9 | 16 |
| B | Behavior-preserving refactor | big-pickle | 2/3 | 77.9 | 68.3–93.5 | 11.1 | 161 |
| B | Behavior-preserving refactor | deepseek-v4.1-flash-free | 3/3 | 94.8 | 92.6–97.0 | 1.8 | 86 |
| C | Ambiguous spec | big-pickle | 3/3 | 88.9 | 88.3–90.2 | 0.9 | 788 |
| C | Ambiguous spec | deepseek-v4.1-flash-free | 2/3 | 92.4 | 92.0–92.6 | 0.3 | 59 |
Scores use the methodology rubric, with partial automation described below.
What we saw
- Reliability, not reasoning, separated the models on task A. deepseek-v4.1-flash-free made the same one-line root-cause fix in 3 of 3 runs. big-pickle fixed it once; in the other two runs it made no edit, after mistyping a folder path and being blocked by a permission rule.
- Task B: every run that edited code preserved behavior, but one big-pickle run stopped after reading files (wrong path again), and another hit the 600 s timeout after producing a correct refactor.
- Task C: no run asked a question and all wrote an assumptions file. One deepseek run accepted some invalid input and failed the garbage-rejection check.
Caveats
- Free models only. Results say nothing about paid agents and are not a vendor comparison.
- Tiny sample: 3 runs per cell. Standard deviations are indicative at best.
- Wall time mostly reflects free-tier server queueing (8 s to 811 s for similar work), so speed points are noisy.
- Quality (/25) used automated proxies only. No blind human review has been done yet.
- Cost (/10) is a flat 10 because every run was free, so it carries no information here.
- Autonomy: runs were one-shot, so no interventions were possible. One run scored 0 for a timeout, which is arguably harsh.
- Task A has a one-line fix, so it tests reliability more than skill. Task B's differential tests cannot catch ugly-but-correct code. Task C's "1h30 = 90 minutes" check follows the ticket literally, and a reasonable alternative reading could be marked down.
- Agents ran on the same filesystem as the hidden tests. We checked the logs and no run read them.
What's next
Paid agents will be added when they can be run properly, with the full five-task methodology. Until then, treat these numbers as a pilot only.