What 1,300+ test runs taught us about cheap AI models

We test every model change against real coding tasks before we believe it works. Not "it looks right in one chat" - a batch of runs, saved, scored, published. Here's what actually moved the needle, and what didn't.

A second, cheaper model reading the diff before commit was the single biggest win. One bug-fixing benchmark went from 67% actually-correct to 100% actually-correct just from adding that habit. It costs about a penny a check.

Multi-step to-do lists were our hardest case, and our biggest improvement. Give a weak model a to-do list with several steps and it's easy for it to lose the thread, wander off, or quietly stop before finishing. Our first measurement here was 54% success. Better prompting, automatic continuation, and a mid-task check that steps in when a model gets stuck took that to 96%.

Planning-first prompting doesn't automatically help - and can actively hurt. We assumed it would, based on an earlier survey of prompting advice. Then we actually tested it: across 6 different task types, a "break this into a to-do list first" instruction scored worse than a plain one-line instruction in 5 of 6 matched pairs. One model in particular started writing detailed, well-formed plans and then never calling the edit tool at all - a failure mode specific enough that we found and fixed a real gap in our own retry logic because of it.

Detailed, task-specific prompting is a different thing from generic "plan first," and it does help. When we gave a model a genuinely specific prompt - name the actual sections to build, ask a concrete planning question about this exact task - it finished up to 2x faster and used roughly half the completion tokens, compared to a plain instruction. That's not "make it plan," it's "make the request less ambiguous." We built that distinction into a real feature (Prompt Guidance, on by default) rather than leaving it as a one-off finding.

Free models can work, they just fail differently. In one overnight batch of 13 creative single-file tasks across 4 free models, only 2% landed on the first try - but a fallback chain of other models rescued 51 of the other 52 runs, taking the effective success rate to 100%. The failures weren't random: they were specific, repeatable patterns (a stalled plan, a claimed-but-never-made edit, a model that says it'll build something and just doesn't) that we could actually detect and catch.

None of this is secret. Every batch we've ever run - the wins and the failures - is public at /stats, with full case studies at /case-study.