How well does codelazr actually work?
We test every model against real coding tasks - bug fixes, tests, refactors, real open-source repos - and check the actual result, not just whether something happened. Here's the real data.
Case study: same prompt, real sites
See all case studies →We sent the exact same "build a SaaS landing site" prompt to 15 models. Real repos, real deployments - a few of the results:
Getting better over time
Our hardest benchmark - multi-step to-do lists - tested repeatedly as we shipped fixes.
Success rate across every batch we've run
Every test batch we've ever run, oldest to newest (81 batches, 1,431 runs total) - not just the highlights. Hover a point for details.
Where codelazr is consistently strong
Rewrite text to sound human
36/36 runs · 7 models
Build a marketing page (plain prompt)
28/30 runs · 10 models
Write a validation function
24/25 runs · 6 models
Write tests for existing code
24/24 runs · 8 models
Refactor a function
24/24 runs · 8 models
Add input validation
23/24 runs · 8 models
Add a small feature
24/24 runs · 8 models
Write docs for existing code
24/24 runs · 8 models
makealist 10tasks
14/14 runs · 7 models
task 05 refactor
14/14 runs · 6 models
task 10 simplify
14/14 runs · 6 models
task 07 debug error
12/13 runs · 6 models
plain do stuff
9/10 runs · 5 models
plain blog dogs
10/10 runs · 5 models
Success rate by task type
The 10 kinds of requests we test most, ranked by how often we've tested them. Numbers are from our most recent measurement of each - real strengths and real weak spots, not a highlight reel.
Cost per task type
Average full cost to complete each task - including any automatic continues or fix-it-and-recheck rounds it took to land. Cheapest first.
Time to completion per task type
Average wall-clock time from request to finished result, same full-task basis as cost above. Fastest first.
Cost vs. success rate, by model
Each dot is one of our 7 default models - further right costs more per task, higher up succeeds more often. The sweet spot is top-left.
Avg task completion time by model, ranked
Our 7 default models, sorted fastest first - avg full-task time including any continues it took to land.
Model leaderboard
Which model is best at which task
Our 7 default models against the 10 most-tested task types. Darker green = higher success rate - scan a column to see which model wins for that kind of work.
| Model | Fix a pricing bug | Multi-step to-do list (4 topics) | Multi-step to-do list + docs | Debug from an error message | Scaffold a Next.js app | Write a short blog post | validate email B plain | validate email A guided | Plan, then build a marketing page | Write a blog post (guided) |
|---|---|---|---|---|---|---|---|---|---|---|
| z-ai/glm-5.2 | 100% | 100% | 100% | 100% | - | - | - | - | 80% | - |
| google/gemini-2.5-flash | 100% | 100% | 100% | 100% | - | 100% | - | - | 75% | - |
| minimax/minimax-m3 | 90% | 100% | 100% | 100% | - | 86% | - | - | 80% | - |
| deepseek/deepseek-v4-pro | 100% | 100% | 100% | 100% | - | 100% | - | - | 75% | - |
| stepfun/step-3.7-flash | 100% | 100% | 100% | 100% | - | 86% | - | - | 100% | - |
| google/gemini-3-flash-preview | 86% | 100% | 100% | 0% | - | 100% | - | - | 75% | - |
| xiaomi/mimo-v2.5 | 90% | 89% | 89% | 78% | 100% | 57% | - | - | 86% | 100% |
What we've learned
Simple, single-focused tasks are a solved problem
Write this function, fix this bug, add this one feature - tasks with a single clear goal succeed 90-100% of the time, across nearly every model we test. This is the bread-and-butter of day-to-day coding, and it's where codelazr is most dependable.
Real open-source code holds up, not just toy examples
We forked a real, popular npm package (dotenv) and gave models a real missing feature to build, with real existing conventions to follow. 12 of 14 runs got it right - 86%, across all 7 models we tested. It's one thing to do well on examples built to be easy; it's another to do well on code you didn't write yourself.
Multi-step task lists were our hardest case, and our biggest improvement
Give a model a to-do list with several steps and it's easy for it to lose the thread, wander off, or quietly stop before finishing. Our first measurement here was 54% success. Targeted fixes - better prompting, automatic continuation, and a mid-task check that steps in when a model gets stuck - took that to 96%.
A cheap second opinion catches what the first model misses
Before any change is proposed as a commit, a second, cheaper model reads the diff and the original task, and flags anything that looks wrong or incomplete. It costs about a penny per check. One bug-fixing benchmark went from 67% actually-correct to 100% actually-correct from adding that single habit.
What codelazr actually is
Codelazr is a web harness with tweaks and stuff to turn weaker OpenRouter models into a full, high-tier coding platform. When you send an instruction, the response you get isn't just the raw model output - it's processed by our system, and optionally checked by a second model before it's proposed as a commit. LLMs checking each other's work, basically. On top of that, a handful of small mechanisms catch the specific ways weaker models tend to fail - a stalled plan, a claim that something was done when it wasn't, a task that just hangs - and handle those gracefully so the task still finishes for you.
Goals for this project
- Use OpenRouter models - no Anthropic models, out of principle, for science, and for cost
- Match top-model results on 80% of tasks, for less than half the cost
- Give people SOME free usage, so they can try codelazr Pro without paying first
- Let people BYOK - bring your own OpenRouter key and use codelazr completely free, unlimited, forever
- Offer several top coding models - consistently Top 10 on OpenRouter for programming
- Integrate seamlessly with GitHub for quick, proper commits
- Work from any device with a browser - phone included
Every test batch we've run
The full public log - 81 batches, newest first. Nothing held back.
| Date | Time | Tasks tested | Models | Runs | Success |
|---|---|---|---|---|---|
| 2026-07-18 | 01:18 | future A plain, future B makealist | 6 | 14 | 86% |
| 2026-07-18 | 01:08 | future A plain, future B makealist | 6 | 12 | 67% |
| 2026-07-13 | 08:03 | task 01 fix bug, task 06 input validation | 4 | 9 | 89% |
| 2026-07-13 | 07:50 | task 01 fix bug, task 02 small feature, task 03 write function, task 04 write tests, task 05 refactor, task 06 input validation, task 07 debug error, task 08 api integration, task 09 write docs, task 10 simplify | 6 | 68 | 81% |
| 2026-07-13 | 06:52 | task 01 fix bug, task 02 small feature, task 03 write function, task 04 write tests, task 05 refactor, task 06 input validation, task 07 debug error, task 08 api integration, task 09 write docs, task 10 simplify | 6 | 65 | 92% |
| 2026-07-12 | 06:33 | whack a mole makealist | 4 | 4 | 50% |
| 2026-07-12 | 05:58 | Plan, then build a marketing page | 1 | 1 | 100% |
| 2026-07-12 | 05:53 | Plan, then build a marketing page | 2 | 2 | 100% |
| 2026-07-12 | 05:45 | Plan, then build a marketing page | 8 | 8 | 13% |
| 2026-07-12 | 05:32 | whack a mole makealist | 4 | 4 | 50% |
| 2026-07-11 | 19:50 | audio visualizer | 1 | 1 | 0% |
| 2026-07-11 | 19:49 | audio visualizer | 1 | 1 | 0% |
| 2026-07-11 | 19:37 | audio visualizer, bar race, solar system | 4 | 12 | 25% |
| 2026-07-11 | 19:31 | blob morph | 1 | 1 | 0% |
| 2026-07-11 | 19:19 | blob morph, fireworks, kaleidoscope | 4 | 12 | 17% |
| 2026-07-11 | 19:12 | interactive dashboard | 1 | 1 | 0% |
| 2026-07-11 | 19:04 | interactive dashboard | 4 | 4 | 0% |
| 2026-07-11 | 19:02 | whack a mole | 1 | 1 | 100% |
| 2026-07-11 | 18:58 | dodge game, memory game, synthwave player, tictactoe ai, whack a mole | 1 | 5 | 100% |
| 2026-07-11 | 18:50 | dodge game, tictactoe ai, whack a mole | 4 | 12 | 25% |
| 2026-07-11 | 18:37 | memory game, space parallax, synthwave player | 4 | 12 | 17% |
| 2026-07-11 | 18:05 | dog landing | 1 | 1 | 0% |
| 2026-07-11 | 17:38 | dog landing | 4 | 4 | 50% |
| 2026-07-11 | 17:35 | dog landing | 4 | 4 | 25% |
| 2026-07-11 | 16:01 | kanban A guided, kanban B plain | 2 | 5 | 20% |
| 2026-07-11 | 15:44 | kanban A guided, kanban B plain | 3 | 21 | 52% |
| 2026-07-11 | 15:11 | apple clone A guided, apple clone B plain | 6 | 12 | 50% |
| 2026-07-11 | 14:19 | validate email A guided, validate email B plain | 4 | 41 | 12% |
| 2026-07-11 | 13:34 | validate email A guided, validate email B plain | 5 | 50 | 46% |
| 2026-07-11 | 13:00 | kanban board v2 | 8 | 15 | 80% |
| 2026-07-11 | 12:37 | kanban board | 5 | 5 | 80% |
| 2026-07-11 | 08:57 | case study nextjs saas | 10 | 11 | 27% |
| 2026-07-11 | 07:57 | Write a short blog post, Fix a pricing bug | 10 | 58 | 69% |
| 2026-07-09 | 14:08 | Write a short blog post, Fix a pricing bug, Write a validation function | 6 | 37 | 95% |
| 2026-07-09 | 14:06 | Write a short blog post, Fix a pricing bug, Write a validation function | 6 | 36 | 100% |
| 2026-07-09 | 13:17 | Real OSS repo: multi-step feature list (dotenv) | 7 | 14 | 64% |
| 2026-07-09 | 11:08 | Real OSS repo: multi-step feature list (dotenv) | 7 | 18 | 56% |
| 2026-07-09 | 10:53 | Real OSS repo: add a feature (dotenv) | 7 | 14 | 86% |
| 2026-07-09 | 10:33 | Multi-step to-do list (4 topics), Multi-step to-do list + docs | 7 | 28 | 100% |
| 2026-07-09 | 09:59 | makealist 10tasks | 7 | 14 | 100% |
| 2026-07-09 | 09:39 | Debug from an error message | 7 | 21 | 86% |
| 2026-07-09 | 09:09 | Plan, then build a marketing page, Build a marketing page (plain prompt) | 8 | 48 | 94% |
| 2026-07-09 | 09:02 | Debug from an error message | 8 | 24 | 83% |
| 2026-07-09 | 08:57 | Debug from an error message | 8 | 24 | 83% |
| 2026-07-09 | 08:52 | Write docs for existing code | 8 | 24 | 100% |
| 2026-07-09 | 08:48 | Add a small feature | 8 | 24 | 100% |
| 2026-07-09 | 08:43 | Add input validation | 8 | 24 | 96% |
| 2026-07-09 | 08:38 | Refactor a function | 8 | 24 | 100% |
| 2026-07-09 | 08:32 | Write tests for existing code | 8 | 24 | 100% |
| 2026-07-09 | 08:22 | Fix a pricing bug | 9 | 29 | 86% |
| 2026-07-09 | 08:08 | Build a marketing page | 9 | 27 | 89% |
| 2026-07-09 | 07:54 | Build a marketing page | 2 | 2 | 0% |
| 2026-07-09 | 07:32 | Build a marketing page | 4 | 5 | 40% |
| 2026-07-09 | 07:27 | Build a marketing page | 4 | 4 | 25% |
| 2026-07-09 | 07:04 | Multi-step to-do list (4 topics), Multi-step to-do list + docs | 3 | 12 | 75% |
| 2026-07-09 | 06:35 | Multi-step to-do list (4 topics), Multi-step to-do list + docs | 3 | 12 | 83% |
| 2026-07-09 | 06:17 | Multi-step to-do list (4 topics), Multi-step to-do list + docs | 3 | 12 | 75% |
| 2026-07-09 | 05:51 | Multi-step to-do list (4 topics), Multi-step to-do list + docs | 3 | 12 | 83% |
| 2026-07-09 | 05:26 | Multi-step to-do list (4 topics), Multi-step to-do list + docs | 3 | 12 | 100% |
| 2026-07-09 | 04:04 | contact form | 3 | 6 | 17% |
| 2026-07-09 | 03:46 | contact form | 3 | 6 | 50% |
| 2026-07-09 | 02:58 | makealist nextjs landing, plain nextjs landing | 3 | 12 | 58% |
| 2026-07-09 | 02:39 | Plan, then build a marketing page, Build a marketing page (plain prompt) | 3 | 12 | 83% |
| 2026-07-09 | 02:22 | makealist blog dogs, plain blog dogs | 2 | 9 | 44% |
| 2026-07-09 | 02:18 | makealist blog dogs, plain blog dogs | 3 | 12 | 83% |
| 2026-07-09 | 02:10 | makealist do stuff, plain do stuff | 2 | 8 | 38% |
| 2026-07-09 | 02:07 | makealist do stuff, plain do stuff | 3 | 12 | 100% |
| 2026-07-08 | 17:09 | Scaffold a Next.js app, Multi-file task completion, Multi-step to-do list (4 topics), Multi-step to-do list + docs | 7 | 57 | 32% |
| 2026-07-08 | 16:18 | Write a blog post (guided), Plan, then write a blog post, Rewrite text to sound human, Research, then build a task list | 7 | 56 | 86% |
| 2026-07-08 | 16:15 | Write a blog post (guided), Plan, then write a blog post, Rewrite text to sound human, Research, then build a task list | 7 | 56 | 91% |
| 2026-07-08 | 14:44 | Scaffold a Next.js app, Multi-file task completion, Multi-step to-do list (4 topics), Multi-step to-do list + docs | 7 | 57 | 82% |
| 2026-07-08 | 14:27 | Scaffold a Next.js app | 4 | 8 | 100% |
| 2026-07-08 | 14:04 | Scaffold a Next.js app | 4 | 8 | 100% |
| 2026-07-08 | 14:01 | Scaffold a Next.js app | 4 | 8 | 13% |
| 2026-07-08 | 13:49 | Scaffold a Next.js app | 3 | 6 | 100% |
| 2026-07-08 | 13:26 | Multi-file task completion, Multi-step to-do list (4 topics), Multi-step to-do list + docs | 3 | 18 | 94% |
| 2026-07-08 | 13:14 | Write a blog post (guided), Plan, then write a blog post, caveman json coding | 3 | 18 | 78% |
| 2026-07-08 | 12:45 | Write a blog post (guided), Plan, then write a blog post, plain blog | 3 | 18 | 61% |
| 2026-07-08 | 11:57 | besure vague feature, Rewrite text to sound human, Research, then build a task list | 3 | 18 | 89% |
| 2026-07-08 | 11:50 | besure vague feature, Rewrite text to sound human, Research, then build a task list | 2 | 7 | 71% |
| 2026-07-08 | 11:32 | ask then blog, multi research articles | 3 | 18 | 100% |