Case study: does planning first actually help?
Same task - a 3-page marketing site - sent to 8 models twice: once with a "/makealist" plan-first step prepended, once as a plain one-line instruction. 3 runs each, 48 runs total.
Key finding - counterintuitive
Planning first didn't help. It made things slightly worse: lower success rate, slower, and more expensive. This cuts against the common assumption (including one we made ourselves, from an earlier 3-LLM prompting survey) that having a model plan before it acts is close to a free win. At this scale, on this task, it wasn't.
/makealist (plan first)
92%
22/24 runs · avg $0.0151 · avg 82.5s
Plain instruction
100%
24/24 runs · avg $0.0123 · avg 60.6s
Plain prompt
Create a small marketing site with three separate pages: a homepage, an about page, and a pricing page.
/makealist prompt
/makealist Create a small marketing site with three separate pages: a homepage, an about page, and a pricing page.
Every model, both conditions
Timed out at 300s on 2 of 3 makealist runs with 0 files - the plan-first step itself ate the clock before any edit happened.
What each model actually built
Fresh renders of the /makealist condition's homepage, one per model, using today's harness (not the exact original stat-generating runs, which predate this harness saving full file content) - illustrative of the real output quality, not a re-measurement.
7 of 8 models scored identically either way - the entire gap came from deepseek/deepseek-v4-flash timing out twice under the plan-first condition. A small sample, but a real, unfiltered result - not cherry-picked to make either side look better.
Full testing history and methodology: /stats.