Gauntlet Loop: 76% of Published Examples Fail Their Own Test
We coded every quality bar in the public gauntlet loop material against the four tests the technique itself demands. Four of seventeen clear all four. The reference material people are copying fails its own standard three times out of four.
Jump to the data, or start here, because the number only makes sense once you know what the bar is for.
Your coding agent has one failure mode that costs you more than any other: it stops too early.
This is not a Claude Code problem or a Cursor problem. It shows up in every agent, because it is a prompting problem.
Not because it ran out of capability. Because nothing in the prompt told it what "done" looks like, so it invented a definition, met it, and reported success. You got a landing page that works. Not one that wins.
The gauntlet loop fixes that with a single structural change: the agent never grades its own work, and the thing it gets graded against is real.
Contents
- What a gauntlet loop actually is
- Where it came from
- Why a bar beats a rubric
- The four tests a bar has to pass
- The audit: what 17 published bars actually score
- The five rules that make the loop work
- A working prompt you can paste today
- The five ways it breaks
- Which agent to run it in
- Free tools and data
- FAQ
What a gauntlet loop actually is
A gauntlet loop is a prompt structure. You give a lead agent a goal plus a named real-world reference. The agent splits the goal into small pieces. Each piece gets a builder subagent and a separate critic subagent with a clean context window. The critic puts the built artifact next to the reference, labels stripped, and picks the better one. When your work loses, the critic names the single biggest gap, the builder closes it, and the piece goes back through with a fresh critic.
The loop does not end after three rounds. It ends when your work wins, or when you stop the run.
That is the whole mechanism. Everything else in this article is about the one part people get wrong.
Where it came from
Matt Shumer posted a browser first-person shooter built by Claude Opus 5 and named the technique afterwards. His own account of the run: he gave Claude Code one prompt, then left it alone, and it spent many hours working, spawned a fleet of subagents, wrote roughly 55,000 lines of code, and generated every texture, mesh, animation, and sound in code from scratch.
The post drew enough disbelief that he open sourced the result. Skeptics started running versions of the prompt to prove it would not work, and ended up with fully working games of their own.
Coverage of the run landed on the same summary: hand an agent a real, inspectable bar instead of a vague instruction, let it split the job into small pieces, and route each piece through a critic that never sees the builder's own reasoning.
Sources worth opening yourself:
- Claude of Duty on GitHub, the full source
- The original prompt
- Shumer's own write-up
- Decrypt's coverage
Sidenote. The pattern is not new in principle. Test-driven development, code review, and editorial review all separate the maker from the judge. What changed is frequency. An agent can run that separation dozens of times an hour, which is not something a human review cycle can match.
Why a bar beats a rubric
Most people already try to hold their agent to a standard. They write a rubric: "clean, modern, accessible, fast." Then they ask the agent to check its work against it.
That fails for a mechanical reason. A rubric is words. The agent wrote or agreed to those words, and it interprets them at grading time. Every criterion becomes a thing it can argue it satisfied.
A bar is an artifact. It exists outside the conversation. The agent cannot redefine it mid-run.
There is a second reason, and it is the one almost nobody accounts for. Scores drift. A critic that saw round three grades round four against round three, not against the goal. Improvement always looks like progress. A blind pick between two artifacts has no memory to drift against.
Further reading:
The four tests a bar has to pass
This is where every failed gauntlet loop fails. Not in the loop. In the bar.
1. Named. A specific thing, not a category. "Stripe's pricing page" is a bar. "Award-winning SaaS design" is a mood.
2. Fetchable. The critic must be able to screenshot it, read it, run it, or open it. If it cannot reach the reference, it does not stop and tell you. It invents a comparison and approves everything you build. This is the single most common way a loop silently turns into a rubber stamp.
3. Comparable. Both artifacts have to sit side by side so a judge can pick one. A running app is not comparable to a spec document. A landing page is not comparable to a Figma board.
4. Capturable. The reference has to be frozen to disk before round one. Screenshot it, save the text, run it and log the output. A bar that gets re-fetched every round will eventually fail to load, and a critic holding a missing reference passes everything.
That fourth test is not in the original write-up. It comes from watching loops run long enough for a live URL to time out. Freeze the bar. It costs one step and removes the failure mode entirely.
Two-sided bars beat one-sided bars. Where both taste and a number exist, use both and require the work to win on each. For a CLI tool, that is a named implementation plus its benchmark. For a landing page, a named competitor plus a Lighthouse score.
Bar patterns by job type
| What you are building | What the bar should be |
|---|---|
| Landing page, app UI | A named live page, captured at desktop and mobile |
| Long-form article | Two or three named published posts by a named writer |
| CLI tool, library | A named tool's implementation plus its benchmark number |
| Research brief | A named analyst report or a named paper's methods section |
| Game, interactive | A named title's specific level, video captured |
| API or backend service | A test suite, a latency target, and a failure-recovery test |
Shumer makes the same point about ambition: a hard bar does not need to be realistically reachable, and his game did not become better than Call of Duty, but the reference kept the agent from stopping when the game merely looked pretty good for AI.
The audit: what 17 published bars actually score
The four tests are easy to nod along to and hard to pass. So we took every quality bar named in the canonical public material, Matt Shumer's original prompt and article, the three public skill repositories, and two pieces of secondary coverage, and coded each one against all four gates.
Four of seventeen clear. 76% fail.
Named is the weakest gate, but the near-misses are the story
Four bars clear three gates and fail only Named. Every one of them fails the same way: it describes the kind of reference rather than naming one.
"a named writer's actual published posts"
"a named tool's implementation plus its benchmark"
Those are instructions to pick a bar, wearing a bar's clothes. Copy either verbatim into a live run and you have shipped a template. The agent then does what agents do with an unfilled slot, which is fill it, quietly, with whatever it feels like.
Capturable is the gate nobody talks about
It fails at 47%, second worst in the corpus. Only three of seventeen bars specify freezing the reference before round one. Everything else assumes the critic can reach a live thing on every round.
That assumption holds for an hour. Then a page changes, a rate limit trips, or a URL times out, and the critic is holding nothing. It does not stop and tell you. It approves.
The four that clear
| Bar | Source |
|---|---|
| The most recent Call of Duty games | Shumer, original prompt |
| Actual Call of Duty screenshots | Shumer, article |
| A collection of Paul Graham paragraphs | Shumer, article |
| Marketing site against Linear in Next.js | duolahypercho, example |
Three of the four are Shumer's own. The shared pattern is unglamorous: a proper noun plus a medium the critic can physically hold.
What this is not. These are published example bars, not prompts scraped from real sessions. Nobody can produce that second dataset from public sources, and any figure claiming to represent "prompts in the wild" is either instrumented from a tool the author owns or invented. The limitation is also the point: these 17 lines are what a person copies on their first attempt, and 13 of them teach a bar that will not hold.
Full corpus, the scanner, per-row coding notes, and source URLs for all 17: the Bar Audit. CC BY 4.0. Disagree with a coding decision, change the cell and rerun.
The five rules that make the loop work
1. Give the goal, not the implementation
Shumer's original game prompt contained no detailed architecture, no list of systems, and no explanation of how the renderer should work. When you prescribe the architecture, you replace the model's judgment with your own, and your own is working from less context about the artifact than the agent has.
Give it the destination. Let it pick the route.
2. Let the agent split the work
Not you. The lead agent understands the artifact and can decide which parts separate cleanly and which have to move together. Four to six pieces is the working range, and subagent orchestration is what makes that range practical. Fewer, and the critic judges everything at once and picks vaguely. More, and rounds crawl.
3. Never let the builder grade itself
The builder remembers every decision it made and why. That makes it excellent at explaining why its output is reasonable. You do not want reasonable. You want an independent judgment from something that has no stake in the previous round.
Spawn the critic fresh. Give it the goal, the captured reference, and the actual artifact. Give it nothing about how the work was made.
4. Make the critic open the real thing
Not a summary. The rendered page in a browser, the running app, the finished prose, the actual test output. A critic reading the builder's report is grading a press release.
The point generalizes past code. An agent claiming a page is now responsive is weaker evidence than the page opened at mobile width. A claim that something is faster is weaker than a measured frame time.
5. Never set a round count
The exit is winning or you calling it, and Shumer's game was still improving when he stopped the run. Three rounds is not a stopping rule, it is a budget disguised as one.
Better: require two consecutive wins from two independent fresh critics. One win can be a soft critic. Two rarely is.
A working prompt you can paste today
Fill the brackets, paste into a fresh Claude Code or Codex session, and leave it alone.
Build [THING] and beat [NAMED BAR] on [2 OR 3 DIMENSIONS THAT MATTER].
Before you build anything, capture the bar. Screenshot it at desktop and
mobile, save the text, or run it and log the output. Store it in /reference.
Every comparison from here uses that frozen capture, never a re-fetch and
never your memory of it.
Split the work into 4 to 6 parts. Each part gets a builder subagent and a
separate critic subagent.
The critic starts with fresh context. It gets the goal, the captured
reference, and the actual artifact. It never sees the builder's reasoning,
the builder's notes, or which round we are on.
The critic opens the real thing: the rendered page, the running app, the
finished prose, the test output. Never a summary the builder wrote. It puts
ours and the reference side by side with labels stripped and the order
randomized, then picks the better one. A pick, not a score out of ten. If
ours loses, it names the single largest gap in one sentence and stops there.
The builder fixes that one gap. Then it goes back through the gauntlet with
a new critic.
Keep looping until two consecutive fresh critics pick ours, or until I stop
you. No fixed number of rounds. Do not tell me it is good enough.
Maintain a simple live HTML progress page I can open on my phone.
Stop and ask me before you spend money, publish anything live, delete
outside the project folder, or change the bar.
Stack: [STACK]. /loop until it wins. Fan out subagents and ultracode.
Two lines are Claude Code specific, and both are covered in our slash command reference. /loop reruns a prompt until you stop it and ultracode opts the turn into heavier multi-agent orchestration. On any other agent, swap the last line for: keep repeating this cycle until the critic picks ours, and run builders and critics as parallel subagents.
Tip. The randomized order line matters more than it looks. Stripping labels is not enough if your artifact is always presented second. Position bias is real in blind comparisons, and a critic that learns "second is ours" is no longer blind.
Further reading:
The five ways it breaks
| Failure | What you see | Fix |
|---|---|---|
| Vague bar | Critic approves round one | Run the bar through the four tests before starting |
| Builder grades itself | Rounds get faster, quality flat | Force a separate agent with a clean context |
| Soft critic | Scores creep up, no rebuilds | Binary pick, never a rating |
| Fixed round count | Work stops mid-improvement | Exit on two consecutive wins or your call |
| Critic reads a summary | Confident approval of broken output | Require it to open the artifact itself |
The first one accounts for most of them. If you only fix one thing, fix the bar.
Which agent to run it in
A gauntlet loop needs an agentic harness, not a chat window. Pasting one into a normal chat will not produce the same result, because the model needs to open files, run code, render the result, inspect screenshots, use tools, and spawn other agents.
Shumer's own split: Claude Code with Opus 5 as the default for visual or creative work, since subagents each get their own clean context window, and Codex for backend engineering where visual creation matters less.
The clean-context property is the part that matters here. If your harness cannot give a subagent a genuinely fresh context window, you do not have independent critics, you have the same agent wearing a hat.
Compare the options: AI coding tools, honestly ranked, including where Cursor and Windsurf sit on subagent support.
Watch it without babysitting it
For long runs, have the lead agent maintain a live HTML page or a workbench file it updates as it goes, with screenshots, drafts, or test results. Open it from your phone. You get visibility without interrupting the run, which is the thing that actually kills long loops.
Video walkthroughs
Loop engineering fundamentals, the five building blocks, and two loops built live:
https://www.youtube.com/watch?v=yaJAMagc_sE
Planning and running loops inside Claude Code end to end:
https://www.youtube.com/watch?v=aVO6E181cNU
Free tools and data
The Bar Test
Most people cannot tell a strong bar from a weak one until four rounds have burned. Given the 76% failure rate above, that is not a personal failing, it is what the examples teach.
The Bar Test scores your proposed reference against the four gates, flags mood language automatically, tells you which gate broke and why, and writes the filled gauntlet loop prompt once all four hold.
No signup, runs entirely in your browser, nothing is stored.
The Bar Audit dataset
All 17 coded bars, the scanner that produced the Named column, the raw output, and a source URL on every row.
Both are CC BY 4.0 and free to embed. If you cite the 76% figure, a link back is the only ask.
FAQ
Is a gauntlet loop the same thing as /loop?
No. /loop is a Claude Code feature that repeats a prompt until you stop it. A gauntlet loop is the structure of what gets repeated: build, blind compare against a real reference, close one gap, repeat. /loop without a bar just repeats mediocrity faster.
Does it work outside code?
Yes. The requirement is that the output can be inspected and compared, which covers writing, design, research, and marketing as easily as software. This article was built with one, which is the same reason vibe coding works better with a reference than without. The bar changes, the structure does not.
How much does a long run cost?
More than a single generation, by a lot, because you are paying for parallel builders, parallel critics, and repeated rounds. Set a spend gate in the prompt and check the live progress page rather than discovering the bill afterwards.
Can I use a competitor's page as my bar?
For comparison and critique, yes, that is ordinary competitive benchmarking. Copying their assets or copy is a different thing and not what this pattern does. The critic judges whether yours is better, it does not clone theirs.
Why two consecutive wins instead of one?
One win can come from a critic that happened to be lenient. Two independent fresh critics agreeing is a much weaker coincidence. It costs one extra round and removes the most common false positive.
Where does the 76% figure come from?
From coding all 17 quality bars named in the canonical public gauntlet loop material against the four gates. Four clear all four. The corpus, the scanner, and the per-row notes are published so the number can be checked or disputed. It measures published examples, not prompts from real sessions, and the writeup says so plainly.
Which gate should I worry about most?
Named fails most often at 65%, but Capturable is the one that bites late. A bar that fails Named usually produces obviously bad output in round one. A bar that fails Capturable works fine until the reference stops loading, at which point the critic starts approving everything and the run looks like it is going well.
What is the minimum viable version?
Named reference, separate critic, blind pick, no round count. Everything else in this article is hardening.
The one-line version
Give the agent something real to beat, never let it judge itself, and do not tell it when to stop.
Most agent output plateaus because nothing is holding it to a standard. A gauntlet loop hands it one it cannot argue with.
And on the current evidence, three out of four published bars are not that standard. Check yours before you spend the compute.
Keep learning
- The Claude Code setup we actually use (harness, config, and the settings that matter for long runs)
- How Claude Code subagents share context (the mechanism the blind critic depends on)
- Turning a prompt into a reusable skill (package the loop so you stop pasting it)
- AI coding tools, honestly ranked (which harnesses can actually give a subagent a clean context)
- Vibe coding without the slop (where a bar fits into a looser workflow)
- The Bar Test (score your reference in thirty seconds)
- The Bar Audit (17 published bars, coded, with the scanner and raw data)
Technique credit: Matt Shumer, who wrote the original prompt and named the gauntlet loop while building Claude of Duty.
