The Model Isn’t the Thing — And Nvidia Just Showed Why

TL;DR: On 21 August, Nvidia published research in which Claude Opus 5 completed every level of a reasoning benchmark it had previously been scoring around 30% on. Same model. What changed was the software wrapped around it — the AI agent harness. It’s a lab result on puzzle games, not a product you can buy. But it’s decent evidence for something worth internalising before your next tool decision: which model your builder uses matters less than how well it drives that model.

What Nvidia actually did

Nvidia Labs published results for AVO — Agentic Variation Operators — on 21 August. It’s an AI agent harness for long tasks, built around two things: memory that carries earlier attempts forward so the agent stops repeating itself, and a supervisor that watches the overall trajectory and redirects the main agent when it stalls.

Pointed at ARC-AGI-3, a benchmark of 2D games where the model has to work out the rules with no instructions, AVO cleared all 183 levels across 25 environments, scoring 100.00 on the benchmark’s efficiency measure and using about 12% fewer actions than VISTA, the previous best — which was also running Claude Opus 5.

The model doing the thinking was Claude Opus 5, the same model TechCrunch reports scored around 30% on that benchmark on its own, which was still the best of any model tested. Nvidia’s own summary is the line to remember: “The model matters, but the model is not the entire agent.”

Nvidia’s VP Adel El Hallak put the definition plainly to TechCrunch: “It is the model. It is the scaffolding around the model, which we call the harness, i.e. the set of tools that it utilizes.”

The caveat, from Nvidia itself: the researchers state the comparison “should not be interpreted as a controlled ablation”, because the systems differ in architecture, observation and memory. We’d hold the 30%-to-100% jump as a striking illustration rather than a clean measurement.

What an AI agent harness actually is

Strip the jargon and it’s simple. The model is the brain. The AI agent harness is everything else: what tools it can reach, what it remembers between steps, how it plans, and what happens when it gets stuck.

If you’ve used an AI builder, you’ve met a harness without being told its name. It’s the thing that decides whether the tool re-reads your project before editing, whether it notices it’s gone in circles, and whether it asks you a question or just guesses.

Nvidia made the same point in separate work in July, and quantified it: harness design alone, it wrote, “can account for double-digit swings in benchmark results and significant differences in token cost, with the same underlying model.”

Why it matters if you don’t code

You are told, constantly, that the newest model is the thing to chase. Tool marketing is built on it. Every launch is a model name.

This is evidence for the opposite, and it lands on a decision you actually make:

• Switching tools to get a newer model is usually the wrong move. Two builders on the same model can behave completely differently, because the AI agent harness around it differs.
• It explains something we wrote about last week. When people said Opus 5 felt worse, the specific complaint — it stopped asking, it assumes — described tool behaviour, not model intelligence. This is the same distinction with a benchmark attached.
• It’s an argument for judging tools on how they behave when things go wrong. Nvidia’s biggest win came from a supervisor that notices when the agent is stuck. That’s exactly the moment beginners lose hours.

Be clear on the limits, though. This is research on puzzle games, not app building. Nvidia’s harness is not something you can install. Nobody has shown these gains transfer to “build me a booking system”.

What to do

1. Stop tool-shopping on model names. “Now with the latest model” tells you very little about whether the thing will work for you.
2. Test how a tool recovers, not how it starts. Give it something slightly ambiguous and watch what happens when it goes wrong. That’s the harness showing itself.
3. Prefer tools that keep context. If you have to re-explain your project every session, that’s a weak AI agent harness, whatever model is underneath.
4. Treat “it asks good questions” as a feature. Being interrupted at the right moment is a sign of good scaffolding, not a slow tool.
5. Judge on your own project. A benchmark on 2D games is not your app. Run the same real task on two tools and keep the one that finishes.

Who should care (and who shouldn’t)

• Choosing between builders right now: the most useful group. Weigh behaviour over model badges — our comparison and the quiz both sort on fit rather than spec sheets.
• Tempted to switch because a rival advertises a newer model: this is your reason to wait.
• Happily building on one tool: nothing to do. Carry on.
• Learning, or not started: interesting but not urgent. Pick a tool that suits your project and start.

Our take

We think this is the most useful research finding of the month for non-developers, precisely because it argues against the thing everyone is being sold. The industry markets models because model names are easy to market. The part that decides whether you finish your project is much duller and much harder to put on a landing page.

We’d temper the excitement, mind. This is one lab, one benchmark, on games, with Nvidia’s own caveat that it isn’t a controlled comparison — and Nvidia has an obvious commercial interest in a world where the scaffolding around models is where the value sits. None of that makes it wrong. It does mean the honest summary is “strong evidence for something several people already suspected”, not “proven”.

The practical version is unchanged and boring: pick the tool that behaves well on your work, not the one with the newest model in its headline.

Not sure which tool actually suits what you’re building? Take the 60-second Vibe Coding Tool Finder quiz →

Some links on this page are affiliate links. If you buy through them we may earn a commission at no extra cost to you.

FAQ