AI Benchmarks: Someone Finally Rated This Week’s Models Without Selling Them

TL;DR: Three frontier models launched in three days this week, and every one of them claimed to beat the others — using tests the vendors published themselves. On 4 September an independent index scored all three on the same yardstick. Claude Fable 5.1 came first, GPT-6 Astra second. The result is worth knowing. How to read AI benchmarks at all is worth more, so most of this piece is about that.

What the independent index says

Artificial Analysis published version 4.2 of its Intelligence Index on 4 September — one day after the last of this week’s three launches.

The headline order:

Rank	Model	Note
1	Claude Fable 5.1 (Anthropic)	Leads overall
2	GPT-6 Astra (OpenAI)	A 4-point gain over GPT-5.6 Sol
3	Meta	Third-ranked lab

It isn’t a clean sweep. On GDP.pdf, a new long-context test that reasons across 4,592 pages of PDFs, the order reverses: GPT-6 Astra scores 33.2% against Claude Fable 5.1’s 26.2%. Which is itself the most useful thing in the whole index — the answer changes depending on the question.

Why vendor AI benchmarks aren’t enough

Look at what we reported this week, all of it accurately, all of it from the companies selling the product:

• Anthropic said Fable 5.1 scores 55.8% on Terminal-Bench 4.0, up from 42.0%.
• Google said Flash Cyber hits 86.2% on CyberGym and found a critical Chrome bug in under two hours.
• OpenAI said Astra outperforms both its own Sol and Anthropic’s Fable on cybersecurity.

Every one of those may be true. But vendor AI benchmarks are three different tests, chosen by three companies, each with an obvious interest in the result. We said as much on Friday — ignore the league tables, they were all published by the people selling the models.

That’s easy to say and unhelpful on its own, because it leaves you with nothing. The point of independent AI benchmarks is that every model goes through the same test, chosen by someone who isn’t selling any of them.

How to read AI benchmarks without being fooled

Here is what makes this particular index worth more than a vendor chart — and every one of these is a question you can ask of any scoreboard you’re shown.

1. Are the weights published? This one’s are. The index is ten evaluations in four groups: Agents 30%, General 30%, Coding 20%, Scientific Reasoning 20%. You can see what it decided matters. A leaderboard that won’t tell you how it weights things is telling you something.

2. Are some tests kept private? Yes, and this is the important one. v4.2 doubled the weight of private test sets to 40%, explicitly to make the index harder to game. Public benchmarks leak into training data — a model can score well because it has effectively seen the exam. Private questions can’t be studied for.

3. Do they retire tests that stop discriminating? v4.2 removed GPQA Diamond as saturated — everyone was scoring near the top, so it no longer separated anyone. A scoreboard that never retires anything is measuring history.

4. Do they publish their settings? Temperature 0 for non-reasoning models and 0.6 for reasoning ones, a 16,384-token output cap, and pass@1 scoring — the model must get it right first time, not best-of-ten. They state a confidence interval under ±1% across ten-plus repeats. You can disagree with those choices, which is the point: you can see them.

5. Does a second scoreboard roughly agree? Worth checking. A different leaderboard, Arena.ai, placed Gemini 3.8 Flash 14th in its Agent Arena and 7th in Text Arena — a reminder that “good” is rank-ordered differently depending on what’s being measured.

What we couldn’t verify

We checked something else, because after writing about manufactured “best software” pages on Thursday it would be poor form not to apply the same test here.

We could not find an independence or funding statement on either of Artificial Analysis’s methodology pages. Several third-party profiles assert the company has a strict policy that providers cannot pay for results, listing or methodology changes. We could not confirm that at source, and we’re not going to repeat it as fact on the strength of aggregator pages — which is precisely the failure mode Thursday’s piece was about.

To be clear about what that does and doesn’t mean: the methodology itself is unusually transparent, published in enough detail to argue with, and nothing we checked looked wrong. The absence of a disclosure statement on the pages we read is not evidence of a problem. It is simply a thing we couldn’t tick off, and you deserve to know which boxes we ticked and which we didn’t.

Why it matters if you don’t code

Honestly? Less than the ranking implies.

The models sit within a point or two of each other, they trade places depending on the test, and — the part that actually decides your week — your tools pick for you anyway. Replit routes automatically. Cursor’s Auto routes automatically. You will be moved onto whichever model your builder thinks fits the task, and a four-point index gap will not be the thing you notice.

So the useful takeaway isn’t “switch to Claude”. It’s this: when a tool or a vendor shows you a number, ask who chose the test. That single question does more work than memorising any leaderboard, and it applies to far more than AI benchmarks — it’s the same muscle as checking where an AI recommendation came from.

If you want one practical rule: care about independent AI benchmarks when you’re choosing something you’ll pay for over months. Ignore them entirely when you’re mid-build. The best model this week is not a reason to restart anything.

Who should care (and who shouldn’t)

• Deciding between paid AI subscriptions: the one group this genuinely helps. An independent same-yardstick comparison beats three vendor charts.
• Building a revenue app: interesting, not actionable. Your routing already handles it.
• Building an internal tool: ignore completely.
• Learning, or still choosing a tool: ignore the rankings, keep the questions. Which tool fits your project matters enormously more than which model is four points ahead — that’s what the 60-second quiz is for.
• Anyone who writes about AI: the five questions above are a decent standing checklist for AI benchmarks of any kind.

Our take

We’ve spent five days reporting model launches, and this piece is the honest correction to all of them.

Each launch came with numbers we passed on to you with attribution, because that’s what you do with a vendor claim. But attribution isn’t the same as verification, and a week of “Anthropic says”, “Google says”, “OpenAI says” leaves a reader no closer to knowing anything. An independent index scoring all three on one yardstick is the first thing this week that actually settles a question — and even then it settles a smaller question than the headline suggests, because the ranking flips on a different test.

What we’d hold onto is the method for reading AI benchmarks, not the medal table. Published weights, private test sets, retired benchmarks, stated settings, a second opinion. Those five checks will outlast Fable 5.1, Astra and every model that replaces them this autumn.

And the thing we couldn’t verify goes in the piece precisely because it’s inconvenient. A scoreboard that says “we’re independent” is making a claim, exactly like a vendor saying “we’re fastest”. We checked, we found the methodology impressively open and the disclosure absent, and telling you both is the whole job.

Want a tool recommendation that shows its reasoning rather than a ranking? The 60-second Vibe Coding Tool Finder quiz asks what you’re building and explains why it picked what it picked →

FAQ