ARC Prize Just Fact-Checked OpenAI's 99.9% — GPT-6 Astra Really Scored 62.7%

 

When OpenAI launched GPT-6 Astra on September 3, 2026, one number stole the show: 99.9% on ARC-AGI-3, the benchmark designed to be nearly impossible to game. Nvidia CEO Jensen Huang cited it days later when he declared on X that "AGI has arrived." Now the people who actually run that test have checked OpenAI's homework — and the number came back very different.

What ARC Prize actually found

The ARC Prize Foundation independently reran Astra on ARC-AGI-3 and scored it at 62.7% on its standard, provider-neutral harness — the same setup every other lab's model must clear. On OpenAI's own provider-adapter harness, the same model scored 98.6% at maximum effort and 99.9% at high effort. Same model, same effort, different scaffolding: 36 to 45 points of difference. ARC Prize's verdict was blunt: the 99.9% figure is not a comparable score, because it was not produced under the conditions every other model faces. The foundation published both runs.

Why the harness changes the score

The difference is what the model is allowed to carry between requests. Under ARC Prize's standard harness, each request starts cold: the model keeps only the notes it chooses to preserve, and nothing else crosses over. Under OpenAI's provider adapter, Astra preserves an opaque reasoning state between requests and compresses long conversations, letting it reuse prior work instead of solving from scratch. The adapter setup is not useless — on the 167 puzzle pairs both harnesses solved, adapter runs were about 3.66 times faster and used 49% fewer tokens. But a model-plus-scaffolding result presented as a model result is a different claim, and it should not be used to rank one lab against another.

Why this still matters — and what it doesn't change

First, the honest headline: 62.7% is still the highest score ever recorded on ARC-AGI-3 under the standard harness. Astra remains a genuinely strong flagship, especially for agentic, computer-use and coding work. Second, the warning: launch-week benchmark numbers are marketing until an independent party reruns them. OpenAI's flagship season has been a parade of huge claims, and this episode shows why outside verification matters. Third, the cost angle: the standard-harness rerun cost a reported $26,098 per run at maximum effort, and OpenAI's adapter run cost nearly $19,000. Benchmarking at the frontier is now a five-figure exercise, which is exactly why independent reruns are rare — and valuable.

The take

Don't throw out the 99.9% and don't worship the 62.7% either. The real lesson is boring and important: a benchmark score is only as honest as the setup behind it. When a lab reports a record number, ask one question — under whose harness? Astra is still the model to beat on ARC-AGI-3. It just isn't the 99.9% model the launch headlines sold.