AI Benchmarks

The classic 57-subject multiple-choice knowledge test (MMLU), now largely superseded by MMLU-Pro, which expands answer options from 4 to 10 and strips trivial questions. Functionally saturated above 88% for frontier models, making top-end differences nearly meaningless — but still the historical reference point everyone cites.

~198 PhD-level biology, chemistry, and physics questions written to be “Google-proof.” Skilled non-experts with unrestricted internet score only ~34%, while PhD experts in-field average ~65%. One of the best discriminators in the 60–90% range, though the very top is approaching saturation.

The current ceiling for closed-ended evaluation: 2,500 questions across 100+ subjects, contributed by ~1,000 experts across 500+ institutions, deliberately unanswerable by retrieval. As of late May 2026, leaders sit in the mid-40s percent (e.g., Gemini 3.1 Pro Preview around 44.7%) versus ~90% for human experts — the widest human–AI gap of any mainstream test.

François Chollet’s fluid-intelligence series. ARC-AGI-1 is saturated; ARC-AGI-2 is the active grid-reasoning test; ARC-AGI-3, launched March 2026, is the first fully interactive version — games with no instructions, where humans score 100% and frontier AI scores ~0.51%. On the model cards of all four major labs.

From AI Explained: ~200 multiple-choice questions built around everyday reasoning rather than academic knowledge — spatio-temporal, social, and “linguistically adversarial” phrasings designed to tempt shortcut answers. Defining feature: humans still win. The non-specialist baseline is ~83.7%, well above frontier models — unusual in 2026 — making it a useful gut-check on commonsense robustness that high MMLU/GPQA scores miss.

A human-vetted subset of 500 real GitHub issues from Python repos; the model must patch the codebase and pass hidden unit tests. The de facto standard for agentic software engineering.

A harder, contamination-resistant successor from Scale AI using larger, more complex enterprise-style tasks, built because Verified is climbing fast. Increasingly the “next rung” for coding agents.

Competitive-programming problems collected continuously from contest sites with timestamps, so you can filter to problems released after a model’s training cutoff to avoid contamination.

Tests agents on real command-line tasks in a sandboxed terminal: compiling, debugging, system admin, file manipulation. A leading measure of practical autonomous tool use.

The American Invitational Mathematics Examination, used yearly as a competition-math benchmark for multi-step reasoning. Frontier models now approach or hit perfect scores, so each new year’s exam matters for freshness and contamination.

Hundreds of original, exceptionally hard problems crafted and vetted by expert mathematicians (Epoch AI), tiered by difficulty up to research level. Even top models score modestly on the hardest tier (Tier 4 in the mid-30s percent).

OpenAI’s benchmark of real knowledge-work deliverables across 44 occupations from the top 9 U.S. industries by GDP contribution, graded by domain experts. The leading attempt to measure economically valuable output rather than puzzles.

369 cross-application tasks for real computer use across actual operating systems, with a large human–AI gap at launch. The most rigorous GUI/desktop-control test; an “OSWorld-Verified” variant is now common on model cards.

Sierra’s customer-service simulation across retail, airline, and telecom domains. Dual-control design: both the agent and a simulated user modify a shared environment, and agent performance degrades sharply moving from single- to dual-control. Targets the multi-turn coordination real deployments need.

466 questions chaining web browsing, file parsing, and multi-document reasoning. GPT-4 with plugins scored 15% at launch versus 92% for humans; top agents are now around 75%. A widely referenced general-assistant benchmark, though scores are notably scaffold-dependent.

Blind A/B “battles” where real users pick the better of two anonymous responses, aggregated into Elo ratings over millions of votes. The main signal for real-world preference rather than static test accuracy.