AI Benchmarks

A personal reference of the evaluations I see cited most often when frontier AI models are released or compared. Grouped by what they measure. Not exhaustive — just the ones worth knowing.

Knowledge & Reasoning
MMLU / MMLU-Pro huggingface
The classic 57-subject multiple-choice knowledge test (MMLU), now largely superseded by MMLU-Pro, which expands answer options from 4 to 10 and strips trivial questions. Functionally saturated above 88% for frontier models, making top-end differences nearly meaningless — but still the historical reference point everyone cites.
GPQA Diamond github
~198 PhD-level biology, chemistry, and physics questions written to be “Google-proof.” Skilled non-experts with unrestricted internet score only ~34%, while PhD experts in-field average ~65%. One of the best discriminators in the 60–90% range, though the very top is approaching saturation.
Humanity’s Last Exam lastexam.ai
The current ceiling for closed-ended evaluation: 2,500 questions across 100+ subjects, contributed by ~1,000 experts across 500+ institutions, deliberately unanswerable by retrieval. As of late May 2026, leaders sit in the mid-40s percent (e.g., Gemini 3.1 Pro Preview around 44.7%) versus ~90% for human experts — the widest human–AI gap of any mainstream test.
ARC-AGI (1 / 2 / 3) arcprize.org
François Chollet’s fluid-intelligence series. ARC-AGI-1 is saturated; ARC-AGI-2 is the active grid-reasoning test; ARC-AGI-3, launched March 2026, is the first fully interactive version — games with no instructions, where humans score 100% and frontier AI scores ~0.51%. On the model cards of all four major labs.
SimpleBench simple-bench.com
From AI Explained: ~200 multiple-choice questions built around everyday reasoning rather than academic knowledge — spatio-temporal, social, and “linguistically adversarial” phrasings designed to tempt shortcut answers. Defining feature: humans still win. The non-specialist baseline is ~83.7%, well above frontier models — unusual in 2026 — making it a useful gut-check on commonsense robustness that high MMLU/GPQA scores miss.
Coding & Software Engineering
SWE-bench Verified swebench.com
A human-vetted subset of 500 real GitHub issues from Python repos; the model must patch the codebase and pass hidden unit tests. The de facto standard for agentic software engineering.
SWE-bench Pro scale.com
A harder, contamination-resistant successor from Scale AI using larger, more complex enterprise-style tasks, built because Verified is climbing fast. Increasingly the “next rung” for coding agents.
LiveCodeBench livecodebench.github.io
Competitive-programming problems collected continuously from contest sites with timestamps, so you can filter to problems released after a model’s training cutoff to avoid contamination.
Terminal-Bench tbench.ai
Tests agents on real command-line tasks in a sandboxed terminal: compiling, debugging, system admin, file manipulation. A leading measure of practical autonomous tool use.
ProgramBench Meta FAIR · Stanford · Harvard · May 2026
Clean-room “rebuild from binary” methodology: agents are given a compiled program — FFmpeg and SQLite are the flagship targets — and must reconstruct working source from scratch. The headline metric was 0% Resolved across 9 models at launch, with Claude Opus 4.7 leading the “Almost” tier at 3.0%; the May 11 update added GPT-5.5 (xhigh) at 0.5%. A live LessWrong debate questions whether the task is achievable as defined.
Math
AIME matharena.ai
The American Invitational Mathematics Examination, used yearly as a competition-math benchmark for multi-step reasoning. Frontier models now approach or hit perfect scores, so each new year’s exam matters for freshness and contamination.
FrontierMath epoch.ai
Hundreds of original, exceptionally hard problems crafted and vetted by expert mathematicians (Epoch AI), tiered by difficulty up to research level. Even top models score modestly on the hardest tier (Tier 4 in the mid-30s percent).
Long Context
Needle in a Haystack github
Greg Kamradt’s late-2023 pressure test: hide one out-of-place fact (the original was about eating a sandwich in Dolores Park) somewhere in a pile of Paul Graham essays, then ask for it back. Sweeping context length against insertion depth produces the familiar green/red heatmaps. Frontier models now pass it almost perfectly, so it shows that a model can find a fact in a long context, not that it can reason over one. It inspired most of the harder tests below.
RULER github
NVIDIA’s synthetic, configurable extension of needle-in-a-haystack: 13 tasks covering multi-key and multi-value retrieval, multi-hop variable tracking, aggregation (most-frequent words), and QA, at any length you choose. Its headline finding was that many models advertising 32K+ windows lost a lot of accuracy well before reaching them. It popularised the idea of an “effective context length” as opposed to the advertised one.
Counting-Stars github
A multi-needle test: sentences like “the little penguin counted N ★” are scattered through a long text (up to 128K tokens, in Chinese and English), and the model has to collect every one in order. Some variants add misleading look-alike stars. It tests whether a model can gather many pieces of evidence spread across a context, not just locate a single fact.
BABILong github
Hides the facts from the classic bAbI reasoning tasks (tracking objects, counting, deduction, induction) inside unrelated book text from PG-19, scalable from a few thousand to millions of tokens. Its NeurIPS 2024 paper found that popular LLMs effectively use only about 10–20% of their context, with accuracy falling steeply as the noise grows. Small fine-tuned recurrent-memory models beat much larger ones at extreme lengths.
LongBench v2 longbench2.github.io
From Tsinghua/Zhipu: 503 hard multiple-choice questions over real contexts from 8K to 2M words. The six categories are single- and multi-document QA, long in-context learning, long dialogue history, code repositories, and structured data. Human experts given 15 minutes manage only ~54%, and at launch the best direct-answer model scored ~50%. Reasoning models gained the most on it, so it measures understanding of the whole context rather than retrieval.
LOONG github
“Leave No Document Behind” (Alibaba, EMNLP 2024): multi-document QA over financial reports, legal cases, and papers, built so that every document contains evidence the answer needs. No document can be skipped. Tasks are spotlight locating, comparison, clustering, and chain of reasoning, across length tiers of up to 250K+ tokens. RAG pipelines do poorly on it because the answer depends on the full set of documents, not the top few retrieved chunks.
AA-LCR artificialanalysis.ai
Artificial Analysis Long Context Reasoning: 100 hard questions, each over a set of real documents (company reports, government consultations, legal and academic texts) totalling around 100K tokens. Answering requires combining information spread across the documents, not pulling out a single passage. It is part of the Artificial Analysis Intelligence Index, so it shows up on many model-comparison charts.
Agentic, Computer Use & Real-World Value
GDPval openai.com
OpenAI’s benchmark of real knowledge-work deliverables across 44 occupations from the top 9 U.S. industries by GDP contribution, graded by domain experts. The leading attempt to measure economically valuable output rather than puzzles.
OSWorld os-world.github.io
369 cross-application tasks for real computer use across actual operating systems, with a large human–AI gap at launch. The most rigorous GUI/desktop-control test; an “OSWorld-Verified” variant is now common on model cards.
τ²-bench github
Sierra’s customer-service simulation across retail, airline, and telecom domains. Dual-control design: both the agent and a simulated user modify a shared environment, and agent performance degrades sharply moving from single- to dual-control. Targets the multi-turn coordination real deployments need.
GAIA huggingface
466 questions chaining web browsing, file parsing, and multi-document reasoning. GPT-4 with plugins scored 15% at launch versus 92% for humans; top agents are now around 75%. A widely referenced general-assistant benchmark, though scores are notably scaffold-dependent.
Human Preference
LMArena (Chatbot Arena) lmarena.ai
Blind A/B “battles” where real users pick the better of two anonymous responses, aggregated into Elo ratings over millions of votes. The main signal for real-world preference rather than static test accuracy.