A personal reference of the evaluations I see cited most often when frontier AI models are released or
compared. Grouped by what they measure. Not exhaustive — just the ones worth knowing.
MMLU / MMLU-Pro huggingface
The classic 57-subject multiple-choice knowledge test (MMLU), now largely superseded by MMLU-Pro,
which expands answer options from 4 to 10 and strips trivial questions. Functionally saturated
above 88% for frontier models, making top-end differences nearly meaningless — but still the
historical reference point everyone cites.
Humanity’s Last Exam lastexam.ai
The current ceiling for closed-ended evaluation: 2,500 questions across 100+ subjects, contributed
by ~1,000 experts across 500+ institutions, deliberately unanswerable by retrieval. As of late
May 2026, leaders sit in the mid-40s percent (e.g., Gemini 3.1 Pro Preview around 44.7%) versus
~90% for human experts — the widest human–AI gap of any mainstream test.
ARC-AGI (1 / 2 / 3) arcprize.org
François Chollet’s fluid-intelligence series. ARC-AGI-1 is saturated; ARC-AGI-2 is the
active grid-reasoning test; ARC-AGI-3, launched March 2026, is the first fully interactive version
— games with no instructions, where humans score 100% and frontier AI scores ~0.51%. On the
model cards of all four major labs.
SimpleBench simple-bench.com
From AI Explained: ~200 multiple-choice questions built around everyday reasoning rather than
academic knowledge — spatio-temporal, social, and “linguistically adversarial”
phrasings designed to tempt shortcut answers. Defining feature: humans still win. The
non-specialist baseline is ~83.7%, well above frontier models — unusual in 2026 —
making it a useful gut-check on commonsense robustness that high MMLU/GPQA scores miss.
ProgramBench Meta FAIR · Stanford ·
Harvard · May 2026
Clean-room “rebuild from binary” methodology: agents are given a compiled program
— FFmpeg and SQLite are the flagship targets — and must reconstruct working source
from scratch. The headline metric was 0% Resolved across 9 models at launch, with Claude Opus 4.7
leading the “Almost” tier at 3.0%; the May 11 update added GPT-5.5 (xhigh) at 0.5%. A
live LessWrong debate questions whether the task is achievable as defined.
Needle in a Haystack github
Greg Kamradt’s late-2023 pressure test: hide one out-of-place fact (the original was about
eating a sandwich in Dolores Park) somewhere in a pile of Paul Graham essays, then ask for it back.
Sweeping context length against insertion depth produces the familiar green/red heatmaps. Frontier
models now pass it almost perfectly, so it shows that a model can find a fact in a long context,
not that it can reason over one. It inspired most of the harder tests below.
RULER github
NVIDIA’s synthetic, configurable extension of needle-in-a-haystack: 13 tasks covering
multi-key and multi-value retrieval, multi-hop variable tracking, aggregation (most-frequent
words), and QA, at any length you choose. Its headline finding was that many models advertising
32K+ windows lost a lot of accuracy well before reaching them. It popularised the idea of an
“effective context length” as opposed to the advertised one.
BABILong github
Hides the facts from the classic bAbI reasoning tasks (tracking objects, counting, deduction,
induction) inside unrelated book text from PG-19, scalable from a few thousand to millions of
tokens. Its NeurIPS 2024 paper found that popular LLMs effectively use only about 10–20% of
their context, with accuracy falling steeply as the noise grows. Small fine-tuned recurrent-memory
models beat much larger ones at extreme lengths.
LongBench v2 longbench2.github.io
From Tsinghua/Zhipu: 503 hard multiple-choice questions over real contexts from 8K to 2M words.
The six categories are single- and multi-document QA, long in-context learning, long dialogue
history, code repositories, and structured data. Human experts given 15 minutes manage only
~54%, and at launch the best direct-answer model scored ~50%. Reasoning models gained the most
on it, so it measures understanding of the whole context rather than retrieval.
LOONG github
“Leave No Document Behind” (Alibaba, EMNLP 2024): multi-document QA over financial
reports, legal cases, and papers, built so that every document contains evidence the answer needs.
No document can be skipped. Tasks are spotlight locating, comparison, clustering, and chain of
reasoning, across length tiers of up to 250K+ tokens. RAG pipelines do poorly on it because the
answer depends on the full set of documents, not the top few retrieved chunks.
AA-LCR artificialanalysis.ai
Artificial Analysis Long Context Reasoning: 100 hard questions, each over a set of real documents
(company reports, government consultations, legal and academic texts) totalling around 100K
tokens. Answering requires combining information spread across the documents, not pulling out a
single passage. It is part of the Artificial Analysis Intelligence Index, so it shows up on many
model-comparison charts.