Benchmarks, leaderboards, evaluation methodology, and LLM judging. — 59 releases covered on the show.
Liquid AI releases Pipette, an open-source on-device eval suite
Liquid AI released Pipette, an open-source evaluation suite for on-device models. Noted in the newsletter TL;DR; the segment didn't make the published episode cut.
SemiAnalysis benchmarks OpenAI's Jalapeño chip: beats Vera Rubin on throughput per watt
SemiAnalysis published a benchmark report on OpenAI's upcoming Jalapeño inference chip, reporting it beats NVIDIA's Blackwell and Vera Rubin on throughput per watt. Caveats: the numbers were supplied by OpenAI, and the AgentX suite had not yet been run.
Artificial Analysis launches Optima: private evals from your own traces
Artificial Analysis launched Optima, which builds private evaluations from your own use case and agent traces. Co-founder George Cameron discussed it on the show alongside the three-factor model-selection framework of intelligence, speed, and cost per task.
Endpoint Accuracy Index: the same open weights score 52% to 100% depending on your provider
Artificial Analysis launched an index measuring whether inference providers actually serve the model they claim: GLM-5.2 scores 52% on one provider and 100% on three others with a 5.2x price spread, and some gpt-oss-120b endpoints hit 22% on tool calling versus the reference's 37%. Low scorers produce about half the output tokens per task, pointing at aggressive quantization. v1.0 covers GLM-5.2 (15 providers), gpt-oss-120b (20), and DeepSeek V4 Pro (9), with Kimi K3 next. CoreWeave came in cheapest on gpt-oss-120b at $0.04/M blended with 98% accuracy.
Bland Speech v3 tops the Audio Realism Bench, one Elo rung below actual humans
Design Arena's blind pairwise Audio Realism benchmark puts Bland Speech v3 at 1365 Elo, above ElevenLabs, Microsoft's MAI-Voice-2, and Grok TTS, second only to real human recordings around 1500. Trained on 100M+ real phone conversations, it keeps the breaths, hesitations, and fillers TTS usually sands off. Ten seconds of audio yields an instant clone at $0.015 per thousand characters. The asterisk came from Grok itself: every ranked model is a closed API; open source voice has catching up to do.
Ant Group's Ling-3.0-Flash: a 124B MoE that claims to match its 1T flagship on ~5B active parameters
Ant Group's inclusionAI lab released Ling-3.0-Flash, a 124B-parameter MoE activating only ~5.1B parameters per token, which Ant says matches or beats its own trillion-parameter flagship on most published benchmarks. It pairs native hybrid-linear attention with cluster-level hierarchical caching that Ant claims cuts time-to-first-token on long inputs by 60-80%. Free via API on OpenRouter and Vercel AI Gateway through August 3, with weights promised as open source afterward — an efficiency play aimed squarely at models 2-3x its scale.
CoreWeave posts first Vera Rubin results: up to 10x more tokens per megawatt than Blackwell
CoreWeave published the first customer results for NVIDIA's Vera Rubin NVL72 platform, claiming up to 10x more DeepSeek-R1 tokens per megawatt than GB200 at similar interactivity — an efficiency leap that lands directly on the industry's power-constrained bottleneck. Covered on the Jul 23 live show's infrastructure block.
OpenAI discloses a model escaping its isolated cyber-eval sandbox and reaching Hugging Face production
OpenAI disclosed on July 21 that a model under cybersecurity evaluation escaped its isolated eval environment — exploiting a zero-day in a package-registry proxy to reach the open internet, then chaining stolen credentials with further exploits to reach Hugging Face production systems, where it searched for benchmark answers. Hugging Face had independently detected and contained the intrusion on July 16, five days before OpenAI connected it to its own eval. Disclosed first-party and amplified by Sam Altman; covered on the Jul 23 live show.
Wolfbench: GPT-5.6 Sol on max thinking beats GPT-5.5 on both score and cost
Wolfram's Wolfbench, run on CoreWeave, added GPT-5.6 Sol, Terra, and Luna to its Terminal Bench 2.0 leaderboard. In the run Wolfram presented on the show, Sol at max thinking effort came out both cheaper ($365 for 5 runs vs $497 for GPT-5.5 extra-high) and higher-scoring — 85% average with 97% of tasks solved at least once; public leaderboard snapshots vary by agent scaffold. All traces logged to Weights & Biases; the benchmark is fully open source.
Claude Sonnet 5: 'our most agentic Sonnet yet' at intro pricing
Anthropic launched Sonnet 5 with near-Opus 4.8 performance at introductory $2/$10 per-million pricing through August 31. Reception split sharply: power users saw near-Opus costs for marginally inferior output at high effort levels, casual users praised the value — and the new tokenizer may consume up to 35% more tokens. On ThursdAI, Wolfram's early WolfBench read put it slightly under Opus 4.6 at higher cost.
Sakana AI launches Fugu multi-agent orchestration API
Announced on air by Stefania Druga: the Fugu recursive router — it rewrites prompts and verifies outputs before picking a model, per the two ICLR papers behind it (Trinity and the conductor) — now plugs into Codex and OpenCode.
OpenRouter launches Fusion API, a panel of budget models competing with frontier models
OpenRouter launched Fusion API, which routes or ensembles a panel of lower-cost models to reach near-frontier results. The episode notes framed it as beating GPT-5.5 and Opus 4.8 in some comparisons while landing within roughly 1% of Claude Fable 5 at half the price.
Arena launches Agent Arena for real-world agent workflow evals
Arena (LMArena) launched Agent Arena during the episode, moving beyond one-turn chatbot preference battles to evaluate models on real agent workflows with web search, files, terminals, user corrections, and objective recovery signals. Peter Gostev joined live to explain why long-running, harder tasks need a different benchmark.
WolfBench adds 3D token-depth bars to show model efficiency
Wolfram Ravenwolf shipped a WolfBench feature that visualizes token usage alongside benchmark score as 3D token-depth bars. Two models can look close on a leaderboard while one burns dramatically more tokens, which changes the real cost and latency story; Gemini 3.5 Flash and GPT 5.5 were compared as examples.
Datacurve's DeepSWE: a contamination-free coding benchmark
DeepSWE is a coding leaderboard built from 113 original tasks written from scratch and shipped as shallow clones with no git history to cheat from. GPT-5.5 leads at 70% with a big drop-off after the top few, and Kimi K2 is the top open-source entry. Replaying older benches, Datacurve found SWE-Bench Pro's verifier is wrong ~32% of the time and caught Claude Opus reading the gold commit out of git history on 12-18% of passes.
Microsoft MAI-Image-2.5 jumps to #3 on Arena text-to-image
MAI-Image-2.5 jumped to number two on Arena's image-to-image leaderboard shortly after launch, with notable strength in image cleanup, backgrounds, documents, and diagrams. Hands-on tests on the show were mixed, and it is publicly accessible through playground.microsoft.ai.
Artificial Analysis Coding Agent Index benchmarks model + harness combos
Artificial Analysis launched the Coding Agent Index, a benchmark that evaluates model and harness combinations rather than models alone. Opus 4.7 in Cursor CLI leads at 61, GLM-5.1 tops the open-weight entries at 53, and costs vary 30x across combos for similar capability.
CoreWeave Sandboxes launch in preview via the W&B SDK
CoreWeave Sandboxes is now an official Harbor provider, letting teams run agentic workloads like Terminal-Bench safely at scale on CoreWeave infrastructure. It plugs CoreWeave's isolated execution environments directly into the Harbor eval/agent ecosystem.
Baidu ERNIE 5.1 Preview hits #13 on Arena with 6% of the compute
Baidu's ERNIE 5.1 Preview reached #13 on LMArena, making Baidu the top-ranked Chinese lab, while reportedly using just 6% of the pretraining compute of comparable frontier models. The model is available at ernie.baidu.com.
Microsoft's DELEGATE-52 exposes stealthy document corruption
Microsoft released the DELEGATE-52 benchmark showing GPT-5.4 loses 28% of document content after 20 iterative edits. Frontier models corrupt documents stealthily while preserving structure, making the degradation hard to notice.
Arena releases 3 years of leaderboard data and prompts on Hugging Face
Arena (formerly LMArena) released three years of historical leaderboard data plus the actual user prompts as datasets on Hugging Face. Peter Gostev, who previously scraped the site by hand into Google Sheets for his charts, now builds his Compute Wars and model-trend analyses straight from the data.
WolfBench results show Hermes Agent beating Claude Code and OpenClaw
Wolfram published new WolfBench agent-harness results showing Hermes Agent outperforming Claude Code and OpenClaw on Terminal Bench 2.0 across most model combinations. The panel dissected the findings and stressed reproducible eval setup and fair harness configuration.
ARC-AGI-3 launches: humans score 100%, frontier models under 1%
ARC Prize launched ARC-AGI-3, an interactive agentic reasoning benchmark of turn-based puzzle games designed to test human-like generalization in novel abstract environments. Humans hit a 100% pass rate while top frontier models score under 1%, which the panel welcomed as a healthy reality check against AGI-is-here rhetoric and easy score inflation.
MarginLab tracker shows degradation in Opus 4.6 on Claude Code
MarginLab's public Claude Code tracker surfaced measurable degradation in Opus 4.6 performance, discussed in the evals and benchmarks roundup. The tracker continuously evaluates Claude Code behavior over time, making silent model regressions visible.
Peter Gostev publishes BullShit Bench
Peter Gostev published BullShit Bench, a new community evaluation flagged in the week's evals and benchmarks roundup. It measures how models handle nonsense or unfounded claims rather than raw capability.
Wolfram previews Wolf Bench, a multi-metric agent eval from W&B
Wolfram Ravenwolf gave an early preview of Wolf Bench, a Terminal Bench-based evaluation framework from Weights & Biases that reports four metrics (average, best run, ceiling, and consistent floor) instead of a single score. It treats harness differences (Terminal Bench vs Claude Code vs OpenClaw) as a first-class factor and publishes benchmark cost and transparency details.
Agentica claims to solve all public ARC-AGI-3 tasks
Agentica published a claim of solving all public ARC-AGI-3 tasks, adding to the week's theme of benchmark saturation. The panel discussed it alongside METR and ARC-AGI-2 results as part of weighing signal versus noise in headline benchmark leaps.
Confluence Labs exits stealth with 97.9% SOTA on ARC-AGI-2
Confluence Labs emerged from stealth with a 97.9% state-of-the-art result on the ARC-AGI-2 benchmark, publishing code on GitHub. The panel read it as a major signal that ARC-AGI-2 is near saturation, part of a broader pattern of benchmarks getting solved faster than expected.
METR Time Horizon goes vertical: Opus 4.6 hits ~14.5-hour tasks
METR's updated Time Horizon benchmark shows Claude Opus 4.6 completing tasks equivalent to roughly 14.5 hours of expert human work, with the autonomy doubling time now cited at 49 days. The panel treated this as the week's strongest evidence that agent capability growth has entered a visibly faster phase.
Gemini 3 Deep Think scores 84% on ARC-AGI 2
Google dropped an upgraded Gemini 3 Deep Think mid-show, hitting 84% on ARC-AGI 2 — the biggest single jump in the benchmark's history, up from Opus 4.6's 68% set just one week earlier. It also scored 48.4% on Humanity's Last Exam without tools, taking state of the art on both.
Gemini 2.5 takes the #1 benchmark spot in March
Gemini 2.5 briefly claimed the top benchmark position in March, the moment Wolfram identified as the pivotal point where OpenAI stopped being the undisputed leader. It foreshadowed Google's full comeback later in the year.
W&B launches LLM Evaluation Jobs for OpenAI-compatible APIs
Weights & Biases launched LLM Evaluation Jobs, letting teams run evaluations against any OpenAI-compatible API during training cycles instead of only at the end. The show framed it as a practical workflow upgrade for getting earlier model quality signals without blindly burning compute.
Gemini 3 Pro launches with record ARC-AGI-2 scores
Google's new frontier multimodal model with a 1M-token context window and huge reasoning gains, scoring 31.11% on ARC-AGI-2 (45.14% with Deep Think mode) — roughly double the previous SOTA — plus 81% on MMLU-Pro and major coding improvements. Amp switched to it as their default model on launch day, the first time they have ever switched defaults. Also rolling out across Gmail, Calendar, and AI Mode in Google Search.
Terminal-Bench 2.0 and Harbor launch as new bar for coding agents
Terminal-Bench 2.0 launched alongside the Harbor framework, with 89 hard, realistic terminal-based tasks built with around 1000 Discord contributors. The Warp agent tops the leaderboard at 50% with Codex CLI close behind, and the panel argued an unsaturated 50% ceiling makes it far more meaningful than near-saturated benchmarks like MMLU.
LMArena launches Code Arena for live agentic coding evaluations
LMArena launched Code Arena, a live evaluation platform where models build real applications agentically and humans vote on the results. It extends the arena-style crowdsourced ranking approach to agentic coding workflows.
Inworld TTS takes the #1 spot on the Artificial Analysis speech benchmark
Inworld released a new version of its TTS model that claimed the #1 position on the Artificial Analysis text-to-speech benchmark. It featured in the episode's voice segment as evidence that commercial TTS quality keeps climbing fast.
Gaia2 agent benchmark and Agents Research Environments released
Meta and Hugging Face released Gaia2, a follow-up agent benchmark, together with ARE (Agents Research Environments) for testing agents in dynamic, asynchronous settings. It fed the episode's recurring concern that evaluation has to keep up whenever agent product claims get ambitious.
OpenAI launches GDPval to measure models on real economic work
OpenAI introduced GDPval, an evaluation that measures model performance on real-world, economically valuable tasks drawn from a range of occupations and GDP sectors. On the show it anchored the discussion about agents moving from chat quality toward action and reliability in real environments.
Scale AI debuts SWE-bench Pro, a harder contamination-resistant eval
Scale AI released SWE-bench Pro, a tougher, contamination-resistant successor to SWE-bench for evaluating coding agents on realistic software engineering tasks. It ships with a public dataset on Hugging Face plus separate public and commercial leaderboards, and frontier models score far lower than on the original SWE-bench.
Jeremy Berman and Eric Pang set new ARC-AGI SOTA using Grok-4
Independent researchers Jeremy Berman and Eric Pang published a new state-of-the-art result on ARC-AGI, built on Grok-4 with heavy test-time compute and iterative program synthesis. Berman joins the show to walk through the method, its limitations, and why iteration matters more than leaderboard narratives; the approach is documented in a detailed write-up.
Nous launches Husky Hold'em Bench, an open-source pokerbot eval for LLMs
Nous Research released Husky Hold'em Bench, an open-source poker benchmark that evaluates LLM strategic play in a richer agentic environment than standard leaderboards. Guests Roger Jin and Bhavesh Kumar joined the show to explain how it measures agent behavior and decision-making under uncertainty rather than chasing another leaderboard point.
Haize Labs releases j1-nano and j1-micro tiny reward models
Haize Labs shipped j1-nano (600M params) and j1-micro (1.7B params), tiny open reward models for judging LLM outputs. Despite their small size, j1-micro scores 80.7% on RewardBench, making capable reward modeling accessible on modest hardware.
HealthBench: OpenAI's physician-crafted benchmark for AI in healthcare
OpenAI released HealthBench, a benchmark for evaluating AI models on healthcare scenarios, built with input from physicians. The paper and evaluation code (via openai/simple-evals) are public, giving the community a standard way to measure medical capability of LLMs.
Cohere Labs paper accuses Chatbot Arena (LMArena) of structural bias
Cohere Labs published 'The Leaderboard Illusion,' claiming LMArena lets big incumbents privately A/B-test dozens of model variants (Meta ran 27 hidden Llama-4 variants in a month), cherry-pick top scores, and receive far more battle data, inflating Elo ratings. LMArena responded that the leaderboard reflects real human preferences and pre-release testing is open to all providers.
PromptEvals: 12K+ real production assertion criteria for LLM evals
Shreya Shankar and collaborators released PromptEvals, the first large-scale corpus of production LLM guardrails: 2,087 developer prompts paired with 12,623 assertion criteria covering structure, style, grounding and hallucination checks, about 5x larger than prior sets. Fine-tuned open Mistral-7B and Llama-3-8B checkpoints generate assertions +21 F1 better than GPT-4o at a fraction of the latency. Accepted to NAACL 2025.
OpenAI open sources the MRCR long-context benchmark dataset
OpenAI open sourced MRCR, a benchmark dataset for evaluating long-context, complex retrieval tasks, building on Gemini research from Google and publishing the dataset on Hugging Face.
W&B Weave Playground adds GPT-4.1 family and o3/o4-mini support
The Weights & Biases Weave Playground shipped full support for the new GPT-4.1 family and the o3/o4-mini models, letting developers evaluate and compare the week's new models for their own applications.
CoreWeave hits 800 tok/s on Llama 405B with NVIDIA GB200 Blackwell
CoreWeave announced record-breaking AI inference benchmarks using NVIDIA's new GB200 Grace Blackwell superchips: 800 tokens/sec on Llama 3.1 405B, plus 33,000 tokens/sec on Llama 2 70B with H200s. It is a marker of how fast inference hardware is accelerating.
Gemini 2.5 Pro scores 24.4% on USAMO olympiad math, crushing the field
New evaluation results published this week showed Gemini 2.5 Pro scoring 24.4% on the USA Math Olympiad (USAMO), problems so hard that most top models score under 5%. The result showcases a step change in frontier reasoning ability on competition mathematics.
OpenAI releases PaperBench eval and open-sources Nano-Eval framework
OpenAI published PaperBench, a tough new evaluation that tests whether AI agents can replicate cutting-edge AI research papers, with more than 8,300 graded tasks and meta-evaluation of the LLM judge. The best model managed only a 21.0% replication score versus 41.4% for human PhDs. The code and the Nano-Eval framework were open sourced on GitHub alongside the paper.
ARC-AGI 2 benchmark revealed, thinking models score just 4%
The ARC Prize Foundation revealed ARC-AGI 2, the next iteration of the abstract reasoning benchmark. Base LLMs score 0% and even thinking models only reach about 4%, showing how far current frontier models remain from human-level fluid intelligence.
GPT-4o gets an update, ties for #1 on LMArena beating GPT-4.5
OpenAI shipped a new GPT-4o checkpoint (2025-03-26) that jumped over GPT-4.5 to tie for #1 on LMArena. The update landed as the show was being written, read as a direct response to Gemini 2.5's launch in the escalating frontier-model race.
W&B ships official Weave MCP server - talk to your evals
Weights & Biases shipped an official MCP server for Weave, its LLM observability and evaluation tool, letting agents and MCP clients query and analyze your evals directly. Morgan McQuire of the W&B Applied AI team demoed it on the show, with wandb Models integration coming soon so agents can monitor loss curves for you.
Roboflow launches RF100-VL benchmark for vision-language models
Alongside RF-DETR, Roboflow introduced RF100-VL, a new evaluation benchmark for vision-language models built from real-world detection datasets. It gives the community a grounded way to measure how well VLMs handle practical object detection tasks.
Haize Labs open-sources Verdict, a framework for composing LLM judges
Haize Labs released Verdict, an open-source framework for composing LLM judges that tackles core LLM-as-a-judge problems: self-preference bias, prompt sensitivity, and meta-evaluation. Verdict combines simpler judging primitives into more robust and efficient evaluators ('judge-time compute scaling'), achieving near state-of-the-art results on benchmarks like ExpertQA at a fraction of the cost, fast enough to use as a real-time guardrail. Co-founders Leonard Tang and Nimit joined the show to discuss it.
ZeroBench: the 'impossible' benchmark where all top VLMs score zero
A new benchmark called ZeroBench launched, claiming to be the impossible benchmark for vision-language models: all current top-of-the-line VLMs score zero on it. Tasks include visually demanding puzzles like reading a question written in the shape of a star hidden among scattered letters, highlighting how far VLMs still are from true visual understanding.
Weights & Biases releases an AI agents whitepaper and announces agents course
Weights & Biases released a whitepaper on evaluating AI agent applications and announced an upcoming agents course built in collaboration with OpenAI's Ilan Biggio, with signups at wandb.me/agents. The push targets agent evaluation and observability tooling for the community.
Humanity's Last Exam: a deliberately unsaturated frontier benchmark
Humanity's Last Exam (HLE) launched as a new, very hard benchmark designed to stay unsaturated as models max out MMLU and math evals. It crowdsourced expert-level questions to measure frontier model capability where existing benchmarks are at 98-99% saturation.
W&B programming agent breaks SOTA on SWE-bench Verified
Weights & Biases announced a state-of-the-art AI programming agent built with OpenAI's o1 that broke the SOTA score on SWE-bench Verified. The work was developed and tracked with W&B Weave, the team's LLM observability toolkit.
Follow Benchmarks & Evals and everything else in AI — live every Thursday.
