What I learned from cheating clankers

In this solo Risky Business Features episode, James Wilson looks at the string of recent AI benchmark cheating escapades and how they’ve reshaped his thinking about LLMs and vulnerability research.
He explores the current benchmarks and exactly how agents go about gaming the testing system to achieve their tasks. And why understanding that has changed the way he interacts with his agents.
Show notes
OpenAI incident account and grading distinction
METR and Redwood independent assessment
FrontierCode original evaluation methodology
BrowseComp aggregation experiments
Sonnet 4 point 5 coding scaffold and parallel selection
Opus 4 point 6 BrowseComp setup
BrowseComp evaluation awareness and contamination investigation
Sonnet 5 point 5 effort and review subagents
FrontierCode 1 point 1 revision and rubric changes
Astra launch comparison and effort conditions
Original OSWorld task construction and worked examples
Sonnet 4 point 6 system card
Original OSWorld benchmark
OSWorld 2 environment and evaluation design
GPT 5 point 4 comparator table
GPT 5 point 5 comparator table
BrowseComp construction and grading paper
GPT 5 point 6 single model and Ultra configurations
METR investigation and artifact evidence
Released task definition
Released scorer for reference only
OpenAI paper sections 3 point 1 to 3 point 3
OpenAI accompanying explanation
Sycophancy to Subterfuge paper
Training a Misaligned Reward Seeker and evaluation appendix
GPT 5 point 2 AIME result