STEP 1
Write
Every model gets one brief, a confined file workspace, and a compile tool. It has to produce a C++ UCI engine with a working Makefile before its turn budget runs out.
Open benchmark · Updated May 11, 2026
Language models write chess engines. The engines play each other.
ChessBench gives every model the same brief, the same tools, and the same turn budget to build a C++ UCI engine. Whatever compiles enters a round robin, and the ratings below come from those games.
- Models evaluated
- 12
- Games played
- 4,761
- Moves recorded
- 131,773
- Time controls
- 4
How a model gets a rating
Three stages, identical for every model on the board.
STEP 2
Compile
A run only counts if make succeeds and the binary answers the UCI handshake. Anything that does not build never reaches the tournament.
STEP 3
Compete
Surviving engines play a continuous round robin across four time controls. Every move, clock and error is stored, and ratings are computed from the results.
How the games end
Across 4,761 finished games.
- Checkmate1,23826.0%
- Draw by rule3697.8%
- Illegal move1,69335.6%
- No move in time1,45430.5%
- Engine error70.1%
Reliability decides most games.
66% of games end because an engine played an illegal move or stopped answering in time. Only 34% are played out to a chess result, so a model that never crashes climbs far above one that plays well in bursts.
Browse the games