BlunderZeroHow the Astra chess benchmark was run
Every design choice was fixed before the games were played, and the record of each game contains enough to re-derive every claim on this site.
The four roles, kept apart
The model never saw engine output of any kind. Four components exist and only one of them talks to the model.
| Role | Talks to the model | Sees the engine |
|---|---|---|
| AI player | yes | never |
| Referee | only through the protocol layer | asks it for a move |
| Playing opponent (Stockfish) | no | is the engine |
| Post-game analysis (Stockfish) | no | runs after the game closed |
The FEN protocol
Each turn the model received the instructions, the chronological move history in SAN, and the current position as a FEN string. It received no legal move list, no evaluation, no best move, no search depth and no principal variation. It had no chess engine, opening book, tablebase or tool of any kind, and it was asked to reply with exactly one move in standard algebraic notation and nothing else.
Move legality
One normalisation is applied to a reply: leading and trailing whitespace is stripped. The token must then match a strict SAN pattern and equal the SAN the referee itself generates for a legal move, ignoring only the check or checkmate suffix. Anything else ends the game immediately as a loss with termination illegal_move. There is no retry, no reinterpretation and no repair, and the raw reply is stored either way.
The opponent
Stockfish 18 · UCI_LimitStrength enabled · UCI_Elo 1320 / 1500 / 1700 · 200,000 nodes per move · 1 thread · 16 MB hash
A fixed node count rather than a movetime, so the opponent is reproducible on the same binary regardless of machine speed. The opponent is asked one question, what is your move, and returns a move. No score, depth or principal variation is ever parsed.
Two separate analyses
The benchmark analysis is frozen inside each game record: Stockfish 18 at depth 18, used for accuracy, average centipawn loss and the inaccuracy, mistake and blunder counts. A second, independent analysis at depth 20 with MultiPV 3 was run afterwards for the video commentary only, labelled separately, and has no influence on any benchmark result.
What this does not measure
Supplying the FEN removes board reconstruction from the task. This is a test of move selection given the position. It is not a memory test, six games are not an Elo rating, and nothing here generalises from chess legality to general model reliability.