Reproduce the benchmark

The source, the six game records and both closed tracks are published at a frozen tag. Nothing on this site asks you to take a number on trust.

The frozen snapshot

The canonical reference is the tag astra-chess-benchmark-v0.1, not a moving branch.

Source: github.com/socialplayai/blunder-zero-ai-chess-benchmark

git clone https://github.com/socialplayai/blunder-zero-ai-chess-benchmark.git
cd blunder-zero-ai-chess-benchmark
git checkout astra-chess-benchmark-v0.1
python3 -m pip install -r requirements.txt
python3 -m pytest -q

The tag resolves to commit 5bf0561. Browse the frozen snapshot on GitHub.

What is in it

1. Six scored games, each with the PGN, the complete record containing every prompt sent and every raw response received, per-move token usage and cost, the post-game analysis, and a manifest with a sha256 of each artifact.
2. The reliability study: the frozen 40-position corpus with its digest, the execution order, all 160 scored responses, and the results document recording the futility stop.
3. The referee, protocol layer and adapters, with the test suite that asserts the integrity properties, including that no module on the path to the model imports the engine.

Replaying a game

The opponent is reproducible: check out the tag and replay the same command with the same fixed node count against the same Stockfish build. The model is not reproducible in that sense, which is why every raw response is stored rather than only the moves.

Checking a claim

Every number on this site is derived from those records. The accuracy and centipawn figures come from the analysis block inside each game.json; the legality counts come from the stored responses; the reliability result comes from the 160-response artifact and the preregistration that fixed its stopping rule before it ran.

← Back to the experiment