Settlers / Research

73 pages · Search titles and descriptions

↑ ↓ to navigate · Enter to open · Esc to closeLocal search
Play the game

An offline arena that agrees with the protocol arena

The engine arena plays complete games inside one process using the server's rules engine. A seat receives only what a protocol participant would receive: its own redacted observation and its own event batches, with theft receipts delivered only to thief and victim. Games take an explicit seed, so every lineup rotation sees the same board and opening order, and the 32 cores of this machine finish 512 builder games in under a second.

Builders on identical boards

95% interval
fast builder (slot 0)19.9% · n=512
ETA builder (slot 1)30.1% · n=512
fast builder (slot 2)19.9% · n=512
ETA builder (slot 3)30.1% · n=512
0%50%100%
Per-slot win rates over 512 games: 128 seeds, each played once per rotation of the four slots. Run 25a7ddd7-8c86-44d7-9bb8-3582f9cf219e: 512 of 512 planned games completed; engine arena, paired seeds.

This tier is for development, sweeps, and structural measurements. It is not the authoritative protocol arena: there is no network, no turn clock, and player trades resolve by asking recipients in seat order. The experiment program defines both tiers and requires a fresh protocol cohort before the evidence boundary changes.

The frozen reference loses to the builders

Run 4969a9d0-cbcf-4b74-9b57-1e25f09ffa2c seated the reference expectimax-v1 at its default configuration against two ETA builders and one fast builder. It won 34 of 256 slot-games (13.3%) against ETA's 30.1% and 32.4% in the other two slots. The paired contrast against ETA is −0.168 (95% interval −0.251 to −0.085). Its decisions took 338 ms on average.

The frozen reference against the builders

95% interval
expectimax-v1 reference (slot 0)13.3% · n=256
ETA builder (slot 1)30.1% · n=256
fast builder (slot 2)24.2% · n=256
ETA builder (slot 3)32.4% · n=256
0%50%100%
Per-slot win rates over 256 games: 64 seeds, each played once per rotation. Run 4969a9d0-cbcf-4b74-9b57-1e25f09ffa2c: 256 of 256 planned games completed; engine arena, paired seeds.

Unregistered exploratory runs on 8 seeds gave the same picture at larger budgets (depth 2 with a 256 node allocation, and depth 4 with beam 6 and a 512 allocation): 4 wins of 32 in each. Depth did not help the reference, which points at its leaf and its trading rather than at search effort. The reference stays frozen as the comparator that the tunable search is measured against.

Agent notes

Experiments 3b4587ea-1ab3-42bb-99fb-597a3324cd77 (builders, run 25a7ddd7-8c86-44d7-9bb8-3582f9cf219e) and e5972f4e-88d1-4832-b70b-e279a536430a (reference, run 4969a9d0-cbcf-4b74-9b57-1e25f09ffa2c) are engine-arena registrations under the log/engine-arena study. Reproduce with just build-arena and just engine-run EXPERIMENT from this repository; the run manifest records the tournament binary hash and both repositories' Git state. The arena is server/crates/arena; its seats, trade resolution, and records are described in server/docs/expectimax.md.

The paired contrast is the per-seed mean of slot-0 wins minus slot-1 wins over the seed's rotations, with a normal-approximation interval over seeds. It is a development-tier statistic and not the protocol arena's preregistered test. Attribution: Claude Fable 5.1 (claude-fable-5-1) through Claude Code.