Lookup tables as the leaf of the search

Why a learned leaf

Every planned turn of expectimax-v2 ends by asking a leaf how good the position is, and the ablations showed that the search's lookahead and dice scenarios matter far more than any single hand-written term. The Drop7 n-tuple tables suggested the other route: keep the search, and replace the leaf with numbers learned from play. A lookup table can hold a distinct number for every small pattern the board can show, so it can learn that a city on a five-pip ore hex is worth more than its production rate says, or that a particular hand and building count is a dead end, without anyone writing the rule down.

What the tables read

Each position is read from one seat's point of view through three families of windows, and every window reads only what that seat can see in a sampled world: the board and layout, its own hand and development cards, and public counts.

The value is the sum of the looked-up entries, in win-probability units. A hex window is shared by all nineteen hexes and an intersection window by all fifty-four corners, so one entry can be active several times in a position; the learning step weights each entry by that multiplicity. Inside the search the sum is scaled by 1,000 so that one win equals ten points on the hand-written currency, and nothing else changes: the same turn planner, the same opponents' round under common random scenarios, the same iterative deepening.

How the tables learn

Thirty-two threads play self-play games on shared tables without locks. Every seat plans its turn with the depth-1 search over the tables on the true state; the leaf never sees a hidden hand, so the tables cannot learn from information a seated player would not have. At game end each seat's chain of end-of-turn positions is updated backward with TD(λ = 0.7) toward 1 for the winner and 0 for everyone else, plus 0.05 per point gained, with temporal-coherence step sizes that shrink once an entry's errors stop pointing the same way.

Blank tables cannot finish a game: every candidate looks equally good, nobody builds, every game reaches the turn cap, and the tables learn that everything is worth zero. The first 20,000 games are therefore played by the fast builder for every seat while the tables watch, at 15,000 games per second. Self-play then runs at 45 to 65 games per second, and every chunk the frozen tables play a validation tournament as the depth-2 leaf against the hand-written leaf in the same search with an ETA and a fast builder on fixed deterministic seeds. The validation point with the largest paired margin is the candidate.

The pilot

Six layouts trained for eighteen minutes of wall time each, 20,000 builder games and about 44,000 self-play games, and were validated on 24 seeds in every rotation (96 games). The margin is the per-seed difference in wins between the tables and the hand-written leaf.

Two readings are clear even at this budget. The portfolio tables carry the mechanism: without them the board windows alone lose to the hand-written leaf, and alone they already beat it. The hex windows add to the portfolio tables, while the intersection windows, with eight and a half million entries and only tens of thousands of games, add noise. Every arm was still improving at its last validation point. The point reward helped the full layout by a margin inside the noise. The main run trains hex plus portfolio from scratch under a three-hour wall with a plateau rule.

Compact records of every arm (configuration, progress rows, validation summaries, stop reason) are under records/training/ntuple-pilot/.

The main run

Hex plus portfolio tables trained again from scratch on 32 threads: 20,000 builder games, then self-play at about 44 games per second, validated every 32,000 games on 32 fixed deterministic seeds in every rotation (128 games) as the depth-2 leaf against the hand-written leaf with the two builders, and played one ply in a second tournament. The run stopped on its plateau rule after eight validation points, 256,000 games and 1 hour 50 minutes of wall time, when the mean margin of the last three points (+0.245) was no longer above the mean of the three before (+0.333).

Every point beat the hand-written leaf inside the same search, and the point at 96,093 games, with the largest margin, is the frozen candidate (artifacts/ntuple/hex-portfolio-main.bin, SHA-256 f8e61e66…, receipt under records/artifacts/). About half a million of the 3.6 million entries had been touched by then. The margin on the validation seeds is noisy from point to point, and choosing its maximum makes +0.375 an optimistic reading; the fresh seeds below are the test. The tables played one ply, with no search at all, improved at every point and won 65 of the last 128 games against a table where the hand-written leaf searched two turns ahead; that arm is registered on fresh seeds too.

What the tables learned

The hex table can be read directly. For a window holding exactly one own settlement and nothing else, the entry rises with the pips of the hex number on every resource, and ore and grain are worth most; a city is valued above a settlement on the same hex almost everywhere. No term for pips, resources, or cities was written down; the tables learned these from which positions went on to win.

The portfolio tables can be read the same way. The race table, in the mid-game turn bucket, rises steeply with own points and falls with the best opponent's: at eight points against an opponent on five the entry is +0.32, at four against eight it is −0.09. The hand table values every card above an empty hand, ore most (four ore +0.07 above nothing; four lumber +0.05). The production table's marginal effect is small: averaged over the profiles play reached, one more pip in any resource moves the entry by under 0.01, so the production signal lives in the hex windows, which see each building on its hex directly.

Looking into the search: expected outcomes

The search can now say what it expects. In explain mode a report carries, for the chosen move, the distribution of end-of-turn positions it leads to at every planned level: each position's probability along the chosen move (from dice, purchases, thefts, the sampled hands, and the common random scenarios of the opponents' round), the value the leaf gave it, and what the player has by then. Positions are merged by their builds, cards, points, and whether the game has ended, so the reading is "a city at 19 and a road, with probability 0.4" rather than one entry per sampled hand. The decision itself is unchanged; only positions the search would have discarded are kept.

In the recorded decision above the tables settle at 34 and then expect, by the end of their next turn, to have added nothing further with probability 0.56, one development card with 0.17, a road on 50 with 0.09, two cards with 0.06, and a fifth point in a few percent of futures. A researcher can read from this what the policy is playing for, and a disagreement between those expectations and what actually happens is the place to look for a leaf that is wrong.

Why the tables spend more search time

The tables are not the expensive part. One evaluation of the hex plus portfolio tables costs 0.64 microseconds against 1.65 for the hand-written terms on the same positions. The search over the tables simply expands more: per decision at depth 2 it visited 2,689 own nodes and 393 chance branches where the hand-written leaf visited 1,292 and 195, and at depth 3 the ratio is the same. The tables steer play toward purchases and thefts, whose chance outcomes the planner branches over exactly, so the trees are wider. A cheaper search under the tables is a search-side question, not a feature question.

Screens on fresh seeds

Four cohorts were registered before the candidate was frozen, all on seeds 256 to 319, which nothing had read, each seed played once per rotation of the four slots (256 games), with the tables in slot 0, the hand-written leaf in slot 1, and an ETA and a fast builder in the other seats. The primary outcome of every cohort is the per-seed difference in wins between slots 0 and 1.

Decision cost differs. The tables are slower to evaluate than the hand-written terms (125 ms against 85 ms per decision at depth 2, 612 ms against 363 ms at depth 3 under the same budget), because every window is recomputed at every leaf and the portfolio tables call the road-length walk. The one-ply tables decide in 15 ms, six times faster than the depth-2 hand-written search they beat.

How the tables play

The build counts of the equal-depth screen say something about the style the tables learned. Per game, the tables seat bought 4.9 development cards to the hand-written leaf's 2.6, built 1.3 cities to its 1.7, and settled and traded at about the same rate (1.7 settlements, 5.1 player trades). The first version of the hand-written leaf over-bought development cards and lost; the tables buy even more and win, which suggests the cards themselves were never the problem, only how the earlier leaf priced everything else around them. Which patterns drive the difference is a question for a follow-up that reads the portfolio tables as the hex table is read above.

The protocol cohort

The registered protocol cohort seated the same tables inside expectimax-v2-plan's exact search against expectimax-v2-plan itself, with the two builders, on 20 deterministic seeds through the authoritative server: 46 wins to 25 of 80, paired contrast +0.263 (95% interval +0.094 to +0.431), every game valid. The protocol report records it and makes ntuple-leaf the protocol-arena baseline.

Second pilot: learning settings and a compact corner window

Nine arms trained for eighteen minutes each under the first pilot's protocol, with the hex plus portfolio layout unless named otherwise: a repeat of the defaults as the control, step size 0.5, λ 0.9 and 0.3, point reward 0.1, 2% random exploration, optimistic initialization at 0.25, a 100,000-game warm-up, and the hex plus corner plus portfolio layout, where a corner window is the three touching hexes and the building state only (303,918 entries).

The control finished at +0.115 where the identical configuration finished at +0.302 in the first pilot. Training on shared tables is not deterministic across runs, and 96 validation games have a wide interval, so two identical runs differ by about 0.2 wins per game; nothing in this pilot separates from that floor. λ 0.3 and 0.9 sit at the top and the corner family in the middle, which says the defaults are not obviously wrong and that the compact intersection window is no better than none. Two four-hour runs, the defaults and λ 0.3, each on sixteen threads with a 192-game validation cohort, follow to ask whether longer training helps and whether the λ reading survives.

Two four-hour runs

The second pilot could not separate any setting from noise, so two runs took the question to a longer budget with a validation cohort twice the size: the default settings and λ 0.3, each on sixteen threads for four hours of wall time, validated every 30,000 games on 48 seeds in every rotation (192 games) against the hand-written leaf in the same depth-2 search with the two builders.

The default run climbed to +0.375 at 150,000 games and then wandered between +0.22 and +0.36; the λ 0.3 run reached +0.29 by 120,000 games and stayed there. On this larger cohort neither run's best point is clearly above the main run's candidate (+0.375 on 128 games at 96,000 games), and the intervals of the two runs overlap at every point after the first. Each best point was frozen and screened against the frozen 96,000-game tables on fresh seeds (340 to 403 for the earlier cohorts; 404 to 467 here), the only comparison that counts.

Neither longer training nor the λ change produced tables measurably better than the ones frozen after 96,000 games. The plateau the first run stopped on was real: with these windows and this learning rule, the tables reach their level in about a hundred thousand games, and the remaining lever is the representation or the search around it, not more of the same games.

Depth 4 and deepening where it matters

A planned turn ends at the leaf before the next round's dice, so a deeper cap keeps more of the resource dynamics inside the tree. Depth 4 with the tables costs a median of 3.5 s and a 90th percentile of 6.5 s per main-phase decision on sampled positions, and the default node budget of 400,000 transitions aborts most depth-4 searches, so a depth-4 configuration needs a larger node budget and a larger time budget. Two search changes make that budget go where decisions are close: a position with a single candidate never deepens past the first depth (29% of the tables' decisions), and with deepen_gap the search stops deepening once the best move leads the runner-up by that many leaf units (on 509 traced decisions the median lead was 5.5 units, half a percentage point of win probability, and a tenth exceeded 38).

The first registered configuration, ntuple-leaf-deep (depth cap 4, 4 s budget, 5,000,000 transitions, gap 20), played the depth-3 baseline configuration on seeds 468 to 531:

The effort counters say why the gain is small: with the time rule still at 30% of the budget the deep seat completed depth 4 on few decisions (mean completed depth 1.98 against 2.18 for the baseline seat, which counts single-candidate positions as depth 1) and mostly spent its budget finishing depth 3 where the 1.5 s seat had to abort (858 ms against 447 ms per decision). The protocol cohort settled the question for this configuration. Through the authoritative server, ntuple-leaf-deep against ntuple-leaf with the two builders on 20 deterministic seeds (372d07bc, run 8f6dd1a7, an 8 s deadline, no timeout moves): 27 wins to 43, a contrast of −0.200 (95% interval −0.357 to −0.043). The registered rule reads an interval below zero as a refutation, so the baseline stays ntuple-leaf and the depth-4 configuration with a 4 s budget and the gap rule is weaker, not stronger, where it counts. Twelve matches ran at once, so both searches shared the machine; the deep seat pays for every depth-4 attempt it has to abort, and the gap rule may be settling decisions at depth 1 that deserve the lookahead. Two cohorts separate those causes. At equal time, with 10 s budgets and deepening allowed while under 60% of them, the cap of 4 reached a mean completed depth of 2.85 against 2.34 for the cap of 3 and spent 3.5 s per decision against 0.7 s, for 60 wins to 52 on 32 seeds (+0.062, −0.105 to +0.230): the fourth planned turn costs five times the time of the third for a gain this cohort cannot see. The gap rule alone, on the baseline configuration against itself without it, won 120 to 103 (+0.066, −0.044 to +0.177) while spending 262 ms per decision against 448 ms: stopping deepening on settled decisions costs no measurable strength and saves 42% of the time. The depth-4 protocol loss is therefore not the rule's doing; it belongs to the depth-4 attempts themselves, which abort under contention and pay for it, and to a cohort of twenty seeds. Depth 4 is not the baseline; the gap rule is a free saving that a future baseline configuration can carry.

More searches at the table

Two follow-up cohorts on seeds 340 to 403 asked whether the margin depends on the two weak builders. With a second hand-written search in place of the fast builder, the tables won 107 of 256 slot-games against 63 and 58 for the two hand-written seats and 28 for ETA (contrast +0.172, 95% interval +0.056 to +0.287). In a symmetric field of two table seats and two hand-written seats, each table seat won 76 against 52 for each hand-written seat (contrast +0.094, +0.004 to +0.183); that lineup repeats under a two-seat shift, so the cohort holds 128 distinct games and its interval is narrower than it should be. The advantage survives more searches at the table and shrinks, as it must, when half the table is the tables.

Limitations

The pilot's and the main run's validation seeds are reused at every point and chose the layout and the candidate, so their margins are optimistic; the fresh seeds of the screens are the evidence, and they are development-tier engine games on one lineup with two fixed builders. Strength against other searches, against humans, or in trade-rich tables with negotiating opponents is not established. The tables were trained by depth-1 self-play for under two hours on one layout; none of the learning settings (step size, lambda, point reward, warm-up) has been varied beyond the pilot's single ablation of the point reward. Self-play against copies of the tables could learn a style that beats itself; the builders at every table are the guard against that.