Publishing an arena competition

A competition is a small, versioned JSON exhibit. It names the entrants, preserves the order and outcome of every scheduled game, and points to full Parquet replays. The LLM and Expert comparison is a complete example.

Store the asset in content/assets/ and embed it by name:

<ResearchCompetition asset="llm-arena-expert-competition" />

The underlying React component takes a typed competition prop. Its TypeScript contract, strict runtime parser, and standings calculation live together in design/src/components/research/competition.ts, exported through @settlers/design/research. Web validates the asset before rendering or staging. Unknown fields are rejected, preventing private transcripts or credentials from accidentally becoming part of this metadata exhibit.

Identities and order

Expert entrants include search with depth, milliseconds, nodeBudget and checkpointSha256. LLM entrants do not carry these baseline settings. The provider is provenance metadata and is not used as a player label. Older assets without kind remain readable.

Entrant IDs identify player instances, not model families. Two Kimi entrants can share the same model value while using different IDs and names. Every round maps four distinct entrant IDs to board seats. A larger competition can rotate entrants between games; a player's win rate uses its completed appearances.

Each round has number, gameId, seed, seats, openingOrder, turnOrder, outcome, and replay. Array indices in seats are the engine's zero-based board seats. Both order arrays are permutations of [0, 1, 2, 3], read from the recorded engine state. openingOrder describes the first setup pass; the second pass reverses it. turnOrder is the repeating regular-turn sequence. Never infer turn order from array position, model name, or final rank.

The UI labels seats one through four for readers. It lists each round's players in regular turn order and preserves a distinct opening order when needed.

Run conditions

Conditions apply to every round in the exhibit. Preserve separate competitions when changing the common runtime or these settings.

Outcomes and recordings

outcome is a discriminated union:

type CompetitionOutcome =
  | {
      status: "completed";
      winnerSeat: 0 | 1 | 2 | 3;
      points: [number, number, number, number];
      turns: number;
      durationSeconds: number;
    }
  | { status: "pending" | "running" }
  | { status: "incomplete" | "failed"; reason: string };

Final points use board-seat order. The winner comes from the engine, not from sorting visible scores. A completed winner must meet the victory target; a pending, running, failed, or incomplete game has no fabricated winner or score.

replay is either null or an object containing the finalized Parquet file's sha256 and bytes. Running and pending rounds cannot claim a published replay. A finalized partial recording may accompany an incomplete game. With a replay present, the round links to replayBaseUrl + "/" + gameId. That viewer must serve the recording at /replays/<gameId>.parquet; the metadata does not contain or upload the recording. No local machine path belongs in the asset.

Derived standings

The component calculates wins, completed appearances, win rates, average final points, average finishing place, and 95% Wilson intervals from the rounds. Standings are not duplicated in the asset. The recorded winner ranks first; remaining players rank by revealed final points, averaging tied places. Incomplete games remain visible and do not enter these averages.

Wins determine the leader. Average finish only orders rows with equal wins; equal-win leaders remain tied. A majority requires more than half of all scheduled games. Five wins out of ten is a plurality when every other entrant has fewer wins. Zero completed appearances show no rate or interval, rather than a measured zero percent.