What a verdict is worth

The hard part of a review is not writing the sentence. It is being right often enough that the sentence is worth reading — and that is measurable, so we measure it.

Take a hand played out by a frozen reference policy. At every decision, substitute a different card into that same position and grade it three ways: the card a competent player chose, a random legal card, and the card the engine ranks last. The positions are identical, so the only thing varying is the card.

A corpus of 10 hands played end to end by the frozen policy, which never moves — so a change to the bot cannot quietly change the test set. dart run eval/run.dart discrimination

What the numbers mean

A decision is scored by what it cost against the best card available in that exact position — in points, over the rest of the hand.

The harsh labels start at 8 on purpose: that is the threshold the table above is measured at. Below it the engine cannot separate a bad card from an unlucky one, so it says nothing rather than guessing at you.

The bot, and what beating it is worth

Strength is measured against a frozen snapshot of the first version, which shares the rules and none of the policy. Self-play was tried and it measures nothing: it reported half a point for a change worth five, because improving the bot improves its opponents in the same breath.

Every row plays the same 1,500 deals with the same bids against the shipping engine, so the only thing that varies is the hero's card play. The weak rows are not competitors — they are the floor the rows above them are measured against. dart run eval/run.dart strength

Above the rules sits a search that samples the hands it cannot see and plays every legal card out. Two seats searching at 160 sampled deals net +7.10 ± 3.43 points a hand over two rule-layer seats, across 300 hands dealt both ways round — a figure about a side, not a seat. Played to 500 instead, the searching side won 95.8% of 24 games: a small sample, printed as one. dart run eval/run.dart searchgain

And it is beatable, which is a brag rather than a hedge, because we can put a number on that too. A side −10.9 points a hand worse than its opponents still takes better than a third of the games off them: the stronger side wins 63.1% over 2,000 games, and no more. A game to 500 is about eleven hands and a noisy unit, which is why a form here is a band rather than a number, and why a single hand gets a verdict rather than a percentage.

A measurement that came back negative

The engine charges 100 points for the tenth bag, and a seat holding nine of them was bidding exactly as it would with none. Letting the bid see the bag column is worth 9.2 points a game ± 7.2 over 3,200 games, and it shipped.

Five rules built on the rest of the match score were measured the same way — 16,000 games each, every game paired against a control on the same deal — and they are all switched off:

The one positive figure moves no games at all, so it is off with the others. The first estimate of the bag column, at 80 games, read 41.2 ± 45.1 and would have been a wrong answer confidently given; it settled only past a thousand games. Read off the tool on 8 and 9 September 2026; it is 16,000 games a variant, so no build re-runs it. dart run tool/score_value.dart

Seventeen ways to misplay a card, each with a name

Not good and bad. A review that cannot say which kind of error you made cannot pick you a drill or describe an opponent. The categories came out of a census of played hands — 29,640 decisions reviewed over 19 seats, 1,726 of them graded a mistake or worse — and not out of a guess about what beginners do.

• Took a trick it did not need23.8%The contract did not need it. It lands as a bag, and the tenth bag costs 100.
• Missed the cut14.7%Void, holding a trump, and let a needed trick go.
• Played a dead card12.1%Nothing it could beat was left. Counting what has gone is the whole fix.
• Led its short suit7.8%Led short when the length was where the tricks were.
• Opened the wrong suit7.5%The lead that hands the trick to the seat behind you, for a reason none of the other names cover. It was a fifth of every error before two narrower leads were cut out of it.
• Broke its own nil1.0%Among the rarest and the dearest of all: 57.0 points on average, because the hundred goes with it.

Six of the seventeen — the five commonest, and the one that costs the most — with each one's share of the graded mistakes. Anything below the mistake band is excluded from the census, because counting it would be counting noise. Read off the tool on 24 September 2026; it reviews every decision it counts and takes about twelve minutes, so no build re-runs it. The corpus is bots, and a bot is wrong in different ways than a person is. dart run tool/mistake_census.dart

Ten rooms, and each one is testing something

The ladder is not a difficulty slider. Every room is a table of players who hold one specific wrong idea, and you move up by learning to punish it. Each idea has a price, measured the same way as everything else.

1. 1Kitchen TableCash what nothing can beat. Holding an ace to stay safe is how you lose it.
2. 2The StoopMost of these hands get called a nil. Lead low and make them take one.
3. 3The BarbershopCount what has gone. Your queen is boss once both aces are down.
4. 4The CookoutBeat it with the smallest card that wins, and keep the rest.
5. 5The YardThe nil is rarer here. Read it out of the bidding before the first lead.
6. 6The BarracksThey still play only their own hand. Take the trick they think is theirs.
7. 7The Union HallOne bad nil a table. Break it and the hand swings two hundred points.
8. 8The RecBid the shape, not only the faces. A fourth spade is a trick.
9. 9The RegionalNothing false left to punish. Only the hands they let slip.
10. 10BostonNothing left but the cards.

Nil means a bad hand — the idea the second room is built on — costs the seat holding it −26.41 points a hand, and the side it sits on −60.60. Those are different questions and the figures are never mixed up. A seat believing nothing false, measured over the same 1,200 deals, reads +1.47 ± 1.67 — the zero this scale hangs from. dart run eval/run.dart misconceptions

A shuffle you can check yourself

Before a card exists, the app publishes a commitment to the deal. Your device then mixes in a seed of its own, so neither side picked the hand alone. After the hand, the server seed is revealed and anyone can recompute the whole thing.

The formula above is the whole scheme, so a hand record carries everything a check needs: the commitment, both seeds and the deal they produce. Export one from the review screen — it is plain text, and the two lines above are enough to recompute it by hand or by script.

What is not true yet: the specification and the verifier that reads these records are written but not published, so checking a deal today means implementing those two lines rather than running ours. Ask at support@spadecraft.app for the specification in the meantime.