What I found in the positions

Shivansh Gupta · Full-census snapshot 2026-09-15 at 17:13 UTC

I used Stockfish and Maia to look for strong moves that a model of human play rarely chooses. That gave me a collection of positions to study, and eventually this trainer.

Here are the examples, the selection method, and the checks that changed my interpretation. I don’t yet know whether practicing these positions improves play in new ones; that needs a controlled study.

Three positions to look at

I chose these examples because the lines help explain the moves. Each was checked at Stockfish depth 20, with at least 70 centipawns between its first two choices. They aren’t part of the random audit below. Each also has a second position from a different game to practice on; those pairs haven’t been tested as a measure of learning.

r . . . r . . .
. . . . . p k .
. . . b . . p p
. N n P . . . .
. . P . p . . P
p p P . P . . .
. . . . . . P .
R B . R . . K .

. . . . . . . .
p . . b . . . p
. p . . . k . .
. P p . . p . .
. . P . p . . P
P . K . . . P .
. . . . B P . .
. . . . . . . .

r n b q r . k .
p . p . . p p .
. p . b . n . p
. . . P N . . .
. . P P . p . .
. . N . . . . .
P P . . B . P P
R . B Q . R K .

r . b q . r k .
p p p . . p p p
. b . p . n . .
. P . . . . . .
. . . n P . . .
P . N P . N . .
. . P B . P P P
R . . Q K B . R

r . . . . . k .
. . . R p p . p
p B n p . . p .
. . . . . . . .
. . P . P . n .
. . b . . P . .
P . . . B . P P
. . . . . R K .

. . . . r . k .
. . . . . . p p
. . p R b . . .
p . p . n r . .
N . P . . . . q
. P . . . P P .
P Q . . B . . P
. . . . . R K .

What “machine unique” means here

I use “machine unique” as shorthand for this filter. It doesn’t mean the move is impossible for a person to find. A position qualifies when Stockfish 17.1 at depth 16 gives its chosen move an evaluation at least 100 centipawns above Maia-2's favourite at 2000, and Maia-2 assigns the engine move at most 5% probability at each of 1100, 1400, 1700, and 2000.

This filter selected 5,155 of 123,405 positions (4.18%) across nine mining batches. Sources include Lichess rated-game archives and the Lichess Elite Database, sampled at intervals during games. The corpus balances rating bands, but differs across batches in dates and time controls. It is not a random sample of all chess play. Ratings refer to the source Lichess pools; I haven’t converted them to FIDE ratings.

A sampling mistake I found: all 123,405 positions in this original corpus are Black to move. Sampling every fourth ply before a move always landed on Black’s turn. I’ve fixed the samplers and kept the old results identified as version 1. The new run below includes both sides; the original figures and trainer still describe the earlier collection.

Some of the original games were played by bots. Recovering the source games identified BOT tags in 17,931 original positions, including 886 of the 5,155 selected positions and 38 trainer positions. The historical agreement rates below mix human and bot accounts. They cannot be read as human success rates, especially in the highest rating bands. Source audit

The criterion compares the engine move with one model favourite. It does not require a unique good move. In 2,034 positions (39.5%), the saved runner-up is less than 20 centipawns behind. Missing the exact engine choice does not necessarily mean making a mistake.

The trainer uses a stricter curated subset: when built, its examples also had to retain the engine move at depth 18 with a runner-up gap of at least 70 centipawns. That extra check does not apply to every position in the research corpus.

What was played in the source games

The recorded move matched the exact engine choice in 1,199 selected positions. Another move was recorded in 3,956 cases, including 1,532 with the moving account rated at least 2500. The data calls those nonmatches the “hard core.” Some alternative moves were good too, so these counts alone don’t tell us how well someone played or understood the position.

Separating the sources changes the picture. Above 2800, the recorded move matched the old engine choice in 53/257 (20.6%) selected positions from recovered games without BOT tags, compared with 455/636 (71.5%) in bot-tagged games. This uses recovered ratings and historical answers; it remains a selected archive comparison.

Confidence intervals use 1,000 bootstrap resamples of source games within each band. They do not account for repeated players or systematic differences between datasets. Rating bands exclude their lower bound and include their upper bound, except the first band.

Checking Maia-3 at higher ratings

Maia-3 ranked moves on 77 selected positions and 56 controls at five rating settings. On the selected positions, the engine move's top-five inclusion rose from 28.6% at 1100 to 57.1% at 2600; top-one agreement at 2600 was 2.6%. This smaller ranking experiment did not re-filter the full corpus using probabilities through 2600.

Higher top-five inclusion suggests the model ranks the move more favourably at higher ratings. It does not reveal whether a person considered the move and rejected it.

Many of the moves are quiet

Among positions meeting the same minimum 100-centipawn gap, the low-probability group has more quiet moves and fewer captures or checks. Thirty binary chess features distinguish the low- and high-probability groups at AUC 0.851 (five folds grouped by game; 7,853 positions in the two extreme groups). AUC measures ranking discrimination: 0.5 is chance-level ranking and 1 is perfect ranking.

The groups share a minimum gap, not identical positions or stakes. They were not matched on rating, phase, time control, or batch. The classifier predicts a model-defined label; it does not establish why humans miss moves. The material-exposure proxy is not a measurement of “no compensation.” A quiet move can begin a tactical combination.

Eight groups for practice

I grouped the first 1,745 positions using their layer-10 representations in Leela Chess Zero, then assigned the remaining 3,410 to the nearest group centre. That covers all 5,155 selected positions. I kept eight groups as a manageable way to organise practice, not because the analysis had discovered eight new chess concepts.

Checking the results

Some answers change at greater depth

I reanalysed eight positions per rating band with Stockfish 17.1, using a seeded random sample from 48 different games. All 48 reached depth 20 in every search. The original move remained first in 35/48.

Of 44 positions without mate scores, the original move still beat Maia-2's 2000 favourite by at least 100 centipawns in 42/44. In the same subset, 12 actual game moves matched the original answer; 14 were within 20 centipawns of the best score found by the audited searches.

The sample is small and stratified. Searches use one thread, 128 MB hash, a cleared hash for each search, and a three-second cap. Mate cases are excluded only from centipawn comparisons. Restricted searches can disagree; “best found” is the maximum across the audited candidates, not a proof of optimality. Maia probabilities were not recomputed for changed engine moves, so this is not a fresh mining run.

The selection rate depends on the thresholds

I tried twelve combinations of the probability and evaluation thresholds on the same data. The selected count changes with those choices. The 4.18% figure belongs to the original filter; it isn’t an estimate of how often these positions occur in chess generally.

Did Leela’s representations improve prediction?

The updated comparison fits the embedding's principal-component transformation using only each training fold. On 1,745 selected positions, the full baseline—surface features, engine evaluations, Maia probabilities, and player rating—scores AUC 0.755. Adding 40 embedding components scores 0.732.

I tried two to forty components; none improved on the full baseline. I can’t claim an added predictive benefit from the representations in this comparison. That doesn’t settle what Leela knows: it only tells us what these models could use on this dataset. I also tried the component counts on the same folds, so a firmer comparison needs separate model selection data.

A concept-extraction attempt did not transfer

An earlier experiment fitted sparse directions on Leela rollouts using an adaptation of Schut et al.'s optimisation approach. None of the 30 individually fitted directions generalized under the held-out separation check. Group fitting achieved 1.00 in-group and 0.515 held out. The rollout method, representation and evaluation differed from the AlphaZero study. After that result, I used the representations to organise practice groups. I don’t treat those groups as successfully extracted concepts.

How much to trust the test score

The saved difficulty model reaches AUC 0.845 and Brier score 0.160 across the original corpus, with five folds grouped by game. It predicts exact agreement with the engine in source games. Those data include bot accounts, so this score does not establish performance on human players alone.

The trainer's fixed-rating estimates and test placement use a refit on the full corpus, including trainer positions. Those item curves have not been calibrated against independent participant responses. A website practice session differs from playing under a game clock; the displayed placement is exploratory, not a measured Lichess or FIDE rating.

The 2,000-position pilot

The earlier pilot contains 2,000 positions from 1,662 games, evenly split between White and Black. Half come from slower club games and half from elite archives. I saved the preceding moves for every position, along with available clocks, and separated source games before selection. Later recovery found that 281 Black pilot states already occurred in the historical corpus; the expanded audit records that earlier development exposure.

This time I’m asking how likely the model is to choose any acceptable move, and how much its likely choices lose. A rare first choice isn’t enough if a common alternative is just as good. Stockfish 18 screens the positions; Maia-3 supplies complete move probabilities with and without history, alongside Maia-2 for comparison.

The first screen shortlisted 48 positions under the stated competitive-position, acceptable-move and expected-loss criteria. Deeper checks determine which can become new puzzles. The searches have time limits, and unexamined moves remain explicit in the probability bounds.

Stricter checks retained 13 of 19 reviewed positions, requiring every legal move to reach depths 20 and 24 and the complete acceptable-move set to stay the same. These checks cover new positions and historical teaching candidates.

I also rechecked 64 historical candidates and 64 matched controls. The new screen retained 26/64 candidates and 1/64 controls. This comparison uses FEN-only policies because those saved rows lack verified move history. It is a selected audit, not an estimate of how common these positions are in all games.

At the fixed 2000 setting, adding history changed Maia-3’s first choice in 322/2000 positions. That shows why I’m keeping the comparison. It doesn’t establish how often a person would solve the same position when presented as a puzzle.

The club games come from a bounded prefix of June 2026; the elite sample uses September–November 2025 and mixed time controls. Clocks are saved but are not inputs to the current model. The sample separates games and selected board states, not players or all openings. New methods · Counts, thresholds and hashes

Filling in the White side

I added 123,405 real White-to-move positions. With the original collection and the earlier pilot, the expanded file now has 124,405 White and 124,405 Black observations. The new White sample follows the corrected Black sample's rating-band and game-phase counts within the slower-game and elite groups.

These are counts of saved observations. The Black side contains 886 repeated board states, retained to keep the original records traceable. The White states are distinct. Positions from the same game are also related, so the number of rows is larger than the number of independent examples.

All 248,810 observations have now had the same first-pass Stockfish 18 and Maia-3 analysis. The models receive the current board, without prior moves or clocks, because those details could not be recovered for every original position.

The screen produced 4,353 candidate observations. They cover 4,345 distinct board states. It uses a fixed search budget and leaves unexamined moves explicit. Each candidate needs deeper, all-move checks and chess review before joining the trainer.

For a closer comparison, I also selected 45,326 positions per colour from recovered games without BOT tags. Each group has the same counts by source cohort, rating band, game phase, time-control class and source month, with repeated states removed. This remains a comparison of selected archive games; it does not measure learning or puzzle-solving ability.

The new sample includes 95 additional White turns from cached June games and 143 slower-game positions from the 2025 elite archives. Their sources are labelled; the full collection is not exactly matched by month or time control.

Some source histories remain unresolved. Previously used games retain their development history; assigning a new split does not turn them into fresh test data. Expansion methods · Dataset counts · Screening results

Checking the first 200 candidates

I selected 100 White and 100 Black candidates from 200 different games without BOT tags. Known public training games were excluded. Each legal move was searched separately at depths 20 and 24, giving 14,208 move searches.

116 positions passed the engine-stability checks. The complete set of moves within 20 centipawns of the best stayed the same at both depths, and none of their legal moves had a mate-valued score. Of these, 62 also kept the original candidate criterion: a roughly balanced evaluation, low Maia probability for the acceptable moves, and a large average evaluation loss under Maia's predicted moves at both depths.

These are engine checks. The surviving positions still need explanations and chess review before joining the trainer. A mate in a poor alternative causes rejection under this conservative rule, but does not by itself make a position useless. The sample was balanced by colour and limited to one position per game; it was not matched on every other feature.

Selection and verification methods · Counts, failure reasons and runtime

Checking the full candidate set

The full run covers all 4,345 distinct candidates, including the first 200 whose searches were reused after validation. Every legal move receives the same independent depth-20 and depth-24 searches. Unlike the initial 200-position sample, this includes all source groups and can include several positions from one game. Bot-tagged and previously public sources remain labelled.

The full checks are complete. 2,379 positions met the engine-stability rule: their complete sets of moves within 20 centipawns of the best were unchanged at both depths, and no legal root move had a mate-valued score. Of these, 1,398 also retained the full selection criterion.

The selection criterion requires a best evaluation within ±200 centipawns, at most 10% Maia probability on the acceptable moves, and at least 50 centipawns of expected loss under Maia's predicted moves, with loss capped at 300 centipawns per move. These conditions must hold at both depths and, for Maia, at both the 1700 and 2000 settings.

These counts describe the screened candidate pool. The positions can share source games, and the White and Black groups were not matched on every feature. The results do not measure human solving rates or learning. These remain candidates for chess review; they have not been added to the trainer.

Full-run method · Verified full results, source groups and runtime

Does the preceding play change the model’s choices?

Follow-up reported 29 September 2026; saved research snapshot, not a live progress feed.

I selected 96 development positions from the earlier candidate pool, balanced across colour and game phase. I compared Maia-3 with the current board alone and with the last eight real source-game states. The same pinned model runs both conditions, with clocks disabled. The saved Stockfish depth-20 and depth-24 root evaluations stay fixed and FEN-only: history changes the model input, not the engine’s board-history context.

The paired audit and same-input controls are complete. Adding history retained the full candidate criterion in 83/96 positions; 13 lost it.

Maia’s first choice changed in 14/96 positions at the equal 1700 rating setting and 17/96 at 2000.

This deliberately selected development sample is not a population estimate or an independent test. It measures model sensitivity to context, not human puzzle solving or learning. None of these positions has been approved for the trainer by this audit.

Context audit counts, controls and frozen settings

Checking predictions against fresh source-game moves

Follow-up reported 29 September 2026; saved research snapshot, not a live progress feed.

The first analysis selects 64 positions from a 256-position cohort of August 2026 games: 16 for calibration development, 16 for evaluation and 32 targeted endgames. Selection was frozen before these results. The games come from a bounded chronological archive prefix, so this is not a random sample of the month. They postdate the pinned model’s published weights; that does not establish new players or novel chess patterns.

The original bounded run stopped for review. 2 root searches remain incomplete under its 30-second limit. The baseline below is preserved with those exclusions.

The baseline completed 2,873/2,875 planned root searches: 2,803 exact numeric results and 70 mate-valued results. Completed searches can still exclude a position from calibration.

Same-input model controls passed. Calibration uses the actual mover and opponent ratings, with clocks disabled in both conditions. The event being predicted is whether the recorded source-game move belongs to the complete set of moves within 20 centipawns of the best. A position qualifies only when every legal root has an exact numeric score at both depths 20 and 24, there are no mate scores, and the acceptable set is unchanged.

Brier score is the mean squared probability error; log loss penalises confident wrong predictions. Lower is better for both, but scores only describe each verified subset. Small samples, repeated players and related positions limit the conclusions. No probability adjustment has been fitted.

The targeted endgames receive an approximate depth-14 screen only. They cannot establish the depth-20/24 candidate criterion or qualify a new teaching group. These are source-game move diagnostics, not human puzzle-solving responses or learning outcomes; no new trainer positions are released here.

Fresh-data coverage, calibration scores and frozen settings

A separate check of the capped development searches

Supplement reported 29 September 2026; saved research snapshot, not a live progress feed.

After the original bounded run stopped, I set up a separate supplement for its 2 capped development root searches. Each receives a 120-second limit at depth 24. The original 30-second baseline remains unchanged above. Only the selected development roots receive the larger budget; the evaluation sample and approximate endgame screen are not rerun.

The supplemental searches are complete. Reaching the requested depth does not by itself make either affected position eligible for calibration: numeric scores and an unchanged acceptable set are still required.

Completed root searches: 2/2; incomplete: 0; retry limit reached: 0.

This supplement was chosen after seeing the capped baseline searches, so it is an exploratory budget-sensitivity check, not a fresh independent validation. The calibration event remains the recorded source-game move, not a human puzzle response. It establishes no learning effect or trainer approval.

Separate supplemental protocol, coverage and results

Does practicing these positions help?

I haven’t run a controlled learning study yet. The proposed experiment compares grouped and shuffled practice using the same positions, explanations and practice time. Only their order changes, so the comparison can test the value of seeing related examples together. Both groups will take a baseline test, an immediate test on different positions, and a delayed test about a week later. Test positions must come from different games than training positions.

The primary outcome will credit moves within 20 centipawns of an independently verified best score. Exact engine matches, time taken and performance on positive and boundary cases will be secondary outcomes. The proposed 24-person pilot is meant to check whether the study is workable and how uncertain the results are. It may be too small to establish a learning effect.

Read the revised study protocol. The allocation and material-review tools are built. The final private bank still needs stable answers, meaningful counterexamples and independent chess review before recruitment. Ordinary site use is not automatically enrolled in the study.

Methods and data

Full details: methods and provenance, thresholds, counts and intervals, per-position engine audit, and embedding rerun. Input hashes and seeds are recorded. Large source archives and embedding caches are not bundled with the site.

The inspiration is Schut et al., Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero. Their human study reports improvements on concept-prototype positions in four grandmasters. I adapted the idea using Stockfish, Maia and Leela. I haven’t replicated their learning result.