Nothing in this trainer came out of a chess book. It came out of a pipeline that hunts for positions where a strong engine is decisively right and essentially no human plays the move. This page goes through what the pipeline does, what it found, and where it falls short.
Positions are sampled from public Lichess games, both club play and games between 2300+ rated players. All ratings on this page are Lichess ratings, whose scale runs roughly 200 to 400 points above FIDE and reaches past 3000 at the top. A Lichess 2600 is a strong expert, not a super grandmaster, and 2800+ here means the genuine top of the pool. Each is analysed twice: by Stockfish 17.1 at depth 16, and by Maia, a neural network trained to predict what a human of a given rating actually plays rather than what is best.
A position counts as machine unique when the engine's move is decisively better (at least 100 centipawns, a full pawn, above the human favourite) and Maia gives that move under 5% probability at every rating from 1100 to 2600. Out of 123,405 positions analysed, 5,155 qualified. That rate of about 4% has held steady across nine separate batches, both rating populations, and every rating band. The dataset is deliberately balanced: each of the six bands from 1800 to 2800+ contributes 19 to 21 thousand positions, so nothing below is an accident of where the data happened to be thick.
In 3,956 of those the player who was actually at the board also failed to find the move. In 1,532, that player was rated 2500 or above. With ~850 machine unique positions per band, the find rate by strength is cleanly measured:
| Mover's rating (Lichess) | Finds the machine unique move |
|---|---|
| 1800 to 2000 | 6.6% |
| 2000 to 2200 | 13.0% |
| 2200 to 2400 | 15.9% |
| 2400 to 2600 | 17.0% |
| 2600 to 2800 | 27.4% |
| 2800+ | 56.9% |
The rate is nearly flat through 2600, then climbs steeply. Players at the very top find a majority of these moves. So “machine unique” really describes everyone below roughly 2800, and how the rest of us search.
Simulated players agree with the engine more often as they get stronger, on ordinary positions. On machine unique positions the rate at which they play the engine's move stays near zero from 1100 all the way to 2600. But the rate at which the move appears anywhere in their top five climbs from 29% to 57%.
Weak players never see the move. Strong players see it and reject it.
Take two sets of positions with identical stakes. In both, the engine's move is at least 100 centipawns better than the human favourite. In one set humans see it (Maia gives it ≥30%; real players found 60%); in the other they don't (≤5% at every rating; real players found 18%). What separates them is not depth or subtlety in the abstract; it is visible in plain chess descriptors. Thirty yes/no features of the move predict which group a position belongs to at AUC 0.851, where 0.5 is a coin flip and 1.0 is a perfect score. No neural network involved.
| The engine's move… | Invisible | In between | Visible |
|---|---|---|---|
| offers material with no visible payoff | 20.5% | 9.2% | 5.4% |
| is quiet (no capture, no check) | 87.2% | 79.8% | 49.4% |
| retreats two or more ranks | 7.8% | 5.9% | 4.5% |
| heads for the board edge | 38.9% | 30.9% | 26.4% |
| is a capture | 10.9% | 16.3% | 43.4% |
| gives check | 3.9% | 5.5% | 11.3% |
| escapes an attacked square | 10.5% | 17.5% | 26.8% |
| creates an immediate threat | 24.8% | 34.3% | 38.4% |
Every row moves in one steady direction across the three columns. That is what a real gradient looks like, not a cutoff artefact. The reading: the invisible moves are the ones that break the rules of thumb humans use to order their search. Look at forcing moves first, don't hang material, answer threats, play toward the king. The engine has no such habits to break. It also explains why the strong find them: checking wrong looking moves is a big part of what separates a 2800 from a 2400.
Each position is turned into a list of numbers taken from inside Leela Chess Zero (layer 10). That list, called an embedding, is the network's own internal description of the position, and those descriptions were clustered into eight groups. The groups were built on the first 1,745 qualifying positions; the 3,410 found by later batches got the same treatment and each joined the group whose centre it sat closest to. That assignment rule reproduces the original labels on the original positions 98.8% of the time. Every group now spans all 5,155 machine unique positions. To describe each group, its direction is compared against twelve named tactical motifs built from thousands of tagged Lichess puzzles.
The groups have numbers, not names, on purpose. A name would claim the group is something chess vocabulary already covers, and the measurements below say that is mostly false. Known motifs explain about half of what makes these positions machine only (R² = 0.46), and the half they do explain looks like sacrifice without a combination to justify it. The other half is not expressible in known terms at all. One group, Pattern 5, has zero or negative similarity to every named motif.
| Group | Positions | Found over the board | Quiet | Avg error (cp) | Closest named motifs |
|---|---|---|---|---|---|
| Pattern 1 | 924 | 29.9% | 85% | −328 | exposedKing 0.19 · defensiveMove 0.11 · quietMove 0.08 |
| Pattern 2 | 444 | 9.5% | 88% | −533 | sacrifice 0.41 · deflection 0.35 · attraction 0.35 |
| Pattern 3 | 1149 | 23.6% | 89% | −224 | pin 0.27 · clearance 0.25 · attraction 0.19 |
| Pattern 4 | 431 | 22.3% | 89% | −219 | intermezzo 0.20 · clearance 0.14 · pin 0.11 |
| Pattern 5 | 258 | 26.0% | 77% | −440 | exposedKing 0.00 · quietMove −0.05 · defensiveMove −0.07 |
| Pattern 6 | 829 | 24.1% | 93% | −300 | skewer 0.29 · attraction 0.28 · deflection 0.28 |
| Pattern 7 | 861 | 20.3% | 85% | −434 | sacrifice 0.42 · attraction 0.41 · pin 0.39 |
| Pattern 8 | 259 | 27.8% | 82% | −313 | exposedKing 0.53 · pin 0.30 · deflection 0.29 |
“Quiet” means the move is neither a capture nor a check. Across all eight groups, between 68% and 94% of these moves are quiet. The part of chess humans cannot see is not tactics. Tactics are what training drills.
Clustering always returns clusters, whether or not they mean anything. So the grouping was put through three tests it could have failed against random chance. It failed one of them.
The structure exists. The control here is a shuffled copy of the data: each embedding dimension is scrambled independently, which destroys any structure while keeping the scale and spread of the numbers. Against that control the real embeddings separate roughly nine times better, measured by silhouette score, a standard number for how cleanly points split into clusters. But the absolute separation is small. These are soft regions in a continuous space, not sharp categories.
| Groups (k) | Silhouette, real | Silhouette, shuffled null | Margin |
|---|---|---|---|
| 2 | 0.0786 | 0.0228 | +0.0558 |
| 3 | 0.0791 | 0.0181 | +0.0609 |
| 4 | 0.0768 | 0.0126 | +0.0641 |
| 6 | 0.0717 | 0.0094 | +0.0623 |
| 8 | 0.0715 | 0.0078 | +0.0637 |
| 9 | 0.0723 | 0.0069 | +0.0654 |
| 12 | 0.0686 | 0.0057 | +0.0629 |
| 16 | 0.0610 | 0.0049 | +0.0562 |
Eight is not derivable from the data. The margin over the shuffled control is essentially flat from k=3 to k=12: no elbow, no natural number of groups. Resampling 80% of the positions 25 times and asking how consistently each k rebuilds the same groups (adjusted Rand, where 1 means identical groupings), k=6 reproduces itself most reliably at 0.90, better than k=8 at 0.80 and far better than k=4 at 0.51. Eight groups is a teaching decision, not a discovery.
Four other lenses agree it is soft. A density based method that is allowed to answer “no cluster” labels 74 to 83% of positions as noise, with small dense cores that match the main grouping. A likelihood criterion (BIC) keeps improving all the way to k=12, never finding a best point. A hierarchical method at k=8 separates as well as the main grouping but only partially agrees with it on which position goes where (adjusted Rand 0.36). And a sparse coding method, which hunts for recurring building blocks in the embeddings, finds the blocks common in these positions lean toward sacrifice and clearance. Every method that could say “continuum” said continuum.
It does predict something outside itself. At k=8, the groups differ in how often real players found the move by more than random groupings of the same sizes would. The odds of a gap that large arising by luck: p = 0.009. At k=9 the same test fails (p = 0.063). So the grouping carries real information about human difficulty, but it is the one arrangement that happened to clear the bar, which is a weaker claim than it first sounds.
An earlier version of this page claimed the engine's internal representation knows something about human difficulty that ordinary chess features do not. That claim was tested properly and it did not hold. It is left here, corrected, rather than quietly deleted.
The question: given a position, how well can you predict whether the player at the board actually found the move? Scoring is by AUC again (0.5 is a coin flip), with the data split into five folds for cross checking, grouped by game so positions from one game never land on both sides of a train/test split.
The first pass compared Leela's embedding against surface chess features (piece, phase, quiet, capture, check, cost) and the embedding won by +0.062. But that baseline had no access to Maia, a network trained specifically to predict human moves, or to the engine's own evaluation. Against a baseline that has both, the picture reverses.
| What the model sees | AUC |
|---|---|
| Maia's probabilities alone | 0.585 |
| Leela embedding alone | 0.606 |
| Surface chess features alone | 0.588 |
| Engine evaluation alone | 0.624 |
| Surface + engine + Maia | 0.628 |
| Surface + engine + Maia + player rating | 0.755 |
| … and the Leela embedding on top | 0.732 |
Adding the embedding to a fair baseline costs 0.023 AUC. That could have been a small sample problem (forty extra features against 246 positive cases can lose on dimensionality alone) so the embedding was retried at every width from forty components down to two. It got worse at every step, and the best result at any width is +0.008: indistinguishable from adding nothing.
So the embedding is redundant for this purpose. Whatever it knows about human difficulty, Maia and the engine evaluation already knew. This does not touch section 4: the clusters still beat the shuffled null, and the k=8 result still holds. What falls is the stronger claim that the representation is a privileged window into what humans cannot see. It is a reasonable way to group positions for teaching. It is not a source of information nothing else has.
The useful half of that experiment. Trained across all 123,405 mined positions, not just the machine unique ones, a model predicting whether a human plays the engine's best move reaches AUC 0.845. Its Brier score, which measures how far its probabilities land from what actually happened (lower is better), is 0.160 against 0.250 for always guessing the average. It is close to honest about its own uncertainty, drifting slightly overconfident in the middle of the range.
| Predicted find rate | Actual find rate |
|---|---|
| 5.7% | 5.3% |
| 13.7% | 13.5% |
| 21.8% | 22.4% |
| 30.8% | 31.5% |
| 40.5% | 40.5% |
| 50.5% | 50.2% |
| 61.9% | 61.6% |
| 74.1% | 73.4% |
| 86.8% | 86.9% |
| 97.6% | 98.0% |
An earlier version of this section claimed the model underestimates machine unique positions. With the repaired game splits (section 6½ below) that gap disappears: the model predicts a 22.3% find rate on them against an actual 23.3%. The earlier gap came from a broken evaluation, not from the positions.
The evaluation had a bug, now fixed. Every elite game carried the same placeholder id, so splits that promised “positions from one game never land on both sides of train and test” were actually treating all 29,741 elite positions as one giant game, with a single fold owning all of them. Real ids were rebuilt by replaying the source archive. With sound folds, the scores stopped swinging between folds (the variance fell to a quarter) and the calibration drift this page previously reported vanished. Numbers on this page reflect the repair.
More data no longer helps, except at the top. Holding out 24,779 positions and training on growing subsets: 2,011 → 0.819, 5,001 → 0.830, 10,006 → 0.840, 20,007 → 0.842, 30,001 → 0.843, 98,626 → 0.844. The curve goes flat past twenty thousand. But the flatness only holds for club positions: splitting the same held out set by the rating of the player at the board, a model trained with the newest 2800+ batch beats one trained without it by +0.0084 AUC at 2800+ and only −0.0000 below 2000. The new data helped exactly where it was aimed, and nowhere else.
Every position in this trainer now carries a prediction from that model: how often a 1900 rated player would find the move. It is shown with the feedback after you answer.
The approach follows Schut et al., PNAS 2025, who mined patterns from AlphaZero that do not exist in human play and taught them to four grandmasters using only example positions and no explanations, because none were possible. All four improved. This project reproduces the idea on open models and turns the result into something anyone can train on.
Stockfish 17.1 · Maia 2 and Maia 3 (CSSLab, Toronto) · Leela Chess Zero · Lichess open database. Every analysis ran on one laptop.