Audit of grokking metrics: early compression estimates are inflated several-fold

In July 2026, researchers published an audit on arXiv of the methods used to measure "grokking", an effect in which a neural network suddenly generalizes after it has stopped improving accuracy on the training set. They show that the popular effective rank metric overstates, by several times, how far a model's representation has compressed if it is measured right at the moment of the grokking transition.

What grokking is and why it is hard to measure

Grokking is a phenomenon in which a neural network trained on tasks such as modular arithmetic first memorizes the training set, then, tens of thousands of training steps later, suddenly generalizes the underlying pattern and starts correctly solving examples it has never seen. Researchers often link grokking to "compression" of the network's internal representation (embedding), and they use the effective rank metric to measure that compression.

The authors of the audit show that measuring effective rank exactly at the moment of the grokking transition, a common practice in the field, gives a distorted picture: embedding compression continues for tens of thousands of steps after the network has already generalized the task.

How badly the estimates are distorted

• On an MLP architecture, effective rank measured at the moment of the grokking transition overstates the final (converged) value by 3-5x
• On a transformer trained to full convergence, the overstatement is smaller: 1.3-1.5x
• On an MLP, such early measurement also "erases" information about which representation cells are actually being compressed
• The lag between the transition to correct answers and compression is on the order of the time spent on grokking itself, at least 10,000 training steps
• Adding LayerNorm to a transformer reduces the share of compression completed by the time of grokking from 0.87 to 0.25

What the architecture check showed

A single-variable ablation experiment, in which only one architectural parameter is changed to isolate its effect, found that LayerNorm determines the size of this lag. Without it, the transformer compresses its representation almost in sync with the grokking transition (0.87); with LayerNorm, most of the compression happens afterward (0.25). A pre-registered control experiment ruled out scale invariance as an alternative explanation for the effect.

The authors also tested a narrower, MLP-specific "depth law" linking the normalization budget to the converged compression value. They found that the law does not carry over to transformers and even changes sign under free weight decay, meaning it is unstable outside the narrow set of conditions for which it was derived.

What this means

The work is not just a new metric but a methodological audit. The authors propose a protocol that separates the moment generalization begins from the moment compression finishes, flags cases of censoring (where the metric hits the measurement ceiling), and excludes "boundary" representation cells that never fully generalize. Their own adversarial test suite caught a false-confidence error in their own code branch, which highlights how easy it is to get a misleadingly confident but wrong result from a superficial grokking measurement. The audit's code and tooling are publicly available.

For interpretability researchers, this means revisiting current practice: if a paper on how a neural network works relies on effective rank measured at a single point in time, right after the model started solving the task correctly, its conclusions about the degree of representation compression are most likely overstated and need rechecking on a converged model. A separate lesson concerns architectural components such as LayerNorm: their presence or absence can radically change not the fact of grokking itself but the dynamics of what happens to the network's representation afterward. That makes it risky to compare results from MLPs and transformers directly, without adjusting for architecture.