The Grokking Phenomenon: When Neural Networks Suddenly Generalize

Train a small network on modular arithmetic long past overfitting, and something unexpected happens: validation accuracy suddenly jumps from chance to near-perfect. This is grokking, and it has changed how we think about generalization.