Structured Matrices in Neural Networks: A Survey

Dense matrix multiplications dominate the compute cost of modern neural networks. Structured matrices -- block-diagonal, Monarch, Kronecker, low-rank, and sparse -- offer a path to faster, smaller models without sacrificing accuracy.