Parallel Experiments

https://arxiv.org/abs/2503.02113 The core idea: Deep learning does not work because neural nets somehow escape generalization theory. It works because very flexible models can still generalize when they have soft inductive biases — preferences for simple, compressible, structured solutions. Key points: - 🧠 Overparameterization is not automatically a problem. Having more parameters than data points does not necessarily mean the learned function is complex. Parameter count is a bad proxy for the complexity of the actual solution. - 📈 Benign overfitting is not unique to neural nets. Models can perfectly fit training data, even noisy data, while still generalizing on structured data. Similar behavior appears in linear models, Gaussian processes, high-degree polynomials, and other classical model classes. - 🔁 Double descent is not just a modern deep learning anomaly. The pattern where test error falls, rises, then falls again as model size increases also appears outside neural networks. It can be understood through effective dimensionality, compression, and the geometry of learned solutions. - 📦 Compression is central. A huge model can generalize if the solution it finds is simple or compressible. The rough intuition is: expected error ≈ training error + complexity/compressibility penalty. - 📚 Some older theories already help explain this. PAC-Bayes and countable hypothesis bounds are more useful here than raw VC dimension, Rademacher complexity, or parameter counting, because they focus on which solutions are likely/simple rather than just how large the hypothesis space is. - 🎯 The paper’s recommended lens: Don’t only restrict what a model can represent. Instead, allow a very rich hypothesis space, but bias the learner toward simpler solutions that fit the data. - ✨ What is still distinctive about deep learning? Not overparameterization or double descent by themselves, but things like representation learning, in-context learning, broad cross-domain usefulness, and mode connectivity in loss landscapes. My takeaway: Deep learning’s famous generalization puzzles may not require rewriting the textbooks from scratch. They may require reading the right parts of the textbooks more carefully — especially the parts about priors, compression, and soft preferences over solutions.