Why Can Random Interpolators Learn Without Errors?

Imagine a teacher that creates perfect data and a student that must learn from it. In this setting, scientists found that if the student uses an "interpolator"—a model that fits every training point exactly—the generalization error can drop to zero as soon as the number of examples exceeds a certain threshold.

The discovery is striking because it does not rely on optimization tricks like Stochastic Gradient Descent, but stems from the geometric structure of the model's parameters. By applying algebraic geometry, researchers mapped the solution space and proved that beyond a certain sample count, random solutions all lie in a region where the model predicts perfectly on unseen data.

This helps explain why huge neural networks, with millions of parameters, still generalize well despite appearing over‑parameterized. In practice, the more data you feed, the more the "solution game" shrinks to error‑free configurations, regardless of how you pick them. A key lesson for AI students: sometimes the magic lies in the shape of the parameter space, not just the learning algorithm.

In the teacher‑student setting, the teacher model generates an output set y = f_T(x) for inputs x drawn from a distribution P(x). The student employs a function class f_S(·;θ) parametrized by θ∈ℝ^p. An interpolator is a choice of θ such that f_S(x_i;θ)=y_i for all N training samples. The proof starts from the fact that the set of interpolators forms an algebraic variety V⊂ℝ^p defined by N polynomial (or rational) equations arising from the interpolation conditions. Using Krull's dimension theorem, one shows that the dimension of V drops by at least N as N grows, until it reaches zero: V becomes a finite set of isolated points. When V has dimension zero, every point in V satisfies the interpolation equations and, by construction, also solves the broader learning problem because the teacher function lies in the algebraic closure of the student class. Consequently, for N≥N_* (where N_* is the smallest integer making dim(V)=0), any randomly chosen interpolator is almost surely identical to f_T on the entire support of P(x), guaranteeing exactly zero generalization error. The threshold N_* depends on the algebraic complexity of the student class, typically proportional to the total degree of the polynomials describing f_S. This approach links generalization ability to topological properties of the solution variety, opening new ways to analyze over‑parameterized models without invoking implicit optimizer bias.