Training a deep network from the beginning spends early compute on layers processing immature representations. Adding depth later offers a different allocation of the same learning effort.
An October 12 study examines progressive growth in masked autoencoders and DINOv3. Its experiments report reduced training FLOPs with improved downstream visual performance, alongside changes in intermediate representations and attention patterns. Read the research paper.
A cheaper schedule can produce a different model
The result is more interesting than skipping early work. Growth changes the sequence in which computation develops, potentially encouraging useful local and global processing at different stages.
That mechanism should remain an interpretation tied to the measured probes. Attention patterns alone do not establish why every downstream gain occurs.
Compare matched compute, training duration and deployment architecture. A model that trains cheaply but needs extra inference work creates a different trade-off from one with an identical final serving footprint.
The study covers selected self-supervised vision families, so its recipe should not be presumed optimal for all modalities. It nevertheless shows that architecture can be a training trajectory rather than a fixed starting object. When layers arrive can affect both the cost of learning and the representations they ultimately contribute.
References
Questions, news tips or corrections? Contact [email protected].
