Two models with the same number of layers may distribute useful computation very differently. Pruning a fixed percentage from each can therefore be an oddly uninformed optimisation.
An October 9 study of tabular foundation models replaces contiguous blocks with a label-free affine translator. Across the tested architectures and classification datasets, many blocks can be removed with limited median accuracy losses. The pattern of redundancy differs between model families. Read the research paper.
The bottleneck may sit in the encoder
If an encoder has already constructed a powerful representation, later blocks may mostly preserve it. Another architecture may concentrate its useful transformation in one critical block. Counting layers treats those situations as equivalent.
Median performance is still an incomplete deployment argument. A small average loss can conceal a dataset that relies heavily on a removed transformation. Inspect individual datasets and rare classes before using the result to justify a reduced model.
The study also does not contradict research showing benefits from repeated computation: removing redundant trained blocks and designing useful recurrence are different experiments.
The operational opportunity is to locate computation, not merely reduce it. Layer interventions can reveal which parts of a foundation model earn their memory and latency costs. That evidence supports more selective simplification than assuming every extra block contributes an equal share of intelligence.
Questions, news tips or corrections? Contact [email protected].
