A pretraining pipeline cannot select a document it never fetched. That sounds obvious, yet data curation is usually discussed as filtering a collection already on disk. The crawler’s allocation determines which sources enter that collection, and a poor allocation can spend most of the budget on pages the downstream filter discards.
The host-aware crawling study moves selection upstream. It asks whether a few observations from a host can guide how deeply to crawl it. The idea is not to replace document-level quality checks, but to use limited fetches where they are more likely to expose material those checks will retain.
A host can provide an early quality signal
An October 9 study finds substantial variation in evaluated quality signals between hosts. Sampling a few pages and allocating more fetches to promising hosts collects more high-scoring pages per fetch than the comparison allocation strategies in its simulations. Read the research paper.
The authors combine document-classifier scores, web-graph centrality and Common Crawl’s per-host coverage. Roughly half the variation in evaluated quality scores occurs between hosts. In the examined crawls, a small sample from a host also helps estimate the score of its remaining pages.
This creates an opportunity for sequential allocation. Fetch a few pages, update the estimate and decide whether further depth is worthwhile. It is a different decision from deciding which host to visit first. The paper finds that centrality-based host ordering has limited benefit and can perform worse than random ordering for some quality classifiers.
A quality classifier also defines what the crawler overlooks
The study uses several quality signals, and centrality-based allocation does not behave identically for all of them. This is a reminder that “high quality” is operationalised by the chosen classifier. A crawl can become more efficient at one score while missing pages another policy would value.
The between-host result creates leverage because a small sample can influence many future fetches. The same leverage amplifies a biased early estimate. A mixed host may contain several languages or content types, so its first few pages need not represent every valuable region. An operational policy could retain uncertainty in the estimate rather than treating the sample mean as a final judgement. This is a suggested implication, not an algorithmic result claimed by the paper. The broader point is to make the selection proxy visible before it governs the part of the web a training pipeline ever gets to inspect.
High-scoring pages are still a proxy
In simulations, sample-informed depth allocation gathers several times more high-scoring pages per fetch than uniform-depth or centrality-based allocation. The authors estimate that tens of millions of such pages remain at known but uncrawled URLs in each examined crawl, concentrated on a small share of hosts.
These are opportunities defined by evaluated quality signals. They are not a direct measurement of how much an eventual language model would improve. The classifier may favour particular languages, formats or styles, and a host may contain valuable minority material that a small initial sample misses. Efficient harvesting and broad representation can diverge.
Exploration protects the crawler from its own success
A policy that repeatedly deepens already promising hosts may become efficient at collecting more of the same. An explicit exploration allowance for unfamiliar hosts could preserve the chance to discover sources the current classifier and sampling policy underappreciate. That is an operational implication, not a reported optimum from the paper.
The broader lesson is that the data budget begins before preprocessing. Fetch allocation, filtering and training utility should be evaluated as connected stages. Improving the first can reduce wasted work, but the final objective remains a useful and appropriately representative training collection, rather than the largest possible pile of high classifier scores.
Questions, news tips or corrections? Contact [email protected].
