Writing a long training example is not the same as writing a verifiable one. If a model loses track of evidence while reading a document, asking that same model to invent a long document and then derive the correct answer can reproduce the weakness inside its supervision. Long-context training needs a way to preserve the connection between the question, the evidence and the label before reinforcing the model for an answer.
Tsinghua University researchers Zhennan Wan and Jianfei Chen propose LongZero in an October 21 preprint. The method first synthesizes a compact question, supporting evidence and answer, then expands the evidence into longer contexts. It combines reasoning and retrieval tasks for reinforcement learning. The paper evaluates two Qwen-based models against synthetic-data baselines on five long-context benchmarks and reports gains under its experimental setup. Read the primary source account.
Reversing the order changes what needs checking
Fixing the answer before expansion narrows the problem, but does not eliminate validation. The longer text must still express the intended evidence without changing the answer or introducing a conflicting interpretation. For a team reproducing the method, the useful artifact is therefore the complete chain from compact specification to expanded context and final label. Keeping only the final training example would discard information needed to diagnose a broken transformation.
The phrase zero data describes the absence of external task data in this additional learning process. The starting models already contain knowledge and capabilities acquired through earlier training. Readers should not interpret the work as training a language model from nothing. The research question is whether a pretrained policy can create useful new exercises for itself, rather than whether data ceases to matter to model development.
Difficulty is a selection decision
The authors retain reasoning questions on which repeated answers include both successes and failures. That creates a learning signal for group-relative policy optimization. A question that every attempt already solves may offer little immediate contrast, while one that no attempt solves can provide a different optimization problem. The filtering rule makes the training distribution dependent on the current model’s behavior, so it belongs in the method’s reproducibility record.
This dependence also raises a practical question about diversity. Tasks that are easy to generate and verify may occupy more of the training mix than tasks that matter to an external user. A model can improve within a self-written task distribution without solving every long-document failure encountered in practice. Held-out evaluations help test that boundary, while further application-specific checks would still be needed before adopting the recipe for a particular workflow.
Long contexts do not all exercise the same skill
The experiments distinguish multi-document reasoning from retrieval-style tasks. Finding an earlier response, combining evidence across documents and answering a question about a supplied source impose different demands. A useful comparison should show which kind of work improves instead of presenting context length as a single capability. More tokens make an input longer, but do not automatically make the relevant reasoning chain deeper or more representative.
The paper evaluates Qwen3-4B-Thinking-2507 and Qwen3-8B, with the latter using context-extension settings for evaluation. That makes the model and evaluation configuration part of the claim. Results should not be transferred without qualification to a larger checkpoint, another architecture or a different effective context window. Matching training and inference settings is necessary to determine whether an observed difference comes from the proposed synthesis method.
The cost of generating and rejecting tasks belongs in the training account too. A recipe may reduce dependence on externally curated examples while still spending substantial computation on proposals that never become useful training items. Reporting accepted-task yield alongside policy-training cost would make the comparison more complete. It would also show whether the method remains practical as the model improves and the filter admits a different fraction of its own exercises. Zero external task data does not imply zero preparation cost or unlimited useful supervision.
The immediate contribution is a testable alternative to building every long-context exercise from an external corpus or a stronger teacher. Its potential value is a more controllable connection between generated evidence and verifiable rewards. The longer-term question is whether repeated self-generation continues to supply diverse, challenging exercises as the policy changes. Independent reproductions that preserve generation failures, filtering costs and benchmark-specific outcomes would make that prospect more concrete than a broad claim of autonomous self-improvement.
Source organizations: Tsinghua University.
References
Questions, news tips or corrections? Contact [email protected].


