A slow demonstration can mean an inattentive operator. It can also mean a difficult insertion that requires patience. A dataset filter sees the duration; it needs more context to tell the difference.
Harpreet Sahota’s October 8 guide to LeRobot dataset quality distinguishes inexpensive action and state checks from more costly video analysis. It recommends comparing appropriate groups and avoiding repeated penalties from correlated metrics. Its illustrative examples explain how to investigate a dataset; they are not reported field results from a new robot deployment.
The larger editorial lesson is easy to miss. Cleaning can change what a policy is being asked to learn, even when the cleaning code works exactly as intended.
The filter is also a curriculum
Suppose a collection contains fast, simple pick-and-place episodes and slower cable-routing episodes. A global duration threshold could disproportionately remove cable work. A smoothness score might do something similar if careful repositioning is common in the harder task.
After filtering, average quality metrics would improve. Training might become easier. A test dominated by the simpler task might improve too. Yet the dataset could have become less representative of the work the robot will eventually face.
That is a selection effect, not an argument against quality checks. Frozen frames, broken timestamps and impossible joint readings deserve investigation. The challenge is to keep a signal of technical corruption separate from a signal of task difficulty. Combining both into one ranking can conceal the distinction.
It also matters whether the metrics are independent. Duration, idle fraction and low motion can partly describe the same event. Treating them as three separate votes can make a single characteristic dominate the ranking without anyone explicitly choosing that priority.
Audit the discarded distribution
A useful review would examine not just the worst-scoring episodes but the composition of the discarded set. Which tasks, objects, operators and recovery behaviours disappear? Does the filter remove the rare examples that distinguish two superficially similar situations?
The decisive experiment is downstream and task-specific. Train with and without the disputed subset, then evaluate the capability that subset was meant to teach. A better average loss cannot answer whether the robot still knows how to recover from a snag.
There is a practical reporting improvement here. Alongside “episodes removed,” a curation report could state how the task distribution changed and show representative disagreements between the score and human review. That would make the filtering decision inspectable rather than presenting the retained dataset as self-evidently clean.
For physical AI, this is more than statistical housekeeping. A dataset can look tidier because it has stopped asking the model to do messy work. The uncomfortable episodes may be the ones that explain why a robot is needed in the first place.
Questions, news tips or corrections? Contact [email protected].
