A robot arm and a human hand can move the same object while using completely different control signals. That makes their videos easy to watch together and difficult to train together. NVIDIA-led Hydra-0 research replaces the machine-specific action interface with something visible in either recording: the motion of points across an image.
An August research release with a shared action interface
The team published Hydra-0 on August 18, 2026, with an official NVIDIA Isaac project page. It is generalist world-model and control research, rather than a newly available production robot service. The project page lists code as coming soon.
Its action flow describes trajectories of visible points, with visibility information, in camera coordinates. A human hand, a handheld gripper and a robot arm can therefore provide a comparable kind of conditioning signal. The model need not first interpret every machine’s joint vocabulary to predict what motion will look like.
Physics moves the known body; the model predicts the rest
At deployment, the forward workflow uses a hybrid simulator. Isaac Lab executes candidate commands for the robot. Known geometry and camera calibration project the resulting body motion into image-space flow. The video model receives that flow and an initial observation, then predicts the scene’s response.
The division is deliberate. The robot’s kinematics are known; the surrounding cloth, contact and visual scene may not have a complete conventional simulation asset. The model learns consequences at that boundary instead of being asked to rediscover the robot’s mechanics from every command.
Training footage can take another route into the same interface. A tracker recovers image trajectories, and masks distinguish the acting body from the manipulated object. This route supports recordings without robot descriptions or calibration, although the quality of tracking and segmentation becomes another dependency.
Changing the interface is different from changing the backbone
The technical report tests action flow with Cosmos 2.5 and Wan2.2 backbones. A within-Cosmos comparison changes the conditioning representation while keeping the video-model family fixed. That is the comparison most directly relevant to whether the interface itself helps.
The headline motion-error reductions come from the best configuration against an action-conditioned baseline. They should not all be assigned to the interface alone: the strongest configuration also uses a different backbone and distillation. Several engineering decisions can improve a final system while answering different experimental questions.
The result is a useful reporting rule for world models. Compare a proposed action representation inside one model first; then assess the complete optimized system. Combining those claims would make an attractive end result look like a cleaner causal experiment than it is.
The policy score is a replay test
Hydra-0’s policy evaluation covers five policies, six RoboLab tasks and 300 episodes. The team initializes the simulator from each episode’s first observation and supplies its achieved trajectory as action flow. Replayed success rates correlate with reference rates at Pearson r=0.96; reported mean absolute error is 5.7 percentage points.
The policy is not queried on the generated observations. The model is reconstructing consequences of a trajectory already executed, rather than forecasting how a policy will react to its own imperfect simulated future. That makes the result evidence of outcome-preserving replay, not validation of prospective closed-loop policy selection.
A simulator could preserve the ranking of known trajectories yet mislead a controller when small prediction errors accumulate and change its next command. Testing that feedback loop is a distinct step. The project explicitly leaves closed-loop policy evaluation as future work.
Reversing the interface still requires robot training
In a separate control proof of concept, desired object flow from a human demonstration conditions the model to generate compatible robot motion. A trained action readout maps latent features into executable commands. The project shows a physical bimanual robot bending a flexible pipe.
“No task-specific expert robot demonstrations” does not mean no robot experience. The readout is trained using paired real-world rollouts that include successes and failures. This matters when assessing how much new data another embodiment would need.
The team also reports grasp imprecision and ambiguous contact in generated footage. A shared visual language can connect heterogeneous experience without making every physical detail observable. Hydra-0’s strongest proposition is that the interface can travel between bodies; dependable contact and feedback must still be demonstrated at the receiving robot.
References
Questions, news tips or corrections? Contact [email protected].
