Nvidia launches DreamDojo, AI model for training robots

Nvidia has announced DreamDojo, a new artificial intelligence (AI) system designed to teach robots to interact with the physical world. Utilizing 44 thousand hours of videos of humans, this advancement aims to reduce time and costs in training humanoid robots.

The study, recently published, is the result of the collaboration between institutions such as UC Berkeley and Stanford. Researchers claim that DreamDojo represents the "first robotic world model" with strong generalization capabilities across various objects and environments.

At the core of DreamDojo is a large video dataset named DreamDojo-HV, which contains videos totaling 44 thousand hours. This dataset is considered the largest for pre-training world models, surpassing previous data in scale and diversity.

Two-phase training model

The system operates in two distinct phases. First, DreamDojo "acquires comprehensive physical knowledge from large-scale human datasets." Then, it undergoes "post-training" with continuous robot actions, learning from observation before touching physical objects.

This method helps overcome a critical bottleneck. Traditionally, teaching a robot to manipulate objects requires gathering large amounts of specific data, which is costly and time-consuming. DreamDojo uses existing videos, allowing robots to learn simply by observing.

One of the technical innovations is speed. With a distillation process, researchers achieved "real-time interactions at 10 FPS for more than 1 minute," enabling practical applications such as teleoperation and dynamic planning. The team demonstrated the system on various robotic platforms, including GR-1 and G1, showcasing "realistic executions" in different environments.

Nvidia's investment in robotics

The launch comes at a crucial time for Nvidia's robotic ambitions. In a statement at the World Economic Forum, CEO Jensen Huang remarked that AI robotics represents a "unique opportunity" for regions with strong industrial bases. He predicts that the next decade will be critical for accelerated development in robotics technology.

Financial stakes are extremely high. Huang stated that technology company infrastructure spending could reach $660 billion this year, characterizing the moment as .