NVIDIA has introduced NVIDIA Cosmos 3 Edge, a new 4B omni-model within the Cosmos 3 family, designed to enable on-device robot control. This model, which includes a 2B NVIDIA Nemotron-based reasoner, is pretrained on the same physical-world data as NVIDIA Cosmos 3 Nano and NVIDIA Cosmos 3 Super. Its size allows it to operate directly on NVIDIA Jetson Thor hardware, addressing the challenge of deploying large world models on robots.
Key Points
- Cosmos 3 Edge is a 4B omni-model, featuring a 2B NVIDIA Nemotron-based reasoner.
- It is pretrained on the same physical-world data as NVIDIA Cosmos 3 Nano and NVIDIA Cosmos 3 Super, providing a foundation in object motion and interaction.
- The model is designed to run on-device on NVIDIA Jetson Thor, fitting within its memory for direct inference.
- A post-trained Cosmos 3 Edge manipulation policy can be evaluated in closed-loop simulation.
- The DROID action policy generates each action chunk in approximately 1.53 seconds on an NVIDIA Jetson AGX Thor T5000 at 640x540 resolution and 15 Hz.
- In closed-loop RoboLab tasks, the post-trained policy achieves 22.9% success.
- Post-training for Cosmos 3 Edge requires significant compute, with a validated run using 64 nodes of 4x GB200 for 60,000 iterations, totaling approximately 68 hours or 17,400 GB200-hours.
Context
According to the NVIDIA Developer Blog, robots require policies that can adapt to their sensors, environments, and tasks while operating on onboard computing hardware. World models offer a basis for learning physical interactions, but their size often complicates on-device deployment. Cosmos 3 Edge aims to overcome these constraints by providing a model small enough for on-device execution, reducing reliance on data-center GPUs and minimizing control latency.
Why It Matters
This development is significant for builders working on robotics, as it enables the deployment of sophisticated world models directly onto physical robots. It shifts the paradigm from offloading computation to data centers to performing real-time inference on the device, potentially impacting the design and capabilities of autonomous systems.
What To Do
- Review the NVIDIA Developer Blog tutorial for steps on post-training Cosmos 3 Edge.
- Access the open
cosmos-frameworkrepository for reproducible steps and the released checkpoint on HuggingFace. - Consult the Cosmos 3 Edge model card for a list of supported robot embodiments.
- Note the compute requirements for post-training, specifically the 64 nodes of 4x GB200 for 60,000 iterations, when planning development.
- Explore the DROID post-training reproduction guide and model card for the latest end-to-end instructions.