NVIDIA Cosmos 3 Edge: Physical AI on Jetson

Résumé

Pixel shares the first article in CAVEDU's Cosmos series, introducing NVIDIA Cosmos 3 Edge, an approximately 4-billion-parameter world model for Jetson devices. It explores multimodal capabilities, deployment options, reported inference performance, and how robots can progress from visual input to action—important context for building responsive on-device Physical AI.

DEV Community

Cosmos 3 Edge, Part 1: Understanding Physical AI on NVIDIA Jetson

NVIDIA Cosmos 3 Edge is an edge-optimized world model with approximately 4 billion parameters, combining multimodal understanding, image and video generation, and robot-action capabilities. This first article in CAVEDU's Cosmos series introduces the model, its Jetson deployment options, reported inference performance, and the path from visual input to decisions. Read the series Model capabilities and a first look at Physical AI Workflows, action reasoning, and action chunks SO-101 sim-to-real and on-device robot control These links open the original CAVEDU articles in Traditional Chinese. From chat windows to Physical AI AI is moving beyond conversations on a screen toward systems that perceive, understand, and interact with the physical world. Autonomous mobile robots, robot arms, and embodied agents need an onboard computer that can process observations with low latency. NVIDIA Jetson provides a platform for this work, from Orin devices to Jetson Thor. The original article also discusses NVIDIA's announced Orin Nano 2 roadmap for 2027. See the announcement referenced in the original for its context rather than treating a future product as currently available. Watch the introductory video. Why put Physical AI on Jetson? A robot working in the real world cannot always wait for a cloud round trip. Manipulating a moving object or responding to a pedestrian makes latency and connection reliability important system requirements. Jetson brings inference closer to the sensors and actuators. Its software stack supports deploying open models on the device, while different hardware tiers serve different memory and compute requirements. The source article references JetPack 7.2 and 7.2.1; select software and containers that actually match your target module and installed release. Local inference alone does not establish a safety guarantee. The model, controller, sensors, and timing must work together. What does Cosmos 3 Edge add? Cosmos is NVIDIA's world foundation model platform for understanding, simulating, and interacting with physical environments. Its applications include robotics, autonomous driving, and intelligent spaces. Rather than only naming objects in an image, the goal is to model relationships involving time, space, motion, and actions. Cosmos 3 Edge brings these capabilities toward embedded deployments. Architecture and serving options Cosmos 3 Edge is the edge-oriented member of NVIDIA's omnimodal Cosmos 3 family. The source describes a 2.4B multimodal reasoner plus a diffusion-based generation module, totaling approximately 4B parameters. Its supported workflows include text, image, and video understanding; image and video generation; and chunked robot-action trajectories. A single checkpoint supports two serving paths: vLLM: Text, image, and video reasoning through an OpenAI-compatible chat API. This path loads the 2.4B reasoner from the full checkpoint. Jetson AI Lab also describes the separately released Cosmos3-Edge-Reasoner. vLLM-Omni: Generation and robot-action workflows through its video API. Deploy this separately or alongside the reasoner as the workload requires. The full checkpoint and the reasoner-only deployment do not have identical resource requirements. Consult the Jetson AI Lab model guide before selecting a configuration. Main capabilities Multimodal reasoning: Text, image, and video understanding. The source reports vLLM serving on Jetson Thor and AGX Orin, plus a validated Hugging Face Transformers BF16 reasoner configuration on an 8 GB Orin Nano. Robot policies: Chunked action inference using vLLM-Omni, with a DROID-post-trained checkpoint. World generation: Text-to-video, image-to-video, and text-to-image generation through the diffusion component. Forward and inverse dynamics: Action-conditioned world modeling for Physical AI pipelines. Reported reasoning performance The original article reproduces these Jetson AI Lab figures: Module Text (tokens/s) Image (tokens/s) Video (tokens/s) Thor T5...
NVIDIA Cosmos 3 Edge: Physical AI on Jetson
Powered by Readybot.io

Les dernières actus de Pixel

Stop Paying for AI Renders in Test Orders

Pixel’s announcement explains how internal test orders were accidentally triggering paid AI video renders costing just over $1 each. It details a server-side dry-run guard …

Sandblasting Grit Guide: 120 vs 220 vs 320

Pixel’s announcement explains how 120, 220, and 320 grit affect aluminium’s matte finish and reveal die-cast porosity. It also covers how pressure, blast distance, and …

Laravel Semantic Search with OpenAI and Pinecone

Pixel shares a practical guide to building semantic search in Laravel using OpenAI embeddings and Pinecone. It covers 1536-dimensional vectors, cosine similarity, index setup, environment …