Mundo world model builds interactive robot training scenes from images

From photos to digital twins

ACE Robotics and the S-Lab at Nanyang Technological University have introduced a new world model called Mundo. This tool is designed to build interactive and explorable environments based on just a few images of a real physical location. It allows for the creation of a digital copy of a specific space where robots can be tested and trained.

The system works by taking between one and 12 context images as input. It can reconstruct a scene and render it from various viewpoints along a chosen camera path. When provided with more images, the model becomes more accurate because it has less to guess about areas that were never photographed.

For larger spaces, Mundo can produce camera movements that last for tens of seconds or even minutes. It achieves this by selecting the most relevant photos for each segment of the path. The system then overlaps these segments to ensure the environment remains consistent as the camera moves through the space.

How the model builds scenes

The underlying technology uses a 3D foundation model to recover camera poses and dense geometry. Simultaneously, a 3D detector draws an initial bounding box for every major object it identifies in the images. Because these detectors can sometimes miss objects or misjudge their sizes, the system includes a vision-language model agent to refine the scene.

This agent edits the scene program by rendering it from the input viewpoints and comparing the result to the original photos. It decides what needs fixing, while numerical optimization handles the specific positioning, size, and orientation of the boxes. A separate review agent then checks the entire scene for consistency without making any further edits to the data.

The final scene program records specific details for every object, including its name, pose, size, mass, and friction. These properties are then fed into a physics engine, currently MuJoCo, which computes the motion, contact, and collisions. To make the simulation look realistic, a video diffusion model renders the scene to match the visual style of the original photos.

Training and rendering details

To achieve realistic video output, the team fine-tuned a MiniMax reference-to-video model using approximately 40,000 paired videos. These videos were processed at a resolution of 640 by 480 to generate 124-frame clips. The system reuses the last 34 frames of each clip to ensure that long videos remain continuous and smooth.

The researchers have demonstrated the model's capabilities through several specific use cases. These include robots navigating a reconstructed hotel floor and opening a washing machine in a laundry room. Other examples show robots performing simpler tasks like putting an apple into a bowl or stacking blocks.

The demos also include recorded trajectories from both head-mounted and wrist-mounted cameras. These views help illustrate how the model handles different perspectives during a task. By providing these diverse viewpoints, the model helps robots learn how to interact with objects from various angles in a simulated space.

Technical limits and performance

While the model is capable, the team has identified several clear limitations for current use. The reconstruction process typically takes between 10 to 20 minutes to complete. Rendering the simulation runs at about 1.5 frames per second when using eight A800 GPUs at 20 inference steps.

The team noted that small objects may sometimes appear to float within the reconstructed environment. Additionally, the model does not currently possess long-term memory for scenes that change significantly over long periods. These constraints are important for developers to consider when planning how to use the tool for complex robotic training.

Despite these limits, the ability to generate a functional environment from a handful of images represents a significant step in automation. The team intends to make the code and weights available by the end of November. This will allow the broader research community to begin experimenting with Mundo for their own robotic applications.