Google DeepMind releases Genie 4 with native audio and live scene edits

A world model with sound

Google DeepMind recently unveiled Genie 4, a research preview of a world model capable of generating interactive environments. Unlike previous versions, this model produces native audio, meaning it creates ambient sounds and character dialogue simultaneously with the visuals. For example, a beach scene will include the sound of crashing waves while characters react to prompts in real time.

The model is designed to handle complex physics, including collisions and aerodynamics. It also features a significant leap in how it simulates people, making them appear more realistic within the generated space. These improvements allow the simulation to feel more cohesive as users interact with the environment.

To ensure responsible use, Google is applying SynthID to every clip produced by the model. This is an imperceptible watermark that identifies the content as AI-generated. While humans cannot see the mark, detection tools can use it to verify the source of the video or audio.

How the hybrid system works

Genie 4 operates using a hybrid architecture that combines different AI models to manage different tasks. Gemini 3.8 Flash acts as the brain of the operation, breaking down user goals into specific steps and reviewing each frame to ensure the simulation stays on track. While Gemini plans the actions, Genie 4 handles the heavy lifting of rendering the actual world.

When a user provides a prompt or intervenes in a scene, Gemini 3.8 Flash rewrites the plan instantly to accommodate the change. This allows for real-time edits where a user can swing the camera, shift the weather, or drop a new person into a running simulation. The system ensures that these edits fit naturally with what is already happening on screen.

In robotics experiments, this loop has been used to rehearse tasks. A model called Gemini 4 Argon can split a complex task into steps and feed them to Genie 4 as text captions. This allows teams to simulate robot movements in a virtual environment before attempting them in the physical world.

Tools for filmmakers and labs

Trusted testers are already using Genie 4 for pre-visualization and film prototyping. Filmmakers can step into a live scene to run multiple takes, adjusting lighting, camera angles, and dialogue on the fly. This serves as a draft mode that allows for rapid ideation before moving into full production.

Photojournalists are also exploring the technology to see if archival photographs can be turned into explorable simulations. By turning a static image into a 3D-like space, they can potentially create more immersive ways to view historical locations. Diego Rivas, a Google DeepMind engineer, noted that real-time generation is most useful for these types of creative tasks.

While the current model is experimental, there is potential for it to evolve into a pipeline for finished assets. Rivas suggested that Gemini Omni could eventually turn these prototypes into final production elements, though that is a long-term goal. For now, the focus remains on storyboarding and high-level concept development.

Current limitations and competition

Despite its capabilities, Google DeepMind acknowledges that Genie 4 still has several technical gaps. Visual details and object permanence can drift during long sessions or after complex edits. Additionally, the model struggles with reliable text inside generated worlds and faces ongoing challenges with spatial accuracy and long-term consistency.

The model also enters a crowded market with several other companies developing world models. Runway recently released GWM Worlds 2 with audio, and Amazon is in talks to acquire Decart. Other companies like AMD and Rivian are also making moves to bring simulation technology in-house for self-driving and other applications.

Google is currently monitoring how paying subscribers will push the model's limits compared to the initial tester group. They are specifically watching for 'object drift,' which is when items in the scene begin to warp or disappear over time. These findings will help determine how long the scene limits can be extended before wider API access is granted.