Google DeepMind unveils Genie 4 for real-time interactive 3D worlds

From video to interactive worlds

For the past two years, most people have viewed generative visual AI as a tool for creating non-modifiable, one-way videos. Models like Sora or Veo allow users to input a prompt and wait for a finished clip, but they do not allow for real-time interaction. Google DeepMind is changing this dynamic with the launch of Genie 4.

Genie 4 is a next-generation world model that blurs the line between video and interactive virtual environments. Unlike traditional 3D game engines that require art teams to spend months on modeling and texturing, Genie 4 calculates every frame and sound in real-time. It responds directly to user keyboard inputs and text commands to build 3D spaces on the fly.

The model represents the fourth iteration in a rapid development cycle. It builds on Genie 1, which learned action correlations from video, and Genie 3, which achieved real-time interactive generation. Genie 4 doubles the continuous interaction duration to 120 seconds while adding more realistic physics and native audio.

The brain and the senses

A major breakthrough in Genie 4 is the deep integration with the Gemini family of models. This creates a hybrid system where different AI components handle different tasks. Gemini models act as the 'brain' or director, while Genie 4 acts as the 'senses' that handle physical rendering.

Specifically, Gemini Omni provides the visual common sense and foundational logic for the environment. Gemini 3.8 Flash functions as a real-time guiding agent that breaks down complex tasks into sub-steps and monitors the simulation for success. These models work together to ensure the world behaves logically while Genie 4 handles the visual output.

For more complex experiments, Google is using Gemini 4 Argon to simulate training for the Apptronik Apollo humanoid robot. In this setup, Gemini 4 Argon breaks down tasks into sequential actions, and Genie 4 simulates the physical consequences in real-time. This allows the robot to learn skills, such as playing checkers, in a virtual space before moving to the real world.

Dynamic environments and audio

Users can generate and modify 3D worlds using simple text prompts. For example, a user can describe a man playing a saxophone on a floating platform and then immediately control the character's movement. The system even allows for on-the-go modifications, such as summoning dragons or friendly otter NPCs while exploring a city.

Genie 4 is also the first real-time world model to feature native audio generation. This means the system produces ambient sounds, like waves crashing or leaves rustling, based on the user's actions in the environment. It also supports dynamic NPC dialogue, where virtual characters generate spoken lines and physical reactions based on the environment rather than following a rigid script.

The physics engine within Genie 4 has also seen significant improvements in simulating complex behaviors. Google demonstrated the model handling watermelons shattering upon impact and maple seeds drifting according to aerodynamics. It can even simulate complex crowd avoidance behavior in a busy rural train station, making the interactions feel more natural.

Industrial use and limitations

Beyond entertainment, Genie 4 provides a zero-cost, zero-risk platform for industrial simulation and robotics. Training humanoid robots in the real world is expensive and dangerous, but this virtual platform allows AI agents to conduct millions of trial-and-error runs. These agents can practice in countless parallel virtual worlds before transferring those skills back to physical hardware.

Despite these capabilities, Google DeepMind admits that the technology is still in the early research stage. The model currently faces a '120-second curse' where inconsistencies can occur as simulations run longer. These issues include drifting object details, shifting positions, and the model occasionally forgetting early scene settings or struggling with text legibility.

The technology is currently open to invited research testers in fields like film pre-visualization, gaming, and engineering. In the future, Google plans to make it available to Gemini AI Ultra subscribers via Project Genie. It will also be gradually integrated into the Gemini API and Google AI Studio for broader commercial use.