Tsinghua and NVIDIA launch StreamEval to benchmark real-time AI video

A new standard for interaction

Tsinghua University, NVIDIA, and several other global institutions have launched StreamEval, a new ranking system for real-time AI video models. While the industry has seen a boom in video generation, the field currently lacks unified definitions and evaluation standards. Many existing tools only check if a single prompt produces a clear, high-quality video clip.

StreamEval addresses a different challenge: continuous interaction. In a real-time system, a model must respond to a stream of user instructions without interrupting the video generation. It must also keep the character's identity, the existing scene, and all previous valid modifications unchanged as time flows. This creates a complex technical hurdle that standard benchmarks often ignore.

The benchmark covers five specific categories: digital human dialogue, virtual human performance, world generation, instruction editing, and reference editing. By testing 38 different models across 300 test cases, the researchers aim to expose the specific shortcomings of current technology. As eu.36kr.com reported, the goal is to see how models handle the most challenging parts of real-time interaction.

How the benchmark works

To simulate real-world use, StreamEval runs models under near-real conditions where instructions arrive in batches. The dataset includes 1,347 online events, such as voice commands, text instructions, and reference images, which are released step-by-step along a timeline. The model is restricted to only seeing the currently released input, meaning it cannot peek at future requirements or the final scoring answers.

To ensure accuracy, the system repeats each test scenario three times using the same model to minimize the impact of random errors on the final results. The evaluation system quantifies four dimensions: task execution, state consistency, visual fidelity, and audio-video coordination. These are broken down into ten specific indicators, such as picture stability, temporal continuity, and character identity matching.

Each indicator is scored from 0 to 100, and the comprehensive score for a task is calculated using equal weights for all applicable indicators. For digital human tasks, systems that support custom images receive an additional 5 points. Because the benchmark is open source, engineers can personally rerun the tests and confirm the scoring logic according to the public documentation.

Key findings on model limits

The initial results show that high image quality and character identity are relatively independent capabilities from real-time control. A model that generates stable-looking videos does not necessarily mean it can reliably execute changing instructions. For example, in digital human dialogue, the Vidu S2-Avatar model achieved a high score for stability and audio-video synchronization but struggled significantly with specific instruction following.

The world generation task also revealed clear differences in how models handle spatial continuity and camera movement. Some models excelled at image quality and stability but received very low scores for continuity and instruction following. These findings suggest that visual perception, instruction response, and long-time continuity are distinct skills that need to be judged separately rather than as a single unified score.

Real-time editing proved to be even more difficult, particularly when using reference images. The benchmark showed that models often struggle to change a character's appearance without destroying the background. For a model to receive a full score, it must successfully manage reference image switching, modification range control, and undo recovery all within the same continuous generation process.

Impact on the AI industry

StreamEval is intended to be a public evaluation framework for the entire industry rather than just a leaderboard. By involving top academic and industrial institutions like CMU, UC Berkeley, and Princeton, the project establishes a reproducible standard. This allows developers to accurately locate the specific technical shortcomings of their models in a transparent way.

The benchmark provides a verifiable basis for progress in a track where even basic definitions are not yet unified. As the leaderboard grows with more participating models, it will become a primary reference for companies developing world models and digital humans. It moves the conversation away from marketing hype toward measurable performance in complex, multi-step interactions.

By providing open data and scoring rules, the project helps developers identify exactly where their models fail—whether it is maintaining a character's face or following a specific command. This transparency ensures that as the technology evolves, there is a clear and verifiable way to measure who is actually making the most significant progress in real-time video.