Microsoft patents AI to rewrite livestream audio in real time

The mechanics of real-time editing

Microsoft recently filed a patent that describes a system for manipulating live broadcast audio using generative AI. The technology is designed to listen to a broadcast's audio track and identify specific words for replacement. These changes would occur in real time as the content is being streamed to an audience.

The AI would determine which words to swap based on the specific profile of the person watching the stream. Factors such as the viewer's native language, age, and geographic location would influence the final output. This allows the system to tailor the experience to a diverse global audience simultaneously.

Beyond simple personalization, the patent claims the AI could be used to clean up a presentation by removing pauses and filler words. It could also be programmed to censor profanity or replace certain terms based on the results of internet searches. This suggests a high level of control over the narrative presented in a live setting.

The visual side of AI manipulation

html

One of the most notable aspects of the filing is how the system handles the visual component of a broadcast. To ensure the edits do not look jarring, the AI could adjust the lip movements of the speaker in the intervening frames. This would allow the system to hide the fact that the words being spoken have been altered by a computer.

The ability to synchronize visual movements with modified audio is a significant technical step for real-time content manipulation. It ensures that the viewer sees a seamless performance even if the underlying speech has been swapped or edited. This creates a high level of realism that could make it difficult for a viewer to notice any intervention.

While the patent does not specify if the system would notify viewers about these changes, the technical capability exists to do so. Without such notifications, the technology could allow for the creation of AI-edited streams where the original intent of the speaker is modified. This creates a potential gap between what is actually said and what the audience perceives as being said.

The rise of AI video content

Since the initial explosion in popularity of generative AI, the creation and editing of video content have emerged as some of the most popular applications. This trend highlights a significant shift in how people interact with and produce digital media. As the technology matures, these tools are becoming standard for creating high-quality visual experiences.

The current interest in real-time AI editing reflects a broader movement toward dynamic content. While the Microsoft patent focuses on audio and lip-syncing, it sits within a larger ecosystem of AI-driven media. These technologies are moving toward a future where the distinction between live human performance and AI-assisted generation becomes increasingly blurred.

Expanding AI into gaming and code

Microsoft is also exploring how AI can reshape the gaming experience through more ambitious technical claims. One filing from July describes a system that could rewrite a game's code in real time to add new content based on how a player behaves. This suggests a move toward games that can adapt their structure and narrative dynamically to individual user actions.

Sony has filed similar patents aimed at improving game performance and reducing input latency. These systems would use AI to predict which buttons a user will press next and what the upcoming frames should look like. Additionally, Sony's patents explore using AI to manage game difficulty, control time progression, or even fill in for players who temporarily leave online sessions. These developments show a shared industry goal of using AI to change the user experience in real time.