Block launches Dessa AI voice model trained only on human speech

A departure from text-based AI

Block has unveiled a new AI voice model called Dessa, which represents a significant shift in how machines learn to communicate. While most current voice assistants first convert a person's speech into text before generating a response, Dessa processes and generates audio directly. This speech-to-speech architecture allows the model to maintain a more fluid and natural flow during interactions.

The development team wanted to explore if an AI could acquire conversational abilities through audio alone, mimicking the way human babies learn to talk before they can read. To achieve this, the model was trained from scratch using only recordings of human conversations. It did not rely on any pre-trained models or written text datasets during its development phase.

According to the company, this method allowed the model to learn language using hundreds of times fewer words than conventional text-based models. This suggests that focusing purely on the nuances of spoken language might be a more efficient path toward human-like communication. The project was led by a small research team at Block that previously worked at a Toronto-based startup.

Simultaneous listening and speaking

One of the most distinct features of Dessa is its full duplex architecture. This technical setup enables the model to listen and speak at the same time, rather than waiting for a user to finish a sentence before it begins to process the input. This capability allows the AI to handle natural conversational behaviors such as responding to pauses or being interrupted by the user.

Because it bypasses the text-conversion step, Dessa is better equipped to recognize non-verbal cues that are often lost in transcription. The model can identify elements like laughter, sarcasm, and specific changes in a person's tone. These nuances are critical for creating an experience that feels less like a command-line interface and more like a real-time dialogue.

In demonstrations shared by the project lead, Ragavan Thurairatnam, the model engaged in various activities including guessing games and movie discussions. It even interacted with other models, such as OpenAI’s GPT Live 1. While the demonstrations showed the model adjusting its tone, they also highlighted some current limitations, such as the AI occasionally misidentifying a speaker or shifting topics unexpectedly.

Emotional awareness and safety

Block designed Dessa with a specific focus on emotional awareness, aiming to solve a problem they describe as the AI being "nearly deaf" to human emotion. Traditional systems often fail to recognize signs of frustration or distress because they only see the words being spoken. By interpreting emotional signals directly from the audio, the team hopes to create systems that can respond more appropriately in sensitive situations.

The company highlighted that a person in serious pain might interact with an AI that fails to recognize their distress if the system only processes text. By analyzing the audio directly, Dessa aims to bridge this gap, potentially making AI interactions safer and more helpful for people in high-stress moments. The goal is to move toward a system that understands the "how" of a conversation just as much as the "what."

To ensure the technology is developed responsibly, Block plans to grant early access to a small group of AI safety researchers. These experts will study how models trained exclusively on speech behave and identify potential risks. This step is intended to provide a deeper understanding of the unique behaviors that emerge when an AI learns language through the same sensory channels as humans.

Hardware and infrastructure needs

The technical requirements for training and running Dessa provide some insight into the future of speech-focused AI. The model was trained using a single NVIDIA GB200 NVL72 system, which suggests that developing these types of models requires specific types of computing infrastructure. This may differ from the massive clusters typically used to train large-scale text models.

For the end user, the model is designed to run locally on high-end consumer hardware. Specifically, Block stated that Dessa can run on a MacBook Pro equipped with Apple’s M4 Max chip or better. This local execution means the model does not require cloud infrastructure for inference, which can have implications for user privacy and speed.

By running locally, the model can provide a more immediate response without the latency often associated with sending data to a remote server. However, Block has not yet disclosed the specific model weights or detailed performance benchmarks to the public. They have also not provided a broader release date for the general public beyond the initial safety research phase.