Block, the company owned by Jack Dorsey, announced the development of a new audio model called Dessa through its research lab. The model features the ability to run locally on consumer devices, delivering vocal tones that closely approximate human expression by incorporating natural pauses, sighs, and emotional expressions without relying on traditional text-to-speech conversion methods.
The Dessa model was trained entirely on human speech data using a single rack of NVIDIA GPUs. It is characterized as a fully-duplex model, allowing it to speak and listen simultaneously, as well as deriving its knowledge directly from conversational data rather than being linked to a large language model dedicated to information retrieval.
The company explained that current audio models often sound unnatural or annoying due to their reliance on prior text processing to generate sound, mostly depending on acoustic tags to force emotional expression. In contrast, Block asserts that the emotional aspect in its model is embedded into its core architecture. Researchers have published audio samples demonstrating the assistant's varying tones and changes according to emotional states, such as expressing fatigue or engaging in role-playing interactions.
What changes practically?
The Dessa model possesses the technical capability to run locally on personal computers with consumer specifications, such as a MacBook M4 Max or higher. This development represents a potential opportunity for companies like Square, owned by Block, which already relies on voice AI for order management. Local execution on devices enables interactive user experiences at a lower operational cost and without the need for complex cloud infrastructures.
Limitations and current availability
To date, the company has not released the Dessa model for public use. The current step is limited to inviting technical security researchers to test the model and evaluate potential risks, particularly those related to the ability of voice agents to mimic humans in ways that could be used to deceive individuals.