Higgs Audio

Most open text-to-speech models force an awkward trade. You can have natural prosody, or broad language coverage, or convincing voice cloning, but rarely all three in one package, and almost never with controllable emotion baked in. Higgs Audio, from Boson AI, is an attempt to collapse those choices into a single text-audio foundation model. The current release, Higgs Audio v3, is a 4-billion-parameter conversational TTS model that covers more than 100 languages, performs zero-shot voice cloning from a short reference clip, and lets you steer emotion, style, and prosody with inline instructions rather than rigid presets. It is positioned less as a narrow TTS engine and more as a general speech-generation backbone you can build conversational products on top of.

Architecturally, Higgs Audio treats speech generation as a language-modeling problem. Earlier versions introduced a DualFFN design that extends an LLM backbone to model acoustic tokens with modest extra compute, paired with a unified audio tokenizer trained from scratch to capture both semantic and acoustic detail. The training corpus, which the project calls AudioVerse, is reported by Boson AI to exceed 10 million hours of audio, cleaned and labeled through an automated pipeline using ASR and sound-event models. That scale is what lets it generalize to many voices and languages without per-speaker training. v3 keeps the language-model framing and adds an OpenAI-compatible streaming API, so it slots into existing client code, with self-hosting handled through the SGLang-Omni serving framework.

The natural audience is teams building voice into products: conversational agents, multilingual narration, dubbing pipelines, accessibility tooling, and prototypes that need a believable speaking voice in a language a hosted vendor may not cover well. The zero-shot cloning is useful for giving an assistant a consistent persona from a single reference recording, and inline prosody control suits dialogue where one line should sound curious and the next reassuring. Earlier versions demonstrated multi-speaker dialogue generation and even melodic humming in a cloned voice, hinting at range beyond plain narration. For researchers, the open weights plus released tokenizer and benchmarks make it a credible base for experimentation rather than a closed black box.

The caveats are real and start with licensing. The code is Apache-2.0, but the v3 model weights ship under a Research and Non-Commercial License, so any production or revenue-generating deployment requires a separate commercial agreement with Boson AI. That single clause removes Higgs Audio from the genuinely open commercial category, despite the open code. Operationally, a 4B model served through SGLang-Omni is not lightweight: earlier versions already wanted a GPU with at least 24 GB of memory, and streaming low-latency speech adds further infrastructure demands. The v3 README is also notably thin, steering users toward Boson AI's hosted API and away from the repo itself, which makes self-hosting feel like a secondary path rather than a first-class one.

Against ElevenLabs, the obvious commercial comparison, Higgs Audio offers something different: open code, inspectable weights, and the option to run everything on your own hardware, at the cost of setup effort and a license that blocks commercial self-hosting without a deal. ElevenLabs remains the simpler choice when you want polished output, low latency, and no infrastructure to manage. Higgs Audio is the stronger fit for research, non-commercial products, and teams that specifically need the broadest multilingual coverage with rich prosody control in open weights. It is too early to adopt if you require a fully open commercial self-hosting license, or if you lack the GPU budget and operational appetite to run a 4B model in production yourself.