Hibiki-Zero
Hibiki-Zero is a real-time speech-to-speech translation model developed by Kyutai, an open-science AI lab based in Paris.
Publisher review
Hibiki-Zero is a real-time speech-to-speech translation model developed by Kyutai, an open-science AI lab based in Paris. It translates spoken French, Spanish, Portuguese, and German into English while preserving the original speaker's voice characteristics. The model is designed for developers, researchers, and organizations needing low-latency, on-premise translation without cloud dependency. It runs on a local GPU with just 8 GB of VRAM and is freely available under an open-source license, with inference code and weights released on GitHub and Hugging Face.
Hibiki-Zero is a decoder-only model that processes audio tokens from the Mimi streaming neural audio codec at a constant 12.5 Hz framerate. It uses a multistream architecture (similar to Moshi and Hibiki) to jointly model source speech, target speech, and a text translation stream, producing continuous output audio with timestamped text. The model is trained on sentence-level aligned data (no word-level alignments needed) and refined via a GRPO-based reinforcement learning technique that automatically optimizes the latency-quality trade-off. It can be adapted to a new input language with less than 1000 hours of speech data. A distilled 1.7B-parameter version, Hibiki-M, targets real-time smartphone performance.
Kyutai positions Hibiki-Zero as a direct open-source alternative to proprietary real-time translation services like Google Translate's speech mode and Microsoft Translator. Unlike those cloud-dependent systems, Hibiki-Zero runs entirely on local hardware, eliminating per-query costs and data privacy concerns. Its multistream architecture and RL-based latency optimization differentiate it from cascaded STT→MT→TTS pipelines, which typically add 2-5 seconds of delay. However, it currently lacks the language breadth of Google Translate (which supports 130+ languages) and does not offer output in languages other than English.
The primary trade-off is language coverage: Hibiki-Zero only translates four source languages (French, Spanish, Portuguese, German) to English, with no English-to-other or other-to-other support. Performance varies by language — German's verb-final structure can increase latency compared to Romance languages. The model requires a GPU with at least 8 GB VRAM for real-time operation; CPU inference is not feasible. As a research release, it lacks polished user interfaces, production SLAs, and commercial support, making it best suited for technical users who can integrate the codebase themselves.
How it works
-
Real-time speech-to-speech translation
Translates spoken French, Spanish, Portuguese, and German to English with low latency, producing continuous output audio at 12.5 Hz.
-
Voice transfer preservation
Maintains the original speaker's voice characteristics (timbre, prosody) in the translated audio output.
-
Simultaneous speech and text output
Generates both a continuous audio stream and timestamped text translations from the same model.
-
Open-source model and weights
Inference code and 3B-parameter weights are freely available on GitHub and Hugging Face under an open-source license.
-
Local GPU deployment
Runs on a single GPU with only 8 GB of VRAM, enabling on-premise translation without cloud dependency.
-
Adaptable to new languages
Can be fine-tuned to a new input language using less than 1000 hours of speech data.
-
Reinforcement learning optimization
Uses GRPO-based RL to automatically balance translation latency and quality without word-level alignments.
Strengths and trade-offs
Strengths
- Runs on a local GPU with just 8 GB of VRAM, making it accessible for individual developers and small teams without cloud infrastructure.
- Preserves the original speaker's voice characteristics during translation, enabling natural-sounding multilingual conversations.
- Achieves real-time performance with a constant 12.5 Hz framerate, producing continuous output audio streams without noticeable pauses.
- Open-source release includes both inference code and 3B-parameter weights, allowing full customization and auditability.
Trade-offs
- Only supports translation from four source languages (French, Spanish, Portuguese, German) to English, with no English-to-other or other-to-other direction.
- Requires a GPU with at least 8 GB of VRAM for real-time operation; CPU inference is not feasible, limiting deployment on older hardware.
- Performance varies by language — German's verb-final structure can increase latency compared to Romance languages like French or Spanish.
- As a research release, it lacks polished user interfaces, production SLAs, and commercial support, requiring technical expertise to integrate.
Pricing context
Free and open-source — no licensing fees, usage caps, or cloud charges. Users only pay for their own GPU hardware and electricity.
Getting started with Hibiki-Zero
-
Clone the repository
Clone the Hibiki-Zero repository from GitHub to your local machine. Ensure you have Git installed and run git clone <repository-url> to download the inference code and model weights.
-
Set up Python environment
Create a Python virtual environment and install the required dependencies listed in the repository's requirements.txt file. Use pip install -r requirements.txt to ensure all libraries are available.
-
Download model weights
Download the 3B-parameter model weights from Hugging Face using the provided script or command. Place the weights in the designated directory within the cloned repository to enable model loading.
-
Run inference on audio
Execute the inference script with a source audio file in French, Spanish, Portuguese, or German. Specify the input file path and output destination to generate translated English audio with voice preservation.
-
Test with microphone input
Modify the inference script to accept real-time microphone input instead of a file. Use a compatible audio library to capture speech and stream it through the model for live translation output.
Frequently Asked Questions
What is Hibiki-Zero and how does it work?
Hibiki-Zero is a real-time speech-to-speech translation model from Kyutai that translates spoken French, Spanish, Portuguese, and German into English while preserving the speaker's voice. It uses a decoder-only architecture with a multistream approach to process audio at 12.5 Hz.
Can I run Hibiki-Zero on my local computer without the cloud?
Yes, Hibiki-Zero runs entirely on a local GPU with just 8 GB of VRAM, eliminating cloud dependency. This on-premise setup avoids per-query costs and data privacy concerns, but CPU inference is not feasible for real-time operation.
What languages does Hibiki-Zero support for translation?
Hibiki-Zero translates four source languages—French, Spanish, Portuguese, and German—into English only. It does not support English-to-other or other-to-other translation directions, and its language breadth is limited compared to services like Google Translate.
How does Hibiki-Zero preserve the original speaker's voice?
Hibiki-Zero maintains the original speaker's voice characteristics, including timbre and prosody, in the translated audio output. This is achieved through its multistream architecture that jointly models source speech, target speech, and a text translation stream.
What are the hardware requirements for using Hibiki-Zero?
Hibiki-Zero requires a GPU with at least 8 GB of VRAM for real-time operation. CPU inference is not feasible, limiting deployment on older hardware. The model is designed for local deployment, so users only pay for their own GPU and electricity.
Is Hibiki-Zero a good alternative to Google Translate's speech mode?
Hibiki-Zero is a direct open-source alternative to proprietary services like Google Translate's speech mode, offering local deployment and voice preservation. However, it lacks Google's language breadth (130+ languages) and does not provide output in languages other than English.
Alternatives
How Hibiki-Zero compares
Direct head-to-head against 3 competitors. Picked by 7wData.
Hibiki-Zero
- Pricing
- Free and open-source — no licensing fees, usage caps, or cloud charges. Users only pay for their own GPU hardware and electricity.
- Target
- Hibiki-Zero is a real-time speech-to-speech translation model developed by Kyutai, an open-science AI lab based in Paris.
- Strength
- Runs on a local GPU with just 8 GB of VRAM, making it accessible for individual developers and small teams without cloud infrastructure.
- Watch for
- Only supports translation from four source languages (French, Spanish, Portuguese, German) to English, with no English-to-other or other-to-other direction.
Google Translate API
- Pricing
- $20/million chars
- Target
- Enterprise cloud translation
- Deployment
- Cloud API
- Strength
- 100+ language pairs
- Watch for
- No voice transfer capability
Meta SeamlessM4T
- Pricing
- Open-source
- Target
- Multimodal translation research
- Deployment
- Self-hosted
- Strength
- Speech/text joint modeling
- Watch for
- No reinforcement learning for latency
Moshi
- Pricing
- Open-source
- Target
- Real-time speech dialogue
- Deployment
- On-device possible
- Strength
- Same multistream architecture
- Watch for
- Not optimized for translation
User reviews
No user reviews yet. Be the first to write one.
Sources
Reporting on this tool draws on these publicly available sources.