Experiments in Fine-Tuning Chatterbox TTS on NaijaVoices for WaZoBia

Resemble AI’s Chatterbox is a state-of-the-art open-source text-to-speech architecture (~1.2B parameters in its T3 configuration) that bridges three distinct deep learning paradigms:

• An Autoregressive Transformer modeling discrete acoustic tokens (Encodec).
• A Conditional Flow-Matching Model that refines latent mel representations.
• A diffusion-inspired vocoder that converts tokens into high-fidelity waveform audio conditioned on a target speaker prompt.

While Chatterbox excels at zero-shot voice cloning and spontaneous prosody in English and European languages, it has no prior conditioning for West African phonetic systems. Out of the box on Nigeria’s three primary indigenous languages, Yoruba (yo), Igbo (ig), and Hausa (ha) (WaZoBia), it produces unintelligible output — the tokenizer and acoustic priors simply have no reference for these scripts or phoneme sets.

In this experiment, I wanted to explore whether Chatterbox could be conditioned on authentic, in-the-wild Nigerian speech to synthesize WaZoBia while preserving zero-shot reference voice cloning.

By leveraging the NaijaVoices dataset, executing cross-lingual embedding transplantation, and conducting 8-epoch training runs, I produced 3 fine-tuned model checkpoints uploaded to Hugging Face. Here is the full technical deep dive.

1. The Dataset: Crowdsourced Nigerian Acoustics

The training data was drawn directly from the Nigerian language splits of naijavoices/naijavoices-dataset:

• Scale: Approximately 8,000 paired audio-text samples per language.
• Acoustic Diversity: Real-world smartphone and laptop recordings featuring natural room reflections, microphone variances, authentic cadence, and everyday colloquial pronunciations.
• Audio Standardization: Resampled to 24 kHz mono with peak normalization.

Because Chatterbox generates speech autoregressively conditioned on a short audio prompt, fine-tuning posed two fundamental questions:

1. Would introducing new language tokens destabilize the pre-trained autoregressive priors?
2. At what point during multi-epoch fine-tuning does the model strike the optimal trade-off between phonetic accuracy and acoustic naturalness before overfitting on discrete codebooks?

2. Cross-Lingual Embedding Transplantation

Chatterbox relies on prefix language tags (such as [en], [es], [sw]) to steer internal attention mechanisms toward target phonetic profiles. To integrate WaZoBia, two critical architectural adaptations were made:

1. Tokenizer Vocabulary Injection

Explicit routing tokens [yo], [ig], and [ha] were dynamically registered within the tokenizer and supported language maps.

2. Swahili Donor Embedding Transplantation

In transformer language models, initializing new token embeddings with random Gaussian weights creates a severe gradient discontinuity during initial training steps. The high loss spikes propagate backward through the attention heads, degrading pre-trained features.

To avoid this, I performed cross-lingual embedding transplantation: cloning embedding weights from Swahili ([sw]), an existing African language in the base vocabulary with rich vowel structures, into [yo], [ig], and [ha]. This provided the autoregressive heads with a warm phonetic anchor from step 0, ensuring smooth gradient descent.

3. Training Trajectory & The Optimal Stopping Dynamic

Fine-tuning ran across accelerated GPU hardware with mixed precision (bfloat16/fp16) using the AdamW optimizer across 8 epochs (~31,360 total optimization steps per language).

Multi-Loss Tracking Across All Three Languages

Yoruba (yo)

Hausa (ha)

Igbo (ig)

The “Optimal Stopping Dynamic”

Across all three languages, an identical phenomenon occurred:

• Total evaluation loss reaches its minimum around Epoch 4 to 5 (steps 15,680–19,600).
• If training is extended toward Epoch 8, Text Cross-Entropy drops precipitously toward near-zero (0.02–0.16), indicating the model is memorizing the text.
• Concurrently, Speech Codebook Cross-Entropy climbs above 6.8–7.3. In listening tests, checkpoints trained past Epoch 6 develop vocoder raspiness and codebook artifacting.
• Optimal Checkpoint Selection: Checkpoints captured between steps 15,680 and 19,600 deliver the sweet spot of crisp articulation and clean acoustic clarity.

Model Weight Blending

To further guard against flow-matching vocoder degradation, fine-tuned weights were merged with the base foundation weights using linear interpolation: weights_final = (0.7 * weights_fine_tuned) + (0.3 * weights_base)

This blending retains 100% of the newly acquired WaZoBia phonetic mappings while stabilizing the diffusion prior.

4. Audio Synthesis Comparisons: Unseen Speaker Cloning

Below are side-by-side listening evaluations comparing original speakers against Chatterbox cloned audio:

Yoruba (yo)

Hausa (ha)

Igbo (ig)

5. Released Hugging Face Checkpoints

The 3 fine-tuned Chatterbox T3 model checkpoints (~1.2B parameters, ~2.4 GB each) are published on Hugging Face:

• Yoruba: samuelolubukun/chatterbox-yoruba-naijavoices
• Igbo: samuelolubukun/chatterbox-igbo-naijavoices
• Hausa: samuelolubukun/chatterbox-hausa-naijavoices

6. Gradio Web Studio Source & GitHub

The complete Gradio app implementation is open-source on GitHub, ready to be cloned and run locally or deployed directly to a Hugging Face Space for conversational speech synthesis, testing prompt controls, and performing zero-shot voice cloning across all three languages:

• GitHub Repository: samolubukun/wazobia-chatterbox — WaZoBia (Hausa, Yoruba, Igbo) Chatterbox Conversational Speech Synthesis Studio