TTS

Speech Synthesis

XTTS-v2

WaZoBia

Deep Learning

African NLP

Voice AI

Fine-Tuning XTTS-v2 for WaZoBia: From Studio Scripture to Conversational Speech

September 28, 2026 9 min read

Fine-Tuning XTTS-v2 for WaZoBia: From Studio Scripture to Conversational Speech
1. Linguistic Challenges & Orthographic Foundations
2. Stage 1 Output: Phonemic Grounding and Prosodic Tradeoffs
3. Tokenizer Vocabulary Extension
4. Training Mechanics & Multi-Stage Curricula
5. Acoustic Evaluation: Stage 1 Foundation vs. Stage 2 Conversational
6. Audio Samples: Side-by-Side Comparisons
7. Released Hugging Face Checkpoints
8. Gradio Web Studio Source & GitHub

Fine-Tuning XTTS-v2 for WaZoBia: From Studio Scripture to Conversational Speech

For this experiment, I fine-tuned Coqui XTTS-v2 to synthesize speech in Nigeria’s three major indigenous languages — Yoruba (yo), Igbo (ig), and Hausa (ha) (collectively termed WaZoBia). I ran training in two deliberate stages: first on OpenBible studio recordings to anchor phonemic and tonal grounding, then a second pass on crowdsourced Nigerian Common Voice clips to shift the model toward natural conversational cadence.

Before any training could produce usable output, one concrete issue had to be fixed at the tokenizer level:

Orthographic Mutilation: Standard tokenizers are not built for these scripts. Tonal markers (acute ´, grave `), sub-dots (ẹ, ọ, ṣ, ị, ọ, ụ), and hooked consonants (ɓ, ɗ, ƙ, ƴ) either get dropped or corrupted into <unk> tokens or mismatched Latin characters — collapsing phonemic distinctions that are fundamental to meaning in all three languages.

With that resolved, training proceeded across both stages. A total of 6 fine-tuned checkpoints were published to Hugging Face. Below is a full breakdown of the linguistic structure, tokenizer changes, training mechanics, loss convergence, and comparative acoustic evaluation.

1. Linguistic Challenges & Orthographic Foundations

Each language has orthographic and tonal structures that the base tokenizer and model had to be explicitly prepared for:

Yoruba (`yo`)

Register Tone System: Yoruba employs three level register tones (High with acute ´, Mid unmarked, and Low with grave `). Tone alters lexical meaning entirely: ọwọ́ (hand), ọ̀wọ̀ (respect), ọwọ (broom).
Sub-dot Vowels & Consonants: ẹ (/ɛ/), ọ (/ɔ/), and ṣ (/ʃ/) denote phonemically distinct entities.
Syllabic Nasals: ń, ǹ, and m̀ function as tone-bearing syllable nuclei.
Continuous Vowel Elision: In fluid speech, adjacent vowels across word boundaries contract (e.g., wá sí ilé $\to$ wásílé). Read-speech audio tends to pronounce each syllable in isolation, which the Stage 2 pass addresses.

Igbo (`ig`)

Ọnwụ Orthography: Codified standard featuring sub-dotted vowels ị (/ɪ/), ọ (/ɔ/), and ụ (/ʊ/), alongside the velar nasal ṅ (/ŋ/).
Vowel Harmony: Vowels divide strictly into Advanced Tongue Root (+ATR) and non-ATR (-ATR) categories. Dotted vowels cannot co-occur with non-dotted equivalents in root words.
Enclitics & Compound Morphology: Verbs carry complex affixes and elisions (na-aga, n’ụlọ). Preserving morphological tone contours is essential for listener comprehension.

Hausa (`ha`)

Hooked Consonants (Implosive / Ejective): Hausa utilizes ɓ (voiced bilabial implosive), ɗ (voiced alveolar implosive), ƙ (voiceless velar ejective), and ƴ (glottalized palatal glide). Standard English tokenizers drop these hooks, fatally collapsing ƙasa (country/earth) into kasa (fail to reach).
Tone & Quantity: While tone and vowel length are rarely transcribed in everyday Boko text, speech synthesis models must accurately generate natural prosodic duration from context.

2. Stage 1 Output: Phonemic Grounding and Prosodic Tradeoffs

OpenBible provided 30 to 45 hours of clean, studio-recorded audio per language — consistent SNR, clear enunciation, and minimal noise. It was a practical starting point for getting the model to correctly produce WaZoBia phonemes and tones. Stage 1 achieved exactly that.

The observable tradeoff was prosodic. Scripture narration has its own register, and the model learned it:

Slower Tempo: Stage 1 output settled around 110–120 WPM — accurate but measured, not conversational (natural Nigerian speech moves at 160–185 WPM).
Formal Intonation: Phrase-final fundamental frequency ($F_0$) contours were long and descending, typical of formal reading rather than everyday speech.
Syllabic Isolation: Individual syllables were cleanly articulated but lacked the co-articulation of natural flowing speech.

This is what Stage 2 was for — not to undo Stage 1, but to shift the temporal and prosodic profile on top of the phonemic foundation already established.

Two-Stage Fine-Tuning Pipeline

Stage 1: OpenBible Studio Foundation31,360 Steps • 8 Epochs

~35 hours clean studio audio per language (22.05 kHz high-SNR)
Extended BPE vocabulary (2,000 tokens for tone marks & sub-dots)
Effective batch size 8-16, learning rate 5e-6 with MultiStepLR
Outcome: Accurate tonal grounding, but solemn, slow recitation cadence (~110-120 WPM)

↓Checkpoint Weights Transfer↓

Stage 2: Common Voice Conversational Adaptation7,895 Steps • 4-5 Epochs

Crowdsourced Nigerian Common Voice clips (filtered 1.5s-10.0s)
Loudness normalized to -23 LUFS via pyloudnorm to curb room reverb
Conservative learning rate (1e-6 to 1.5e-6) to guard against noise drift
Outcome: Fluid spoken pacing (160-185 WPM), natural co-articulation (+75%), wider pitch dynamics

3. Tokenizer Vocabulary Extension

To prevent tonal marks from collapsing into <unk>, the base Byte-Pair Encoding (BPE) vocabulary was explicitly expanded to 2,000 tokens per language. Special language routing identifiers ([yo], [ig], [ha]) were injected into the tokenizer dictionary, preserving diacritics throughout text encoding.

This guarantees that tone contours directly inform the autoregressive latent mel predictor rather than getting stripped at input ingestion.

4. Training Mechanics & Multi-Stage Curricula

Training was orchestrated using high-memory GPU compute across both stages:

Stage 1: Foundation Training

Audio Processing: Studio audio resampled to 22.05 kHz mono with speaker latents extracted.
Optimization: AdamW with MultiStepLR, effective batch size of 8–16 (batch size 2 with gradient accumulation of 4), learning rate 5e-6.
Duration: 8 full epochs (31,360 optimization steps).
Loss Metric: Joint GPT text cross-entropy loss, mel spectrogram cross-entropy, and VAE vocoder loss.

Stage 2: Domain Adaptation

Audio Curation: Crowdsourced Nigerian Common Voice clips filtered strictly between 1.5s and 10.0s duration. Integrated loudness normalized to -23 LUFS to prevent room reverb and recording level jumps from distorting the vocoder.
Optimization: 4 to 5 epochs (7,488–7,895 steps) with a conservative learning rate of 1.0e-6 to 1.5e-6. This rate is low enough to prevent unlearning Stage 1 phonetic mastery while adjusting temporal attention weights.

Loss Trajectories Across Languages

Stage 1: Foundation Convergence (OpenBible - 31,360 Steps)

Yoruba (`yo`) — Stage 1 Foundation

XTTS-v2 Yoruba Stage 1 Foundation Training Curves

Text Cross-Entropy Loss: Decreased by 86.7% (0.2430 $\to$ 0.0323), confirming solid phonetic grounding of tone markers and diacritics (ẹ, ọ, ṣ).
Mel Spectrogram Loss: Converged by 45.6% (3.9983 $\to$ 2.1748), sharpening acoustic fidelity and vocal clarity.
Validation Loss: Monotonic decrease from 2.6773 to 2.2072 without acoustic collapse.

Hausa (`ha`) — Stage 1 Foundation

XTTS-v2 Hausa Stage 1 Foundation Training Curves

Text Cross-Entropy Loss: Decreased by 87.6% (0.2383 $\to$ 0.0294), stabilizing hooked consonants (ɓ, ɗ, ƙ, ƴ).
Mel Spectrogram Loss: Dropped from 3.0465 to 2.0128 (33.9% reduction), removing synthetic buzz.
Validation Loss: Consistently declined across all 8 epochs, reaching final Eval Total Loss of 2.1034.

Igbo (`ig`) — Stage 1 Foundation

XTTS-v2 Igbo Stage 1 Foundation Training Curves

Text Cross-Entropy Loss: Decreased by 86.1% (0.2446 $\to$ 0.0339), stabilizing articulation across dotted vowels (ị, ọ, ụ) and nasal ñ.
Mel Spectrogram Loss: Dropped from 3.6523 to 2.0790 (43.1% reduction), producing clean acoustic formant synthesis.

Stage 2: Conversational Adaptation Convergence (Common Voice)

Yoruba (`yo`) — Stage 2 Conversational

XTTS-v2 Yoruba Conversational Training Dashboard

Stage / Step	Global Step	Train Loss	Text CE Loss	Mel Spectrogram Loss	Convergence Note
Adaptation Start	0	0.9682	0.0461	3.8420	Initialized directly from OpenBible 31,360-step foundation weights
Epoch 2 Midpoint	2,650	0.8696	0.0460	3.4325	Mel loss descends as conversational acoustic priors take effect
Epoch 4 Midpoint	5,500	0.8411	0.0430	3.3215	Steady convergence; text loss improves with zero forgetting
Adaptation Final	7,895	0.8298	0.0385	3.2082	Mastered fluid Yoruba speech rhythm and question tone glides

Hausa (`ha`) — Stage 2 Conversational

XTTS-v2 Hausa Conversational Training Dashboard

Stage / Step	Global Step	Train Loss	Text CE Loss	Mel Spectrogram Loss	Convergence Note
Adaptation Start	0	0.9753	0.0358	3.8655	Initialized directly from OpenBible 31,360-step foundation weights
Epoch 1 Complete	1,898	0.8105	0.0368	3.2052	Rapid acoustic convergence on crowdsourced Hausa voices
Epoch 2 Complete	3,796	0.7707	0.0351	3.0477	50% adaptation milestone; significant decrease in scripted reading cadence
Epoch 3 Complete	5,694	0.7424	0.0344	2.9402	Pitch and breath boundaries align with natural colloquial speech
Epoch 4 Final	7,592	0.7247	0.0344	2.8578	Fully adapted conversational model (`checkpoint_7500.pth`, eval mel: 2.8535)

Igbo (`ig`) — Stage 2 Conversational

XTTS-v2 Igbo Conversational Training Dashboard

Stage / Step	Global Step	Train Loss	Text CE Loss	Mel Spectrogram Loss	Convergence Note
Adaptation Start	0	0.9812	0.0425	3.8210	Initialized directly from OpenBible 31,360-step foundation weights
Epoch 2 Midpoint	2,100	0.8760	0.0421	3.4620	Mel loss drops steadily as conversational acoustics take hold
Epoch 3 Midpoint	4,200	0.8607	0.0411	3.4019	Consistent descent; text loss stable with zero forgetting
Adaptation Final	7,488	0.8487	0.0384	3.3547	Mastered human conversational pace and downstep contours

5. Acoustic Evaluation: Stage 1 Foundation vs. Stage 2 Conversational

Evaluating speech outputs across test speakers confirms dramatic shifts in prosody, tempo, and fluidity:

Evaluated Dimension	Stage 1 (OpenBible Foundation)	Stage 2 (Conversational Adapted)	Relative Change	Practical Significance
Complex Clause Duration	17.64s	6.12s	-65.3%	Cuts unnatural pauses; fluid conversational flow
Average Utterance Duration	8.81s	5.09s	-42.2%	Speaking rate aligns with human conversation
Speaking Rate (WPM)	110–120 WPM	160–185 WPM	+45.0%	Matches natural Nigerian cadence
Pitch Dynamic Swing (StdDev)	31.3 Hz	38.3 Hz	+22.3%	Expressive tonal swing on questions and emphasis
Syllable Co-Articulation Score	52.0%	91.0%	+75.0%	Words blend naturally; eliminates robotic staccato