Experiments in Fine-Tuning Moonshine-Tiny ASR for WaZoBia: v1 vs. v2

UsefulSensors’ Moonshine-Tiny (~27 million parameters, ~55 MB memory footprint) is an ultra-compact encoder-decoder speech recognition model engineered specifically for edge mobile execution and compute-constrained micro-devices. Out of the box, it has no support for Nigeria’s three primary indigenous languages — Yoruba (yo), Igbo (ig), and Hausa (ha) (WaZoBia) — and produces largely unusable transcriptions on all three.

The primary cause is an acute vocabulary mismatch: default SentencePiece tokenizers strip sub-dot marks and tonal diacritics, while the pre-training acoustic distribution lacks West African phone sets. Furthermore, small encoder-decoder architectures run a persistent risk of getting trapped in cyclic autoregressive hallucination loops.

For this experiment, I built a comparative training methodology producing two distinct model generations across all three languages (6 checkpoints total):

• v1 (Studio Foundation): Fine-tuned exclusively on clean studio scripture recordings from OpenBible to learn baseline phonetic and diacritic orthography.
• v2 (Multi-Stage Domain Curriculum): A 3-stage progressive curriculum cascading from OpenBible through Google FLEURS (multi-speaker formal broadcast reading) to NaijaVoices (crowdsourced everyday smartphone audio).

Below is the complete architectural walkthrough of the orthographic fixes, curriculum dynamics, hallucination prevention strategies, and the dual-track evaluation dilemma.

1. Orthographic Engineering & Boundary Healing

African tonal languages rely on diacritics for lexical differentiation. Moonshine’s standard SentencePiece vocabulary routinely decomposed or discarded these symbols, resulting in cascading error rates before the decoder even generated tokens.

Tokenizer Expansion Across WaZoBia

The SentencePiece model was modified and the model token embeddings resized to natively represent:

• Yoruba (yo): 29 diacritic and tone units: ['ẹ', 'ọ', 'ṣ', 'ẹ́', 'ẹ̀', 'ì', 'ò', 'ọ́', 'ọ̀', 'ù', 'ń', 'ǹ', '<|yo|>']
• Igbo (ig): Ọnwụ standard orthography units and dotted vowels: ['ị', 'ọ', 'ụ', 'ṅ', 'ch', 'gw', 'kp', 'kw', 'nw', 'ny', 'sh', '<|ig|>']
• Hausa (ha): Ingressive and glottalized hooked letters: ['ɓ', 'ɗ', 'ƙ', 'ƴ', 'R̃', 'ts', '<|ha|>']

Orthographic Boundary Healing

Raw audio transcripts frequently contain optical character recognition (OCR) artifacts or compound-word formatting glitches:

• In Igbo, enclitic affixes and compound prepositions often fracture or glue incorrectly (e.g., n'ime, maọbụ, ga-e). Automated regex cleaning normalized these boundaries prior to tokenization.
• In Yoruba, common contractions and separated morphemes (such as tani + ọba or ọlọ́gbọ́ n $\to$ ọlọ́gbọ́n) were systematically repaired.

2. Preventing Autoregressive Hallucination Loops

Compact encoder-decoder models are notorious for autoregressive loop degradation, especially on liturgical datasets where repetitive chapter-and-verse structures cause models to emit continuous scripture citation hallucinations.

To completely eradicate this failure mode:

1. Right-Shifted BOS Formatting: Inputs to the seq2seq trainer were explicitly right-shifted with Beginning-Of-Sequence (BOS) tokens, conditioning the decoder strictly on acoustic encodings rather than self-referential token recurrence.
2. Duration-Scaled Token Clamping: During beam search inference, token budgets are dynamically scaled based on active speech duration, preventing degenerate runaways: token_limit = min(160, max(24, int(duration_sec * 7.5) + 6))

Across hold-out benchmark splits, this reduced citation hallucination rates from over 18% down to 0.0%.

3. The 3-Stage Training Curriculum in v2

In v1, training strictly on OpenBible gave the model clean phonetic representations, but accuracy degraded sharply on colloquial mobile audio due to ambient room noise, mic distortion, and natural co-articulation.

For v2, I established a staged domain curriculum allowing the attention layers to adapt smoothly across acoustic regimes:

Stage 1 Training Loss Progression Across Languages

Yoruba (yo)

Hausa (ha)

Igbo (ig)

Stage 2 & Stage 3 Refinement Details (The Final v2 Checkpoint)

Following Stage 1 foundation training, Moonshine progressed through two successive refinement stages to transition from formal studio recitation to multi-speaker broadcast speech and finally authentic conversational Nigerian speech:

Final Stage 3 (v2 Checkpoint) Quantitative Benchmark Performance

Below is the evaluation performance of the final Stage 3 (v2) model across holdout splits on formal broadcast reading vs. everyday spontaneous conversation:

4. The Dual-Track Evaluation Dilemma (Strict vs. Toneless)

A major revelation from this project concerns how African language speech recognition is benchmarked in academic literature:

• In formal benchmarks like Google FLEURS, reference transcripts are often written tonelessly (e.g. annotators write ogun or ọwọ without accents).
• When a fine-tuned model correctly outputs full, grammatically accurate tonal orthography (e.g. ọ̀wọ̀ or ọwọ́), standard Levenshtein Word Error Rate (WER) marks every single word as a deletion and substitution error.
• This creates an artificial paradox where Strict WER approaches 100%, even though the acoustic transcription is phonetically spot-on and character accuracy is high.

Evaluating low-resource African ASR requires dual-track benchmarking: measuring both strict diacritic precision and toneless phonetic word recognition.

5. Quantitative Benchmark Comparison (All 6 Checkpoints)

Below is the complete benchmark breakdown across hold-out sets (100 unseen samples per domain):

On spontaneous conversational audio, Moonshine v2 achieves a Toneless CER of 28.8% to 38.9%, correctly transcribing over two-thirds of all spoken characters even in noisy, casual settings.

6. Live Audio Samples & Edge Execution Speed

Because Moonshine-Tiny possesses only ~27M parameters, it achieves an ultra-fast 0.305× real-time factor (RTF) on low-power edge hardware. A 5.75-second audio clip decodes in approximately ~1.75 seconds.

Here are playable test samples from the evaluation benchmark splits:

1. Google FLEURS Audio Evaluation

• Sample Audio:
• Decoded Text: “Ònkọ̀wé náà fi ìwé rẹ̀ hàn fún àwọn ènìyàn.”

2. NaijaVoices Conversational Audio Evaluation

• Sample Audio:
• Decoded Text: “Mo gbọ́ ohun tí o sọ nípa ọ̀rọ̀ náà.”

3. OpenBible Studio Audio Evaluation

• Sample Audio:
• Decoded Text: Clear studio scripture narration with 0% citation hallucination.

7. Released Hugging Face Checkpoints

All 6 Moonshine-Tiny ASR checkpoints (~55 MB footprint each) are available on Hugging Face:

Moonshine-Tiny v2 (3-Stage Multi-Domain Checkpoints)

• Yoruba: samuelolubukun/moonshine-tiny-yoruba-v2
• Igbo: samuelolubukun/moonshine-tiny-igbo-v2
• Hausa: samuelolubukun/moonshine-tiny-hausa-v2

Moonshine-Tiny v1 (OpenBible Studio Checkpoints)

• Yoruba: samuelolubukun/moonshine-tiny-yoruba-openbible
• Igbo: samuelolubukun/moonshine-tiny-igbo-openbible
• Hausa: samuelolubukun/moonshine-tiny-hausa-openbible

8. Gradio Web Studio Source & GitHub

The complete Gradio web app implementation is open-source on GitHub, ready to be cloned and run locally or deployed directly to a Hugging Face Space to test live microphone recordings and audio file transcriptions with any of the 6 fine-tuned checkpoints:

• GitHub Repository: samolubukun/wazobia-moonshine — WaZoBia (Hausa, Yoruba, Igbo) Moonshine-Tiny Automatic Speech Recognition Studio