I fine-tuned Whisper Turbo with LoRA and pointed it at my own music. The track is “whatsapp girl” from POP OUT! — mostly Korean-English rap, which makes it a near-adversarial stress test for ASR. In the video the model’s cumulative word error rate updates line by line, in real time, as the song plays.
Final score: 42.6% WER / 29.2% CER — strict scoring, no normalization.
Three things the errors taught me:
-
Language-switching is the failure mode. 11 of 33 lines are pure Korean — all scored 0% WER. Nearly every error sits on a Korean↔English boundary (“frenchy한 accent” → “French I’m next and”). A monolingual song would have crushed this.
-
The trilingual countdown broke it completely. 셋 둘 하나 / twah duh uh / quatro tres dos uno all collapsed into “3, 2, 1” — number normalization stacking on top of language-ID confusion.
-
The model heard right but decoded in the wrong language. 피식피식 (a Korean giggle) became “piss it piss it” — phonetically faithful, semantically chaos. That’s why CER runs ~13 points below WER: these are misinterpretations, not mishearings.
Caveat: one song, 148 words — small sample, wide error bars. So this becomes a series: I’ll keep a running cumulative WER as I record more, including monolingual covers as controls, and the graph picks up where this one ends.
Method note: WER was measured against a separate spoken recording of the lyrics — Whisper is trained on speech, not live singing.