Forensic Transcription of Kazakh Speech Evidence: Evaluating Whisper and LLM Post-Correction Reliability

Аңдатпа

Digital forensic investigations increasingly depend on transcribing recorded speech evidence—wiretaps, witness interviews, and seized audio—for use in legal proceedings, yet automatic speech recognition (ASR) for low-resource languages such as Kazakh remains unreliable, and no prior work has evaluated whether current ASR and large language model (LLM) post-correction pipelines meet the accuracy and integrity requirements of forensic transcription. Methods. This study presents the first comprehensive evaluation of OpenAI's Whisper models for Kazakh speech-to-text, examining both generic multilingual and language-specific fine-tuned variants, and testing whether general-purpose LLMs (Gemma-2-9B and Qwen2.5-7B) can serve as forensically sound post-correction tools. We evaluate two Whisper variants on 30,489 samples from the Kazakh Speech Dataset, applying rigorous bootstrap statistical analysis and character-level error analysis relevant to the preservation of evidentiary meaning. Results. The Kazakh-specific fine-tuned model achieves a 44.2% relative WER reduction compared to the generic multilingual model (32.86% vs. 58.90% WER), with non-overlapping 95% confidence intervals demonstrating statistical robustness. Critically, general-purpose LLMs fail to improve—and in the case of Gemma, actively degrade (+4.1% WER)—transcription quality, raising integrity concerns for any forensic workflow that would apply uncontrolled LLM post-editing to evidentiary transcripts. Error analysis identifies systematic character-level confusions between Kazakh-specific Cyrillic characters and Russian equivalents, a class of error capable of altering names, locations, and other evidentiary details, with historical or archival recordings exhibiting 2× higher error rates than contemporary material. Conclusions. These results indicate that only language-specific fine-tuned ASR, combined with mandatory human expert verification rather than automated LLM post-correction, can approach the reliability threshold required for admissible forensic transcription of Kazakh speech evidence, and they establish reproducible benchmarking practices and error-risk indicators for forensic deployment of low-resource-language ASR.