Authors - Aprna Tripathi, Akhilesh Kumar Sharma, Avisikta Pal, Srikanth Prabhu, Ramakrishna Mundugar, Reet Ginotra Abstract - This paper presents a novel approach to identifying translation errors in Thai-English machine translation through the comparative analysis of multiple automatic evaluation metrics. Using a rank deviation methodology, we evaluate 350 Thai-English translations produced by seven contemporary systems provid ing online translations — including dedicated Machine Translation systems and large language models — across nine automatic evaluation metrics. By ranking translations within each metric and comparing individual metric rankings against the mean average rank, we identify translations that receive solitary punishment from a single metric, isolating these as candidates for manual error analysis. Our results demonstrate that individual metrics exhibit distinct sensitivity to specific error types, and that surface-level metrics retain diagnostic value along side advanced neural metrics. Neural metrics effectively identify meaning and adequacy errors, but surface-level metrics uniquely identify morphological vari ation, word order errors, preposition choice, and number formatting issues that neural metrics fail to penalize. The diversity of metric sensitivity is therefore an asset rather than an inconvenience, enabling a more complete characterization of translation error than any single metric can provide. This research supports the development of high-quality training data for MT fine-tuning by identifying the specific error types that individual metrics can and cannot detect and provides a repeatable diagnostic methodology applicable to other language pairs.
Open Zoom