Translation quality checks challenge both humans and language models

Apple research finds that large language models remain unreliable substitutes for human judgment in most machine translation evaluation settings. Although model agreement with human annotators exceeds agreement between human annotators on some tasks, performance varies substantially across annotation schemes, language pairs and domains.

The evaluation compares models and human annotators using Multidimensional Quality Metrics (MQM) and Error Span Annotation (ESA). It examines agreement on quality scores and annotated error spans using a long-context test set covering 70 language pairs, alongside publicly available WMT23 and WMT25 data.

Humans face particular difficulties with fine-grained MQM annotations and low-resource language pairs. Models struggle with minor errors, incorrect language variants and identifying error spans. These distinct weaknesses underscore the need to evaluate translation assessment tools across languages, domains and annotation granularity before relying on them as alternatives to human evaluation.

Human-model collaborative annotation pipelines could offer a path toward improvement by addressing the reliability problems affecting both groups. The findings suggest potential for collaboration rather than establishing models as dependable replacements.