MIT CSAIL Research Finds 40% Performance Gap Between Leading LLMs on Asian Language Reasoning Tasks vs English

New research from MIT CSAIL's Natural Language Processing group has quantified the performance gap between leading large language models on English versus Asian language reasoning tasks — finding an average 40% reduction in reasoning accuracy across complex tasks in Japanese, Korean, Vietnamese, Bahasa Indonesian, and Thai compared to the same models' performance on equivalent English-language tasks.

The research evaluated GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3 70B across four reasoning categories: multi-step mathematical reasoning, document comprehension and synthesis, logical inference from domain-specific text, and structured information extraction. The 40% gap is an average across languages — Vietnamese and Bahasa Indonesian show larger gaps (50–60%) while Japanese and Korean show smaller gaps (25–30%) due to higher representation in training data.

For APAC enterprise AI leaders making model selection decisions, the MIT research provides empirical grounding for assumptions that practitioners have observed informally: that off-the-shelf Western foundation models perform significantly below their English benchmarks on the Asian language tasks that APAC enterprise deployments require. The research validates investment in localised model fine-tuning, APAC-specific model evaluation frameworks, and the ongoing development of Asian-language foundation models (EXAONE, phoBERT, SeaLLM) that APAC enterprises are increasingly considering for language-sensitive applications.