KAIST and Seoul National University Publish Korean Financial Regulatory LLM Benchmark Favoring Fine-Tuned APAC Models

Researchers at KAIST (Korea Advanced Institute of Science and Technology) and Seoul National University published a benchmark evaluating twelve large language models on Korean financial regulatory text comprehension tasks, using a curated dataset of FSS (Financial Supervisory Service) and FSC (Financial Services Commission) regulatory documents, enforcement actions, and compliance interpretation questions that Korean financial institutions must navigate.

The benchmark results showed that domain-specific fine-tuned Korean language models — EXAONE 3.5 (LG AI Research) and HyperCLOVA X (NAVER) — outperformed larger frontier models including GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro on Korean FSS regulatory interpretation tasks, with the performance gap widening on documents requiring understanding of Korean-specific regulatory concepts (예금자보호법, 금융소비자보호법, 전자금융거래법) that frontier models trained primarily on English regulatory text underperform on.

The research team noted that the performance advantage of fine-tuned Korean models held specifically on interpretation tasks requiring understanding of FSS enforcement precedents and FSC policy guidance — regulatory knowledge that is primarily available in Korean and not well represented in frontier model training corpora. For APAC AI adoption teams at Korean financial institutions evaluating LLM deployment for compliance automation, the benchmark suggests that domain-specific fine-tuning on Korean regulatory text provides a substantial quality advantage over general-purpose frontier models for Korean FSI compliance use cases, even when the fine-tuned models have significantly smaller parameter counts (7B-70B vs 175B+ for frontier models).