Evaluation of Small Language Models for Arabic Language Processing

Summary

A new paper evaluated twelve Small Language Models (SLMs) on Arabic natural language processing tasks, utilizing a benchmark of 240 Arabic test items across eight domains and ten language skills. The models were assessed in a zero-shot setting, with responses scored using a multi-model LLM-as-a-judge framework involving GPT-4.1 Mini, Claude Haiku 4.5, and DeepSeek-Chat. Gemma 3 (12B) achieved the highest overall score (4.548/5), followed by Aya and C4AI Command Arabic, with results suggesting that strong Arabic alignment and instruction-following are crucial for performance. Why it matters: This benchmark offers a standardized method for evaluating compact Arabic language models, guiding future development towards more efficient, reliable, and culturally relevant Arabic AI systems.