Skip to content
GCC AI Research

Search

Results for "Distillation"

State-of-the-Art Arabic Language Modeling with Sparse MoE Fine-Tuning and Chain-of-Thought Distillation

arXiv ·

Arabic-DeepSeek-R1 is an application-driven, open-source Arabic Large Language Model (LLM) that has achieved a new state-of-the-art (SOTA) across the Open Arabic LLM Leaderboard (OALL). The model utilizes a sparse Mixture-of-Experts (MoE) backbone and a four-phase Chain-of-Thought (CoT) distillation scheme, which incorporates Arabic-specific linguistic verification and regional ethical norms. It records the highest average score on the OALL suite and outperforms proprietary frontier systems like GPT-5.1 on a majority of benchmarks evaluating comprehensive Arabic language-specific tasks. Why it matters: This work offers a validated and cost-effective framework for developing high-performing, culturally-grounded AI for under-represented languages, addressing the digital equity gap.

EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewards

arXiv ·

Researchers at MBZUAI have introduced EvoLMM, a self-evolving framework for large multimodal models that enhances reasoning capabilities without human-annotated data or reward distillation. EvoLMM uses two cooperative agents, a Proposer and a Solver, which generate image-grounded questions and solve them through internal consistency, using a continuous self-rewarding process. Evaluations using Qwen2.5-VL as the base model showed performance gains of up to 3% on multimodal math-reasoning benchmarks like ChartQA, MathVista, and MathVision using only raw training images.

Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillation

arXiv ·

Researchers at MBZUAI have demonstrated a method called "Data Laundering" to artificially boost language model benchmark scores using knowledge distillation. The technique covertly transfers benchmark-specific knowledge, leading to inflated accuracy without genuine improvements in reasoning. The study highlights a vulnerability in current AI evaluation practices and calls for more robust benchmarks.

Distillation Policy Optimization

arXiv ·

The paper introduces a novel actor-critic framework called Distillation Policy Optimization that combines on-policy and off-policy data for reinforcement learning. It incorporates variance reduction mechanisms like a unified advantage estimator (UAE) and a residual baseline. The empirical results demonstrate improved sample efficiency for on-policy algorithms, bridging the gap with off-policy methods.

Sustainable membranes for future energy

KAUST ·

KAUST researchers have developed polytriazole membranes for energy-efficient crude oil fractionation, as detailed in a recent Science Magazine paper. Led by Dr. Suzana Nunes and Dr. Stefan Chisca, the team created membranes that can withstand harsh industrial conditions like high temperatures and organic solvents. The membranes offer a low-carbon footprint alternative to traditional separation techniques like distillation. Why it matters: This innovation could significantly reduce energy consumption and promote a circular carbon economy in the petrochemical industry within the GCC region and beyond.

Solar desalination—from lab to plant

KAUST ·

KAUST's Water Desalination and Reuse Center (WDRC) is developing solar-powered seawater desalination technologies, including the MEDAD cycle which combines adsorption desalination (AD) and multi-effect distillation (MED). The MEDAD cycle, developed by Professor Kim Choon Ng, doubles water production at the same temperature, reducing costs to $0.48/m3 compared to $1.201/m3 for multi-stage flash distillation. A 100 m3/day commercial-scale MEDAD project was commissioned in Riyadh in 2017 in collaboration with KACST, and a larger 2,000 m3/day project is planned for Yanbu. Why it matters: This highlights Saudi Arabia's move towards sustainable energy and the role of research institutions like KAUST in developing cost-effective desalination technologies suitable for the region.

Knowledge distillation and the greening of LLMs

MBZUAI ·

Researchers from MBZUAI, University of British Columbia, and Monash University have created LaMini-LM, a collection of small language models distilled from ChatGPT. LaMini-LM is trained on a dataset of 2.58M instructions and can be deployed on consumer laptops and mobile devices. The smaller models perform almost as well as larger counterparts while addressing security concerns. Why it matters: This work enables the deployment of LLMs in resource-constrained environments and enhances data security by reducing reliance on cloud-based LLMs.

MBZUAI research at ICLR 2023

MBZUAI ·

MBZUAI had 22 papers accepted at ICLR 2023, with faculty Kun Zhang co-authoring seven of them. Yuanzhi Li, an affiliated assistant professor at MBZUAI, received an honorable mention for his paper on knowledge distillation. Additionally, a paper co-authored by MBZUAI President Eric Xing was recognized as a top 5% paper at the conference. Why it matters: MBZUAI's strong presence at a top-tier machine learning conference like ICLR demonstrates the university's growing influence and research capabilities in the global AI landscape.