Xiaomi Unveils MiMo-7B – Xiaomi’s First Open-Source AI Model for Reasoning and Code Generation

Highlights

• MiMo-7B is Xiaomi’s first open-source AI model designed for logic-driven tasks, outperforming OpenAI & Alibaba in math benchmarks.
• It is built with 200B reasoning tokens and trained on 25 trillion tokens.
• It uses multiple-token prediction for faster inference and reinforcement learning for improved accuracy.
• Available on GitHub & Hugging Face

Caption – Xiaomi Unveils MiMo-7B. (Image credit – Xiaomi)

Xiaomi has officially launched its first open-source large language AI model named MiMo-7B. Created by its newly formed Big Model Core Team, MiMo-7B is built to handle reasoning-heavy tasks and coding.

According to reports, the AI Model is already outperforming some big names in the field such as OpenAI and Alibaba when it comes to math and code-related benchmarks. Here’s what we know about the model as of now.

Xiaomi MiMo-7B

Caption – Xiaomi MiMo-7B outperforms OpenAI and Alibaba’s model. (Image credit – Gizmochina)

Xiaomi’s MiMo-7B is a 7-billion-parameter model. Even though it’s much smaller than most of today’s top models, Xiaomi says it can match the performance of larger systems like OpenAI’s o1-mini and Alibaba’s Qwen-32B-Preview. All three models are designed for tasks that involve logical reasoning.

How Xiaomi MiMo-7B Was Trained?

The strength of MiMo-7B comes from its intense training process. Xiaomi put together a dense dataset with 200 billion reasoning tokens and trained the model with a total of 25 trillion tokens across three phases.

Instead of the usual next-token prediction method, Xiaomi used a multiple-token prediction approach, which it says speeds up inference without sacrificing quality.

After pre-training, Xiaomi applied several techniques to fine-tune the model. It introduced a unique reinforcement learning method called Test Difficulty Driven Reward to solve the issue of weak reward signals in complex RL tasks. It also used something called Easy Data Re-Sampling to keep the training stable.

On the infrastructure side, Xiaomi built a Seamless Rollout system that helps reduce GPU downtime during training and validation. Thanks to this, Xiaomi claims a 2.29× boost in training speed and nearly 2× better validation performance. This system also supports more advanced inference strategies like multiple-token prediction in vLLM environments.

MiMo-7B Goes Open Source

Captopm – MiMo-7B Goes Open Source versions. (Image credit – Gizmochina)

There are four versions of MiMo-7B available to the public:

• Base – The raw, pre-trained version
• SFT – Fine-tuned using supervised learning
• RL-Zero – Reinforcement-learned from the base version
• RL – Built on top of the SFT version, this one is said to deliver the best accuracy

And Xiaomi has benchmark scores to back it up. The MiMo-7B-RL model reportedly scores

• 8% on MATH-500
• Over 68% on the 2024 AIME dataset
• 8% on LiveCodeBench v5
• Just under 50% on LiveCodeBench v6

For broader tasks like DROP, MMLU-Pro, and GPQA, the scores sit in the mid-to-high 50% range, not groundbreaking, but pretty solid for a 7B model.

You can now find MiMo-7B on Hugging Face under an open-source license. If you want to dive deeper, all the model checkpoints and documentation are available on GitHub.

FAQs

Q.1 What is Xiaomi MiMo-7B, and what makes it unique?

Answer. MiMo-7B is Xiaomi’s first open-source AI model designed for reasoning-heavy tasks and coding. Despite having 7 billion parameters, Xiaomi claims it can match larger models like OpenAI’s o1-mini and Alibaba’s Qwen-32B-Preview in logic-driven benchmarks.

Q2. How was MiMo-7B trained, and what techniques were used?

Answer. Xiaomi trained MiMo-7B with 200 billion reasoning tokens and 25 trillion tokens using a multiple-token prediction approach, which speeds up inference. It also applied reinforcement learning techniques, including Test Difficulty Driven Reward and Easy Data Re-Sampling, to improve accuracy while maintaining stability.

Q3. Where can developers access MiMo-7B, and what versions are available?

Answer. MiMo-7B is available on Hugging Face and GitHub under an open-source license. Xiaomi has released four versions including Base, SFT, RL-Zero and RL.

Also Read: OpenAI launches O3 Mini: A fast and affordable AI for STEM and coding

Also Read: Xiaomi 15S Pro officially confirmed, expected to feature Xiaomi’s in-house chip