OpenAI’s o1 model has captivated the AI community with its exceptional reasoning capabilities. The model showcased outstanding performance on platforms like AIME and CodeForces. Inspired by this success, the AIDC-AI team, part of Alibaba’s International Digital Commerce division, aimed to enhance their reasoning abilities of Large Language Models (LLMs) to tackle complex, real-world challenges. So, they introduced an open-source large reasoning model (LRM) named Marco-o1. Let’s get into the details of this model.
Table of Contents
Introducing Marco-o1
Marco-o1 is an open-source large reasoning model focusing on disciplines with standard answers, such as mathematics, physics, and coding. It also places greater emphasis on open-ended resolutions. The team at AIDC-AI aims to address the question: “Can the o1 model effectively generalize to broader domains where clear standards are absent and rewards are challenging to quantify?”
Example Question Reasoned by Marco-o1
Key Highlights of Marco-o1
The model uses a range of advanced techniques to elevate its reasoning power, including:
1. Chain-of-Thought (CoT) Fine-Tuning
The AIDC-AI team developed Marco-o1-CoT by performing full-parameter fine-tuning on the base model using a combination of the open-source CoT dataset and their own self-developed synthetic data. This approach has significantly improved the model’s handling of complex tasks.
2. Monte Carlo Tree Search (MCTS)
By integrating LLMs with Monte Carlo Tree Search (MCTS), Marco-o1-MCTS can explore multiple reasoning paths using confidence scores derived from softmax-applied log probabilities of the top-k alternative tokens. This guided search process helps the model to identify optimal solutions.
3. Reasoning Action Strategy
The team implemented novel reasoning action strategies and a reflection mechanism, referred to as Marco-o1-MCTS Mini-Step. This includes exploring different action granularities within the MCTS framework, as well as prompting the model to self-reflect, significantly enhancing its problem-solving abilities.
4. Application in Translation Tasks
This model is the first to apply Large Reasoning Models (LRMs) to Machine Translation tasks, exploring inference time scaling laws in the multilingual and translation domain. The model has demonstrated its superior grasp of colloquial nuances, as evidenced by its ability to translate slang expressions accurately.
Performance Evaluation of Marco-o1
The model has demonstrated significant accuracy improvements across various datasets. For instance, the model achieved a +6.17% accuracy increase on the MGSM (English) dataset and a +5.60% improvement on the MGSM (Chinese) dataset. These enhancements showcase Marco-o1’s robust reasoning capabilities and its adaptability across different languages and contexts.
In addition to its reasoning prowess, this model excels in translation tasks, particularly with colloquial expressions. The model has successfully translated complex phrases, showcasing its ability to understand and convey nuanced meanings. For example, the phrase “This shoe offers a stepping-on-poop sensation” was accurately translated to “This shoe has a comfortable sole,” highlighting Marco-o1’s superior grasp of contextual language.
Marco-o1 Reasoning Datasets
To bolster its reasoning capabilities, Marco-o1 utilizes a diverse set of datasets, including:
1. Open-O1 CoT Dataset
This filtered dataset enhances the model’s structured reasoning patterns.
2. Marco-o1 CoT Dataset (Synthetic)
Generated using MCTS, this dataset helps formulate complex reasoning pathways.
3. Marco Instruction Dataset
This dataset is critical for instruction-following tasks, ensuring that the model remains effective across various applications.
In total, these datasets comprise over 60,000 samples, providing a comprehensive foundation for training and refining the model.
Solution Space Expansion via MCTS
The integration of LLMs with MCTS is a key component of Marco-o1, enhancing its reasoning capabilities.
1. Each node in the MCTS represents a reasoning state of the problem-solving process.
2. The possible actions from a node are the outputs generated by the LLM, representing potential steps or mini-steps in the reasoning chain.
3. During the rollout phase, the LLM continues the reasoning process to a terminal state, and the reward score is calculated based on the average confidence score of the tokens in the rollout sequence.
4. The reward score is used to evaluate and select promising paths within the MCTS, effectively guiding the search towards more confident and reliable reasoning chains.
Reasoning Action Strategy of Marco-o1
The Marco-o1 team explored different levels of granularity in the MCTS search to enhance the model’s problem-solving capabilities.
1. Action Selection
In addition to using complete reasoning steps as actions, the team experimented with using mini-steps of 32 or 64 tokens as actions. This finer granularity expanded the solution space, allowing the model to navigate complex reasoning tasks by considering more nuanced steps in the search process.
2. Reflection after Thinking
The team introduced a reflection mechanism by adding the phrase “Wait! Maybe I made some mistakes! I need to rethink from scratch.” at the end of each thought process. This prompt encourages the model to self-reflect and reevaluate its reasoning steps, leading to significant improvements, especially on difficult problems.
Marco-o1 vs Other Tools
1. Comparison with Google Translate
To illustrate its capabilities, Marco-o1 was put to the test against Google Translate in translating colloquial expressions. The results revealed that Marco-o1 outperformed standard translation tools.
2. Comparison with Other Models
When compared to the Qwen2-7B-Instruct model, Marco-o1-CoT showcased superior performance in the MGSM datasets, particularly in English. However, it experienced a slight decline in the Chinese dataset, indicating the challenges associated with language transfer in fine-tuning.
Future Developments
Marco-o1 is a significant step forward in the world of open reasoning models. By using advanced techniques like CoT fine-tuning, MCTS, and innovative reasoning strategies, the model has shown promising results in tackling complex, real-world problems with open-ended solutions. The team at AIDC-AI remains committed to continuous optimization and ongoing improvement of Marco-o1. Upcoming developments include the incorporation of Reward Models, such as Outcome Reward Modeling (ORM) and Process Reward Modeling (PRM), to provide more accurate reward signals for MCTS, as well as exploring Reinforcement Learning techniques to further refine the model’s decision-making processes.
Concluding Remarks
With its impressive capabilities, Marco-o1 holds the potential to become one of the best open-source large reasoning models available. To get started, head to HuggingFace. There, you will find details about loading the model and running inference. And, to learn more technical details, visit GitHub or the model’s arXiV paper.
- Forget Towers: Verizon and AST SpaceMobile Are Launching Cellular Service From Space
by Faizan Ali Naqvi - This $1,600 Graphics Card Can Now Run $30,000 AI Models, Thanks to Huawei
by Faizan Ali Naqvi - The Global AI Safety Train Leaves the Station: Is the U.S. Already Too Late?
by Faizan Ali Naqvi - The AI Breakthrough That Solves Sparse Data: Meet the Interpolating Neural Network
by Faizan Ali Naqvi - The AI Advantage: Why Defenders Must Adopt Claude to Secure Digital Infrastructure
by Faizan Ali Naqvi






