[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-267782-105":59,"doc-detail-267782-en":130},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":123,"head_meta":125,"extra_data":127,"updated_unix":129},105,"en","mammoth-building-math-generalist-models-through-hybrid-instruction-tuning","MAMMOTH - Building Math Generalist Models Through Hybrid Instruction Tuning","","MAMMOTH introduces an open-source large language model series for general math problem-solving, built through hybrid instruction tuning. The approach trains on MathInstruct, a curated dataset compiled from 13 math instruction sources, combining chain-of-thought and program-of-thought rationales. This design supports effective tool use, differentiates reasoning styles by problem type, and broadens coverage across mathematical fields, yielding sizable gains on multiple reasoning benchmarks.",{"@graph":69,"@context":122},[70,84,105],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/mammoth-building-math-generalist-models-through-hybrid-instruction-tuning/267782/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/mammoth-building-math-generalist-models-through-hybrid-instruction-tuning/267782.png","ImageObject",300,407,{"name":92,"@type":93},"Felix Montgomery","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-10-18","2026-09-14",true,{"@type":102,"interactionType":103,"userInteractionCount":44},"InteractionCounter",{"@type":104},"ViewAction",{"@type":106,"mainEntity":107},"FAQPage",[108,114,118],{"name":109,"@type":110,"acceptedAnswer":111},"What is MAMMOTH and what problem does it target?","Question",{"text":112,"@type":113},"MAMMOTH is a series of open-source LLMs tailored for general math problem-solving, aiming to strengthen mathematical reasoning performance for open models.","Answer",{"name":115,"@type":110,"acceptedAnswer":116},"How is MathInstruct constructed and what makes it different?",{"text":117,"@type":113},"MathInstruct is curated from 13 math datasets, incorporating intermediate rationales. It uniquely blends chain-of-thought and program-of-thought rationales while covering diverse math fields and complexity levels.",{"name":119,"@type":110,"acceptedAnswer":120},"Why combine CoT and PoT rationales in a single training approach?",{"text":121,"@type":113},"The hybrid design enables tool use benefits from program-style reasoning while keeping stepwise natural-language reasoning for more abstract scenarios, improving generalization across varied math tasks.","https://schema.org",{"og:url":83,"og:type":124,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":126,"canonical":83},"index,follow",{"doc_id":128,"site_id":62},267782,1789424882,{"code":4,"msg":5,"data":131},{"doc_id":128,"user_id":132,"nickname":92,"user_avatar":133,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":44,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":56,"language":139,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":140,"faqs":141,"seo_title":142,"seo_description":67,"update_tm":129,"read_time":143,"preview_page_count":4},549768064778,"https://ap-avatar.wpscdn.com/davatar_6f874abed73319feea01a86fa6f0fab8","arXiv :2309 .05653v 3 [ cs .CL] 3 Oct 2023  \nMAMMOTH: BUILDING MATH GENERALIST MODELS THROUGH HYBRID INSTRUCTION TUNING  \n♣Xiang Yue ∗ ,‡Xingwei Qu,†Ge Zhang, ○Yao Fu, §Wenhao Huang,♣Huan Sun, ♣Yu Su,†Wenhu Chen ∗  \n†University of Waterloo, ♣The Ohio State University,‡HKUST, ○University of Edinburgh, §01.AI [yue.149@osu.edu](yue.149@osu.edu) , [wenhuchen@uwaterloo.ca](wenhuchen@uwaterloo.ca)  \n[https://tiger-ai-lab.github.io/MAmmoTH/](https://tiger-ai-lab.github.io/MAmmoTH/)  \nABSTRACT  \nWe introduce MAmmoTH, a series of open-source large language models (LLMs) specifically tailored for general math problem-solving. The MAmmoTH models are trained on MathInstruct, our meticulously curated instruction tuning dataset.  \nMathInstruct is compiled from 13 math datasets with intermediate rationales, six of which have rationales newly curated by us. It presents a unique hybrid of chain-of-thought (CoT) and program-of-thought (PoT) rationales, and also ensures extensive coverage of diverse fields in math. The hybrid of CoT and PoT not only unleashes the potential of tool use but also allows different thought processes for different math problems. As a result, the MAmmoTH series substantially outperform existing open-source models on nine mathematical reasoning datasets across all scales with an average accuracy gain between 16% and 32% . Remarkably, our MAmmoTH-7B model reaches 33% on MATH (a competition-level dataset), which exceeds the best open-source 7B model (WizardMath) by 23%, and the MAmmoTH-34B model achieves 44% accuracy on MATH, even surpassing GPT- 4’s CoT result. Our work underscores the importance of diverse problem coverage and the use of hybrid rationales in developing superior math generalist models.  \nMathInstruct Hybrid Instruction Tuning  \nChain-of-Thought (CoT)  \nDiverse Math Problems  \nProgram-of-Thought (PoT)  \nMAmmoTH  \nhourly_rate = 12; time_worked = 50/60; earnings = hourly_rate * time_worked; print(round(earnings, 2))  \nIn-domain datasets Out-of-domain datasets  \nAccuracy (%)  \n80  \n60  \n40  \n20  \n0  \n7B 13B 30B 70B  \nAccuracy (%)  \n80  \n60  \n40  \n20  \n0  \n7B 13B 30B 70B  \nFigure 1: The superior performance of MAmmoTH, a series of models instruction-tuned to solve a diverse set of mathematical problems using hybrid CoT and PoT rationales. MAmmoTH significantly outperforms base and SoTA models on both in-domain and out-of-domain test sets, across all scales.  \n∗Xiang Yue and Wenhu Chen are the leading authors of the paper. They contributed equally to this project.  \n1 INTRODUCTION  \nThis work focuses on mathematical reasoning, a critical capability of modern large language models (LLMs) (OpenAI, 2023; Anil et al., 2023) . Despite the recent advances in this field, a noticeable gap exists between closed-source and open-source LLMs—closed-source models like GPT-4 (OpenAI, 2023), PaLM-2 (Anil et al., 2023), and Claude 2 (Bai et al., 2022) dominate popular mathematical reasoning benchmarks such as GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b), while open-source models like Llama (Touvron et al., 2023a;b), Falcon (Penedo et al., 2023), OPT (Zhang et al., 2022) lag behind on all benchmarks by a wide margin.  \nCurrent efforts to bridge this gap are twofold: (1) Continued pre-training like Galactica (Taylor et al., 2022) and MINERVA (Lewkowycz et al., 2022), which continues to train an LLM on math-related web data of more than 100B tokens. This approach improves a model’s general scientific reasoning capability but incurs a high computation cost. (2) Dataset-specific fine-tuning like rejection sampling fine-tuning (RFT) (Yuan et al., 2023) and WizardMath (Luo et al., 2023), which fine-tunes LLMs using supervised data specific to certain datasets. Although such approaches improve in-domain performance, they cannot generalize to a wider range of math reasoning tasks beyond their finetuning data. For instance, both RFT and WizardMath can increase the accuracy on GSM8K (Cobbeet al., 2021) by 30%+, one of t","cbCaioSRiuskXQ3u","https://ap.wps.com/l/cbCaioSRiuskXQ3u","pdf",1062653,"English","# Abstract\n## MAMMOTH\n## MathInstruct Hybrid Instruction Tuning\n## Chain-of-Thought (CoT)\n## Program-of-Thought (PoT)\n# Introduction\n## Gap between closed-source and open-source LLMs\n## Twofold prior efforts: continued pre-training and dataset-specific fine-tuning\n## Limits of pure CoT and pure PoT approaches","[{\"question\":\"What is MAMMOTH and what problem does it target?\",\"answer\":\"MAMMOTH is a series of open-source LLMs tailored for general math problem-solving, aiming to strengthen mathematical reasoning performance for open models.\"},{\"question\":\"How is MathInstruct constructed and what makes it different?\",\"answer\":\"MathInstruct is curated from 13 math datasets, incorporating intermediate rationales. It uniquely blends chain-of-thought and program-of-thought rationales while covering diverse math fields and complexity levels.\"},{\"question\":\"Why combine CoT and PoT rationales in a single training approach?\",\"answer\":\"The hybrid design enables tool use benefits from program-style reasoning while keeping stepwise natural-language reasoning for more abstract scenarios, improving generalization across varied math tasks.\"}]","MAMMOTH - Building Math Generalist Models Through Hybrid Instruction Tuning | PDF",48]