[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-0-en-105":3,"doc-seo-332113-105":59,"doc-detail-332113-en":131},{"code":4,"msg":5,"data":6},0,"success",[7,13,18,23,28,33,38,43,48,51,55],{"id":8,"doc_module":4,"doc_module_name":9,"category_name":10,"show_sort_weight":11,"slug":12},1,"Document","Story & Novel",90,"story-novel",{"id":14,"doc_module":4,"doc_module_name":9,"category_name":15,"show_sort_weight":16,"slug":17},2,"Literature",80,"literature",{"id":19,"doc_module":4,"doc_module_name":9,"category_name":20,"show_sort_weight":21,"slug":22},4,"Exam",70,"exam",{"id":24,"doc_module":4,"doc_module_name":9,"category_name":25,"show_sort_weight":26,"slug":27},5,"Comic",60,"comic",{"id":29,"doc_module":4,"doc_module_name":9,"category_name":30,"show_sort_weight":31,"slug":32},6,"Technology",50,"technology",{"id":34,"doc_module":4,"doc_module_name":9,"category_name":35,"show_sort_weight":36,"slug":37},7,"Healthcare",40,"healthcare",{"id":39,"doc_module":4,"doc_module_name":9,"category_name":40,"show_sort_weight":41,"slug":42},8,"Research & Report",30,"research-report",{"id":44,"doc_module":4,"doc_module_name":9,"category_name":45,"show_sort_weight":46,"slug":47},9,"Religion & Spirituality",20,"religion-spirituality",{"id":46,"doc_module":4,"doc_module_name":9,"category_name":49,"show_sort_weight":46,"slug":50},"World Cup","world-cup",{"id":52,"doc_module":4,"doc_module_name":9,"category_name":53,"show_sort_weight":52,"slug":54},10,"Lifestyle","lifestyle",{"id":56,"doc_module":4,"doc_module_name":9,"category_name":57,"show_sort_weight":24,"slug":58},19,"General","general",{"code":4,"msg":60,"data":61},"ok",{"site_id":62,"language":63,"slug":64,"title":65,"keywords":66,"description":67,"schema_data":68,"social_meta":124,"head_meta":126,"extra_data":128,"updated_unix":130},105,"en","evaluating-large-language-models-on-multimodal-chemistry-olympiad-exams","Evaluating Large Language Models on Multimodal Chemistry Olympiad Exams","","Multimodal scientific reasoning remains a major challenge for large language models (LLMs), especially in chemistry, where solutions depend on symbolic diagrams, molecular structures, and structured visual data. This work systematically evaluates 40 multimodal LLMs across a curated benchmark of Olympiad-style chemistry questions sourced from over two decades of U.S. National Chemistry Olympiad (USNCO) exams. Results show many models struggle with vision-language modality fusion, with some cases where removing images improves accuracy. Chain-of-Thought prompting improves accuracy and visual grounding, supported by ablation and occlusion-based interpretability. The study highlights current multimodal reasoning limitations and provides strategies for building more robust and interpretable systems for chemistry.",{"@graph":69,"@context":123},[70,84,106],{"@type":71,"itemListElement":72},"BreadcrumbList",[73,77,79,82],{"item":74,"name":75,"@type":76,"position":8},"https://docshare.wps.com","Home","ListItem",{"item":78,"name":9,"@type":76,"position":14},"https://docshare.wps.com/document/",{"item":80,"name":40,"@type":76,"position":81},"https://docshare.wps.com/document/research-report/",3,{"item":83,"name":65,"@type":76,"position":19},"https://docshare.wps.com/document/evaluating-large-language-models-on-multimodal-chemistry-olympiad-exams/332113/",{"url":83,"name":65,"@type":85,"image":86,"author":91,"headline":65,"publisher":94,"fileFormat":97,"inLanguage":63,"description":67,"dateModified":98,"datePublished":99,"encodingFormat":97,"isAccessibleForFree":100,"interactionStatistic":101},"DigitalDocument",{"url":87,"@type":88,"width":89,"height":90},"https://docshare.wps.com/thumbnails/evaluating-large-language-models-on-multimodal-chemistry-olympiad-exams/332113.png","ImageObject",300,407,{"name":92,"@type":93},"Bintang","Person",{"url":74,"name":95,"@type":96},"DocShare","Organization","application/pdf","2026-11-10","2026-09-21",true,{"@type":102,"interactionType":103,"userInteractionCount":105},"InteractionCounter",{"@type":104},"ViewAction",15,{"@type":107,"mainEntity":108},"FAQPage",[109,115,119],{"name":110,"@type":111,"acceptedAnswer":112},"What benchmark and question source are used to evaluate multimodal LLMs?","Question",{"text":113,"@type":114},"The evaluation uses a curated benchmark of Olympiad-style chemistry questions drawn from over two decades of U.S. National Chemistry Olympiad (USNCO) exams.","Answer",{"name":116,"@type":111,"acceptedAnswer":117},"How do the results characterize modality fusion in current multimodal LLMs?",{"text":118,"@type":114},"Many models struggle with modality fusion; in some cases, removing the image improves accuracy, suggesting misalignment in vision-language integration.",{"name":120,"@type":111,"acceptedAnswer":121},"What effect does Chain-of-Thought prompting have in this study?",{"text":122,"@type":114},"Chain-of-Thought prompting consistently enhances both accuracy and visual grounding, supported by ablation studies and occlusion-based interpretability.","https://schema.org",{"og:url":83,"og:type":125,"og:title":65,"og:site_name":95,"og:description":67},"article",{"robots":127,"canonical":83},"index,follow",{"doc_id":129,"site_id":62},332113,1790153433,{"code":4,"msg":5,"data":132},{"doc_id":129,"user_id":133,"nickname":92,"user_avatar":134,"doc_module":4,"category_id":39,"category_name":40,"doc_title":65,"doc_description":67,"doc_content":135,"file_id":136,"file_url":137,"file_type":138,"file_size":139,"view_count":105,"is_deleted":4,"is_public":8,"is_downloadable":8,"audit_status":8,"page_count":105,"language":140,"language_code":63,"site_id":62,"html_lang":63,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":67,"update_tm":144,"read_time":145,"preview_page_count":4},962085564381,"https://ap-avatar.wpscdn.com/davatar_6f874abed73319feea01a86fa6f0fab8","arXiv :2512 . 14989v1 [ cs .CL] 17 Dec 2025  \nEvaluating Large Language Models on Multimodal Chemistry Olympiad Exams  \nYiming Cui 1,2* , Xin Yao 1,2 , Yuxuan Qin2 , Xin Li 1,2 , Shijin Wang 1,2* , and Guoping Hu 1,2  \n1 State Key Laboratory of Cognitive Intelligence, Hefei, China  \n2 iFLYTEK AI Research, Beijing, China  \n* [ymcui@ieee.org](ymcui@ieee.org) , [sjwang3@iflytek.com](sjwang3@iflytek.com)  \nABSTRACT  \nMultimodal scientific reasoning remains a significant challenge for large language models (LLMs), particularly in chemistry, where problem-solving relies on symbolic diagrams, molecular structures, and structured visual data. Here, we systematically evaluate 40 proprietary and open-source multimodal LLMs, including GPT-5, o3, Gemini-2.5-Pro, and Qwen2.5-VL, on a curated benchmark of Olympiad-style chemistry questions drawn from over two decades of U.S. National Chemistry Olympiad (USNCO) exams. These questions require integrated visual and textual reasoning across diverse modalities. We find that many models struggle with modality fusion, where in some cases, removing the image even improves accuracy, indicating misalignment in vision-language integration. Chain-of-Thought prompting consistently enhances both accuracy and visual grounding, as demonstrated through ablation studies and occlusionbased interpretability. Our results reveal critical limitations in the scientific reasoning abilities of current MLLMs, providing actionable strategies for developing more robust and interpretable multimodal systems in chemistry. This work provides a timely benchmark for measuring progress in domain-specific multimodal AI and underscores the need for further advances at the intersection of artificial intelligence and scientific reasoning.  \nMultimodal reasoning plays a central role in many scientific domains, where interpreting structured visual information such as diagrams, equations, and charts is essential for problem-solving. Among these, chemistry presents unique challenges due to its reliance on symbolic representations, molecular structures, reaction mechanisms, and experimental data visualizations. As large language models (LLMs) evolve into increasingly capable multimodal LLMs (MLLMs) 1–5 , their ability to reason in such scientifically structured settings remains poorly understood, especially when compared to recent advances in natural image understanding.  \nRecent years have witnessed significant progress in visuallanguage tasks, including image captioning6 , visual question answering (VQA)7 , and document understanding8. Benchmarks like ScienceQA9 and MMSci 10 have helped evaluate models in STEM contexts. However, they often focus on physics, math, or general science education. Chemistry, with its highly structured and symbolic multimodal representations, remains underrepresented in this context. OlympiadBench 11 covers math and physics but omits chemistry, and GPQA 12 assesses graduate-level science knowledge without a visual component.  \nChemistry problems are especially challenging for multimodal models due to the abstract and compressed nature of their visual artifacts. Problems may involve Lewis structures, titration curves, electrochemical setups, or molecular conformations, requiring models to understand spatial and symbolic relationships rather than natural scenes. Unlike standard VQA benchmarks, chemistry visuals utilize domain-specific nota-  \n* Pre-print version. Check published version at: [https://www](https://www) . [nature.com/articles/s42004-025-01782-x](nature.com/articles/s42004-025-01782-x)  \ntion and require a deeper integration of visual and textual signals.  \nMost existing chemistry datasets emphasize perceptionlevel tasks such as chemical structure recognition 13–15 or reaction parsing 16 , with relatively fewer efforts directed at integrative and comprehensive reasoning. Some attempts, such as Chemma-RC 17 and ChemistryQA 18 , extend beyond single-modality recognition toward text-graph integrat","cbCaig0DWOhuKsqC","https://ap.wps.com/l/cbCaig0DWOhuKsqC","pdf",3575967,"English","# ABSTRACT\n## Problem scope and motivation\n## Related benchmarks and gaps in chemistry\n## Challenges of chemistry visuals for multimodal models\n## Existing chemistry datasets and multimodal systems","[{\"question\":\"What benchmark and question source are used to evaluate multimodal LLMs?\",\"answer\":\"The evaluation uses a curated benchmark of Olympiad-style chemistry questions drawn from over two decades of U.S. National Chemistry Olympiad (USNCO) exams.\"},{\"question\":\"How do the results characterize modality fusion in current multimodal LLMs?\",\"answer\":\"Many models struggle with modality fusion; in some cases, removing the image improves accuracy, suggesting misalignment in vision-language integration.\"},{\"question\":\"What effect does Chain-of-Thought prompting have in this study?\",\"answer\":\"Chain-of-Thought prompting consistently enhances both accuracy and visual grounding, supported by ablation studies and occlusion-based interpretability.\"}]","Evaluating Large Language Models on Multimodal Chemistry Olympiad Exams | PDF",1789996838,38]