[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-569328-105":53,"doc-detail-569328-en":127},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":120,"head_meta":122,"extra_data":124,"updated_unix":126},105,"en","can-large-language-models-do-analytical-reasoning","Can Large Language Models do Analytical Reasoning?","","This paper investigates whether large language models can perform analytical reasoning in a sports setting. It evaluates their ability to count quarter scores in NBA and NFL games using play-by-play data from ESPN, withholding final totals from the models. Results show GPT-4 as most effective, followed by Claude-2.1, while GPT-3.5, Gemini-Pro, and Llama-2 lag behind. Prompting comparisons indicate divide-and-conquer improves performance, whereas chain-of-thought helps only selected models. Task complexity is linked to context length, information density, and related information.",{"@graph":63,"@context":119},[64,80,102],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":51,"@type":70,"position":76},"https://docshare.wps.com/template/general/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/can-large-language-models-do-analytical-reasoning/569328/",4,{"url":78,"name":59,"@type":81,"image":82,"author":87,"headline":59,"publisher":90,"fileFormat":93,"inLanguage":57,"description":61,"dateModified":94,"datePublished":95,"encodingFormat":93,"isAccessibleForFree":96,"interactionStatistic":97},"DigitalDocument",{"url":83,"@type":84,"width":85,"height":86},"https://docshare.wps.com/thumbnails/can-large-language-models-do-analytical-reasoning/569328.png","ImageObject",442,249,{"name":88,"@type":89},"Fans","Person",{"url":68,"name":91,"@type":92},"DocShare","Organization","application/pdf","2026-10-27","2026-10-13",true,{"@type":98,"interactionType":99,"userInteractionCount":101},"InteractionCounter",{"@type":100},"ViewAction",6,{"@type":103,"mainEntity":104},"FAQPage",[105,111,115],{"name":106,"@type":107,"acceptedAnswer":108},"What analytical reasoning task do the authors test on large language models?","Question",{"text":109,"@type":110},"They test whether models can count the total points each team scores within a specific quarter using play-by-play descriptions from NBA and NFL games.","Answer",{"name":112,"@type":107,"acceptedAnswer":113},"Which models perform best in this quarter-scoring task?",{"text":114,"@type":110},"GPT-4 is reported as the top performer with the highest accuracy, followed by Claude-2.1, while GPT-3.5, Gemini-Pro, and Llama-2-70b lag behind.",{"name":116,"@type":107,"acceptedAnswer":117},"How do divide-and-conquer and chain-of-thought affect model accuracy?",{"text":118,"@type":110},"Divide-and-conquer is found to be the most effective prompting approach. Chain-of-thought improves outcomes for some models like GPT-4 and Claude-2.1, but has negligible or even harmful effects for others such as GPT-3.5 and Gemini-Pro.","https://schema.org",{"og:url":78,"og:type":121,"og:title":59,"og:site_name":91,"og:description":61},"article",{"robots":123,"canonical":78},"index,follow",{"doc_id":125,"site_id":56},569328,1792190130,{"code":4,"msg":5,"data":128},{"doc_id":125,"user_id":129,"nickname":88,"user_avatar":130,"doc_module":9,"category_id":50,"category_name":51,"doc_title":59,"doc_description":61,"doc_content":131,"file_id":132,"file_url":133,"file_type":134,"file_size":135,"view_count":101,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":20,"language":136,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":137,"faqs":138,"seo_title":139,"seo_description":61,"update_tm":140,"read_time":47},5909892330395,"https://ap-avatar.wpscdn.com/davatar_6f874abed73319feea01a86fa6f0fab8","arXiv :2403 .0403 1v 1 [ cs .CL] 6 Mar 2024  \nCan Large Language Models do Analytical Reasoning?  \nYebowen Hu⋆, Kaiqiang Song†, Sangwoo Cho†, Xiaoyang Wang† Hassan Foroosh⋆, Dong Yu†, Fei Liu⋄  \n⋆ University of Central Florida †Tencent AI Lab, Bellevue, WA ⋄ Emory University {[yebowen.hu](yebowen.hu) , [hassan.foroosh](hassan.foroosh}@ucf.edu)[}](hassan.foroosh}@ucf.edu)[@ucf.edu](hassan.foroosh}@ucf.edu)  \n{riversong, swcho, shawnxywang, [dyu](dyu}@global.tencent.com)[}](dyu}@global.tencent.com)[@global.tencent.com](dyu}@global.tencent.com)[ ](dyu}@global.tencent.com)[fei.liu@emory.edu](fei.liu@emory.edu)  \nAbstract  \nThis paper explores the cutting-edge Large Language Model with analytical reasoning on sports. Our analytical reasoning embodies the tasks of letting large language models count how many points each team scores ina quarter in the NBA and NFL games. Our major discoveries are in two folds. Firstly, we find among all the models we employed, GPT-4 stands out in effectiveness, followed by Claude-2.1, with GPT-3.5, Gemini-Pro, and Llama-2-70b lagging behind. Specifically, we compare three different prompting techniques and a divide-and-conquer approach, we find that the latter was the most effective. Our divide-and-conquer approach breaks down play-by-play data into smaller, more manageable segments, solves each piece individually, and then aggregates them together. Besides the divide-and-conquer approach, we also explore the Chain of Thought (CoT) strategy, which markedly improves outcomes for certain models, notably GPT-4 and Claude-2.1, with their accuracy rates increasing significantly.  \nHowever, the CoT strategy has negligible or even detrimental effects on the performance of other models like GPT-3.5 and Gemini-Pro. Secondly, to our surprise, we observe that most models, including GPT-4, struggle to accurately count the total scores for NBA quarters despite showing strong performance in counting NFL quarter scores. This leads us to further investigate the factors that impact the complexity of analytical reasoning tasks with extensive experiments, through which we conclude that task complexity depends on the length of context, the information density, and the presence of related information. Our research provides valuable insights into the complexity of analytical reasoning tasks and potential directions for developing future large language models.  \n1 Introduction  \nIn recent years, advancements in large language models have been remarkable, with OpenAI’s ChatGPT (OpenAI, 2023a) leading the way in demonstrating exceptional capabilities across a broad spectrum of natural language processing tasks. These tasks encompass information extraction (Lu et al., 2023; Liu et al., 2023a), question answering (Tonmoy et al., 2024; Tan et al., 2023), natural language understanding (Mao et al., 2023; Liu et al., 2023b), text generation (Sun et al., 2024; Zhao et al., 2023), translation (Fan et al., 2020; Wu et al., 2024), and summarization (Liu et al., 2023c; Hu et al., 2023) . Additionally, ChatGPT exhibits a profound capacity to manipulate other data formats, including tables and code. Competing models, such as Anthropic’s Claude (Anthropic, 2023), Google’s Gemini (Team, 2023), Meta’s Llama (Hugo et al., 2023), and Mistral’s Mixtral (MistralAI, 2023) have also displayed performance levels that are on par with benchmarks, showcasing their significant contributions to the field. However, such models remain imperfect as they lack some tasks like math (Zhu et al., 2023; Cobbe et al., 2021; Hendrycks et al., 2021) and complex reasoning (Masry et al., 2022; Chen et al., 2022) . To evaluate the capabilities of recent cutting-edge language models in interpreting and analyzing sports statistics, our research focused on examining play-by-play data from the NBA and NFL, as provided by ESPN. The primary  \nFigure 1: Play-by-play descriptions of an NBA game, including timestamps, play-by-play descriptions, team affiliations and total poi","cbCaieYfrmwUXp5W","https://ap.wps.com/l/cbCaieYfrmwUXp5W","pdf",928377,"English","# Introduction\n## Analytical reasoning task design\n## Model evaluation and prompting strategies","[{\"question\":\"What analytical reasoning task do the authors test on large language models?\",\"answer\":\"They test whether models can count the total points each team scores within a specific quarter using play-by-play descriptions from NBA and NFL games.\"},{\"question\":\"Which models perform best in this quarter-scoring task?\",\"answer\":\"GPT-4 is reported as the top performer with the highest accuracy, followed by Claude-2.1, while GPT-3.5, Gemini-Pro, and Llama-2-70b lag behind.\"},{\"question\":\"How do divide-and-conquer and chain-of-thought affect model accuracy?\",\"answer\":\"Divide-and-conquer is found to be the most effective prompting approach. Chain-of-thought improves outcomes for some models like GPT-4 and Claude-2.1, but has negligible or even harmful effects for others such as GPT-3.5 and Gemini-Pro.\"}]","Can Large Language Models do Analytical Reasoning? | PDF",1791875216]