[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-564386-105":53,"doc-detail-564386-en":120},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":113,"head_meta":115,"extra_data":117,"updated_unix":119},105,"en","llm-as-a-prophet-understanding-predictive-intelligence-with-prophet-arena-paper","LLM-AS-A-PROPHET: UNDERSTANDING PREDICTIVE INTELLIGENCE WITH PROPHET ARENA - Paper","","Large language models trained on abundant data face persistent evaluation difficulties, including data contamination and benchmark overfitting. This work studies a forecasting-based benchmarking paradigm, “LLM-as-a-Prophet,” where models predict real-world future events using capabilities that must remain robust to contamination. Prophet Arena is introduced as a continuously updated evaluation benchmark that decomposes live forecasting tasks into pipeline stages for controlled experiments. Results show strong calibration, confidence consistency, and promising returns, while revealing bottlenecks such as poor event recall and slower information aggregation near resolution.",{"@graph":63,"@context":112},[64,80,95],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":46,"@type":70,"position":76},"https://docshare.wps.com/template/paper-templates/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/llm-as-a-prophet-understanding-predictive-intelligence-with-prophet-arena-paper/564386/",4,{"url":78,"name":59,"@type":81,"author":82,"headline":59,"publisher":85,"fileFormat":88,"inLanguage":57,"description":61,"dateModified":89,"datePublished":89,"encodingFormat":88,"isAccessibleForFree":90,"interactionStatistic":91},"DigitalDocument",{"name":83,"@type":84},"Kyle","Person",{"url":68,"name":86,"@type":87},"DocShare","Organization","application/pdf","2026-10-12",true,{"@type":92,"interactionType":93,"userInteractionCount":4},"InteractionCounter",{"@type":94},"ViewAction",{"@type":96,"mainEntity":97},"FAQPage",[98,104,108],{"name":99,"@type":100,"acceptedAnswer":101},"Why is evaluating LLM intelligence difficult in traditional benchmarks?","Question",{"text":102,"@type":103},"Because training-data contamination and benchmark overfitting can make results unreliable, motivating evaluation methods that remain free from these issues.","Answer",{"name":105,"@type":100,"acceptedAnswer":106},"What does “LLM-as-a-Prophet” mean in this paper?",{"text":107,"@type":103},"It treats forecasting of real-world future events as a rigorous way to measure whether AI systems can reliably connect present information to future outcomes.",{"name":109,"@type":100,"acceptedAnswer":110},"What is Prophet Arena and how does it evaluate models?",{"text":111,"@type":103},"Prophet Arena is a general, continuously collecting benchmark for live forecasting tasks, decomposing each task into pipeline stages to enable controlled large-scale experimentation.","https://schema.org",{"og:url":78,"og:type":114,"og:title":59,"og:site_name":86,"og:description":61},"article",{"robots":116,"canonical":78},"index,follow",{"doc_id":118,"site_id":56},564386,1791841874,{"code":4,"msg":5,"data":121},{"doc_id":118,"user_id":122,"nickname":83,"user_avatar":123,"doc_module":9,"category_id":45,"category_name":46,"doc_title":59,"doc_description":61,"doc_content":124,"file_id":125,"file_url":126,"file_type":127,"file_size":128,"view_count":4,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":129,"language":130,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":131,"faqs":132,"seo_title":133,"seo_description":61,"update_tm":119,"read_time":134},3985741905716,"https://ap-avatar.wpscdn.com/davatar_994ba38a5ba835b3df7d355c54d3ed8d","LLM-AS-A-PROPHET: UNDERSTANDING PREDICTIVE INTELLIGENCE WITH PROPHET ARENA  ∗  \nQingchuan Yang 1†, Simon Mahns 2†, Sida Li 3†, Anri Gu 3†, Jibang Wu 4‡, Haifeng Xu 3‡  \n1 University of Southern California 2 Meta 3 The University of Chicago 4 New York University  \nABSTRACT  \nWith the rapid progress of large language models (LLMs) trained on every available piece of data, it becomes increasingly challenging to reliably evaluate their intelligence due to potential data contamination and benchmark overfitting. To overcome these challenges, we investigate a new angle of benchmarking LLMs’intelligence by evaluating their capability in forecasting real-world future events, a paradigm we call “LLM-as-a-Prophet”. Such forecasting tasks require combination of sophisticated capabilities while remaining free from data contamination or overfitting. To systematically evaluate such predictive intelligence of LLMs, we introduce Prophet Arena, a general evaluation benchmark that continuously collects live forecasting tasks and decomposes each task into distinct pipeline stages, supporting our controlled and large-scale experimentation. Our comprehensive evaluation reveals that many LLMs already exhibit impressive forecasting capabilities, reflected in, e.g., their small calibration errors, consistent prediction confidence and promising market returns. However, we also uncover key bottlenecks even in frontier models, such as inaccurate event recalls, misunderstanding of data sources and slower information aggregation compared to markets when resolution nears.  \n1 INTRODUCTION  \nForecasting is a fundamental intellectual pursuit that has shaped human progress from the earliest scientific inquiries to modern economics and finance. In machine learning, forecasting has also been a central theme, with rich traditions ranging from time-series analysis (Chatfield, 2000) and online learning (Foster & Vohra, 1998) to conformal prediction (Barber et al., 2023) . Yet, somewhat surprisingly, the challenge of open-domain forecasting, producing accurate predictions across a wide range of topics without domain-specific tuning or specialized datasets, remains largely unexplored. Achieving reliable foresight in this setting would represent a qualitative leap in AI capability, with far-reaching societal implications, from enhancing market efficiency to guiding high-stakes policy decisions (Arrow et al., 2008) .  \nAt its core, forecasting is the process of connecting present knowledge to anticipate future outcomes. Large language models (LLMs) seem natural candidates for this role. Trained on massive corpora of human knowledge through the seemingly narrow objective of next-word prediction, LLMs have developed emergent capabilities that extend far beyond their training objective (Bubeck et al., 2023) . This motivates the prospect that the ability to predict the next word may also give rise to the ability to predict the next event. Indeed, recent works by Zou et al. (2022); Halawi et al. (2024) have already made promising progress in employing language models for forecasting tasks. We envision that, with growing interest and continued progress, this would position LLMs not only as repositories of human knowledge but also as instruments of reliable foresight, leading to the prospect of LLMas-a-Prophet: Can AI systems reliably predict the future by connecting the dots across existing real-world information?  \nIn this paper, we seek to systematically examine the prospects and challenges of building general  \npurpose systems for open-domain forecasting. On one hand, forecasting is a natural next pursuit ∗ Correspondence to contact@prophetarena .co.  \nPart of the work by Qingchuan Yang and Jibang Wu are done while they are at the University of Chicago. We thank Alex Gu, Chaplin Huang and Lucien Liu for help at the early stage of this work.  \n†Equal contribution.  \n‡Equal advising.  \ngiven the rapid progress of AI, as it draws on a combination of advanced capabilities tha","cbCaijcevJEr6Y61","https://ap.wps.com/l/cbCaijcevJEr6Y61","pdf",2449802,53,"English","# Abstract\n# Introduction","[{\"question\":\"Why is evaluating LLM intelligence difficult in traditional benchmarks?\",\"answer\":\"Because training-data contamination and benchmark overfitting can make results unreliable, motivating evaluation methods that remain free from these issues.\"},{\"question\":\"What does “LLM-as-a-Prophet” mean in this paper?\",\"answer\":\"It treats forecasting of real-world future events as a rigorous way to measure whether AI systems can reliably connect present information to future outcomes.\"},{\"question\":\"What is Prophet Arena and how does it evaluate models?\",\"answer\":\"Prophet Arena is a general, continuously collecting benchmark for live forecasting tasks, decomposing each task into pipeline stages to enable controlled large-scale experimentation.\"}]","LLM-AS-A-PROPHET: UNDERSTANDING PREDICTIVE INTELLIGENCE WITH PROPHET ARENA - Paper | PDF",19]