[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84305-en":3,"doc-seo-84305-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},84305,1374391974585,"Genevieve","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","What LLM Forecasters Know but Don’t Say: Probing Internal Representations for Calibration and Faithfulness","Large language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. Representation-pooling probes trained on intermediate activations for Eternis-Forecaster 8B improve calibration substantially on OpenForesight. The same calibration gains hold for GLM-4.7-Flash and GLM-4.5-Air. Evidence ablation and diversionary injection show partial CoT faithfulness and that probe activations track behavioral shifts better, predicting directions in 84% of cases. Forced answering further indicates forecasts are mostly fixed before reasoning begins.","arXiv :2607 .08046v 1 [ cs .CL] 9 Jul 2026  \nWhat LLM Forecasters Know but Don’t Say:  \nProbing Internal Representations for Calibration and Faithfulness  \nA UT H O R S  \nRaphaël Sarfati ∗,g, Pratyush Ranjan Tiwari ∗,e  \nSiddharth Boppana g , Christopher J. Earls g , Srikar Varadaraj e , Eric Ho g  \n∗ core contributors: [raphael@goodfire.ai](raphael@goodfire.ai), [pratyush@eternis.ai |](pratyush@eternis.ai | goodfire & eternis)[ g](pratyush@eternis.ai | goodfire & eternis)[oodfire &](pratyush@eternis.ai | goodfire & eternis)[ e](pratyush@eternis.ai | goodfire & eternis)[ternis](pratyush@eternis.ai | goodfire & eternis)  \nA B ST RA CT  \nLarge language models fine-tuned for forecasting can be accurate yet poorly calibrated, and their chain-of-thought (CoT) reasoning may not faithfully reflect the evidence behind a forecast. We ask whether internal representations offer a more direct window into both. Working with  Eternis-Forecaster 8B  on OpenForesight , we train representation-pooling probes on intermediate activations and find they achieve substantially better calibration; a result that also holds for  GLM-4 .7-Flash  and  GLM-4 .5-Air . We then assess CoT faithfulness through evidence ablation and diversionary injection: removing an influential source in the prompt often changes the model’s forecast while leaving the reasoning trace untouched. The same probes function as lie detectors: their activations track behavioral shifts far better than the reasoning trace does, and they also predict the direction of change in 84% of cases, including when the CoT conceals the perturbation’s influence. Finally, forced answering reveals that forecasts are largely fixed before reasoning begins: a single pre-reasoning pass recovers the committed answer and confidence, and routing questions by the spread of this pre-set answer distribution saves 30–47% of generated tokens, with no loss of accuracy. Together, these results establish probing internal representations as a practical tool for calibrating, auditing, and triaging language model forecasters and reasoning models more broadly.  \nTwo windows into aforecaster's confidence  \nEF-8B  \nwhat it says vs what it encodes  \n\n| 1. Probes calibrate confidence\u003Cbr>\u003Cbr>ECE 0.044 vs 0.093\u003Cbr>probe (ember) vs verbalized (slate) | 2. CoT is only partly faithful |  |\n| --- | --- | --- |\n|  | CoT change ↑ |  |\n|  | behavioral impact →\u003Cbr>ρ = 0.22\u003Cbr>forecast change vs reasoning change |  |\n\n3. Probes catch silent shifts  \nCoT  0.22  \nprobe  0.57 tracking of behavioral shift (Spearman ρ)  \ndirection correct 84%  \neven when the CoT is silent  \nContents  \n1 Introduction 4  \n2 Related Work 4  \n2.1 Time series extrapolation ...................................... 4  \n2.2 Open-ended forecasting ....................................... 5  \n2.3 Calibration-aware RL for reasoning ................................. 5  \n3 Methods 5  \n3.1 Models ................................................. 5  \n3.2 Dataset ................................................ 5  \n3.3 Probes ................................................. 6  \nMean-pooling ......................................... 6  \nAttention probes ....................................... 7  \nCovariance probes ...................................... 7  \n3.4 Metrics ................................................ 7  \n4 Experiments 8  \n4.1 Accuracy and Confidence ...................................... 8  \n4.1.1  EF-8B  is stable but overconfident across sampling temperatures ............ 8  \n4.1.2 Forecasts are most accurate when the gold answer is contained in the prompt ..... 8  \n4.2 Calibration of confidence ....................................... 9  \n4.2.1 Probing internal representations for correctness ...................... 9  \n4.2.2 Probe calibration surpasses verbalized confidence ..................... 9  \n4.3 Probe-only calibration on GLM models .............................. 10  \n4.4 Math reasoning calibration ..................................... 12  \n4.5 Ch","cbCaio7lA4f2zAeb","https://ap.wps.com/l/cbCaio7lA4f2zAeb","pdf",1470828,6,1,29,"English","en",105,"# Introduction\n# Related Work\n## Time series extrapolation\n## Open-ended forecasting\n## Calibration-aware RL for reasoning\n# Methods\n## Models\n## Dataset\n## Probes\n## Metrics\n# Experiments\n## Accuracy and Confidence\n## Calibration of confidence\n## Probe-only calibration on GLM models\n## Math reasoning calibration\n## Chain-of-Thought Faithfulness\n## Forced answering\n## Triaging questions by their pre-reasoning answer\n# Conclusion\n# Appendix","[{\"question\":\"How do internal-representation probes improve forecast calibration compared with verbalized confidence or CoT?\",\"answer\":\"Representation-pooling probes trained on intermediate activations achieve substantially better calibration than verbalized (reasoning-trace) confidence. Results are shown to hold for Eternis-Forecaster 8B and also transfer to GLM-4 variants.\"},{\"question\":\"Is chain-of-thought reasoning fully faithful to the evidence used for forecasts?\",\"answer\":\"No. Evidence ablation and diversionary injection indicate that removing influential prompt sources can change forecasts while leaving the reasoning trace largely unchanged, showing only partial CoT faithfulness.\"},{\"question\":\"How can probes act as lie detectors for silent changes in model behavior?\",\"answer\":\"Probe activations track behavioral shifts far better than the reasoning trace. The probe-based signals also predict the direction of behavioral change in 84% of cases, including scenarios where CoT conceals the perturbation’s influence.\"}]",1784194704,73,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"what-llm-forecasters-know-but-dont-say-probing-internal-representations-for-calibration-and-faithfulness","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/what-llm-forecasters-know-but-dont-say-probing-internal-representations-for-calibration-and-faithfulness/84305/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":24,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-07-25","2026-07-16",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"How do internal-representation probes improve forecast calibration compared with verbalized confidence or CoT?","Question",{"text":76,"@type":77},"Representation-pooling probes trained on intermediate activations achieve substantially better calibration than verbalized (reasoning-trace) confidence. Results are shown to hold for Eternis-Forecaster 8B and also transfer to GLM-4 variants.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"Is chain-of-thought reasoning fully faithful to the evidence used for forecasts?",{"text":81,"@type":77},"No. Evidence ablation and diversionary injection indicate that removing influential prompt sources can change forecasts while leaving the reasoning trace largely unchanged, showing only partial CoT faithfulness.",{"name":83,"@type":74,"acceptedAnswer":84},"How can probes act as lie detectors for silent changes in model behavior?",{"text":85,"@type":77},"Probe activations track behavioral shifts far better than the reasoning trace. The probe-based signals also predict the direction of behavioral change in 84% of cases, including scenarios where CoT conceals the perturbation’s influence.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":93},[94,98,102,106,111,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":107,"slug":138},19,"General","general"]