[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82930-en":3,"doc-seo-82930-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82930,8796095461610,"Oliver","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","How Much is Left LLMs Linearly Encode Their Remaining Output Length","Large language models generate tokens sequentially, yet responses exhibit stable length patterns: multi-step solutions converge to predictable token counts, retrieval snippets end after a few sentences, and retractions extend outputs by measurable amounts. This work tests whether models internally estimate remaining response length. Minimal linear probes trained on frozen hidden states of three open-weight 7–8B LLMs across multiple completion datasets show linear decodability from the prompt’s last hidden state, broad transfer in one probe direction, and probe shifts synchronized with retraction-and-restart behavior.","arXiv :2607 .053 16v 1 [ cs .CL] 6 Jul 2026  \nHow Much is Left? LLMs Linearly Encode Their Remaining Output Length  \nMohamed Amine Merzouk1,2 Dmitri Carpov3 Mirko Bronzi3 Damiano Fornasiere3  \nAdam Oberman2,3  \n1Mila, Quebec AI Institute 2McGill University 3LawZero  \nAbstract  \nLarge language models generate one token at a time, yet their responses show remarkably consistent length structure: step-by-step solutions converge in predictable token counts, retrievals stop after a few sentences, retractions extend responses by measurable amounts. We ask whether the model carries an internal estimate of how much response remains. Training minimal-capacity linear probes on frozen hidden states of three open-weight 7-8B models across seven completion-style datasets, we find three converging pieces of evidence. First, total response length is linearly decodable from the prompt’s last hidden state alone, before any output is emitted.  \nSecond, probe directions trained on natural-language datasets transfer broadly, including to controlled synthetic completions never seen in training, outperforming a statistical baseline; the converse direction generally fails, and this asymmetry is itself informative. Third, on curated high-loss completions, the probe’s per-position estimate shifts upward at the moment the model retracts and restarts a partial solution, a directional behavior no position-only predictor can reproduce (we note in §4.3 that this is qualitative, not aggregate) . We frame this as approximate estimation of remaining generation length, distinct from exact-counting impossibility results for transformers, and interpret it as evidence that LLMs maintain a plan-like internal representation of output length (decodable, not necessarily used causally) .  \nCode: [https://anonymous.4open.science/r/llm-output-length](https://anonymous.4open.science/r/llm-output-length)  \n1 Introduction  \nWhen a large language model produces a step-by-step solution, retrieves a fact, or writes a paragraph, the result has a length: a number of tokens emitted before an end-of-sequence (EOS) token. That length is often surprisingly predictable from the prompt alone. Asked to solve a grade-school arithmetic problem, current LLMs tend to produce three-to-five lines of working before the answer; asked to retrieve a date, they produce a single short clause. This consistency is at odds with the standard description of how an autoregressive model computes: each token is sampled conditioned on the prefix, with no explicit notion of total response length anywhere in the computation graph. This paper asks whether that consistency is an artifact of decoding (token-by-token sampling that happens to terminate at similar lengths) or whether the model’s intermediate representations encode an estimate of how much response remains. The distinction matters: in the first case, length is a downstream statistical regularity of the conditional distribution; in the second, the model carries an internal variable for “remaining work,” informally a plan. Recent mechanistic work shows that LLMs plan ahead over content, e.g., committing to a rhyme word several positions before writing the line that ends in it [Lindsey et al., 2025]; we ask the analogous question for length.  \nWe attack this question with linear probing [Alain and Bengio, 2018, Belinkov, 2021] . For a frozen LLM and a (prompt, completion) pair (x, y) with completion length T, we extract the residual-stream hidden states ht at every position t and train minimal-capacity linear probes to predict the remaining  \nPreprint.  \ntoken count rt = T − t. The use of a linear probe is deliberate: any signal we recover is information that is already linearly available in the hidden state, not a result of the probe’s own computation. We compare against a constant statistical baseline (the train-split median of rt , the optimal constant predictor under L 1 loss) and against a length-minus-position predictor seeded with the model’s ow","cbCaiiaEsL74kd86","https://ap.wps.com/l/cbCaiiaEsL74kd86","pdf",2217992,1,18,"English","en",105,"# Introduction\n## Problem setup and motivation\n## Method: linear probing on residual stream\n## Baselines and evaluation strategy\n## Qualitative behavior and practical applications\n## Contributions","[{\"question\":\"What question does the paper ask about LLMs and output length?\",\"answer\":\"It asks whether the consistency of response length reflects a hidden internal estimate of how much output remains, rather than only a statistical regularity from token-by-token decoding.\"},{\"question\":\"How do the authors test whether remaining length is encoded?\",\"answer\":\"They train minimal-capacity linear probes on frozen hidden states to predict remaining token counts from residual-stream states across multiple completion datasets.\"},{\"question\":\"What key evidence supports the remaining-length encoding claim?\",\"answer\":\"Total response length is linearly decodable from the prompt’s last hidden state, probe directions trained on natural datasets transfer broadly, and the probe estimate shifts upward at retraction and restart moments in curated high-loss completions.\"}]",1784184066,45,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"how-much-is-left-llms-linearly-encode-their-remaining-output-length","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/how-much-is-left-llms-linearly-encode-their-remaining-output-length/82930/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What question does the paper ask about LLMs and output length?","Question",{"text":74,"@type":75},"It asks whether the consistency of response length reflects a hidden internal estimate of how much output remains, rather than only a statistical regularity from token-by-token decoding.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How do the authors test whether remaining length is encoded?",{"text":79,"@type":75},"They train minimal-capacity linear probes on frozen hidden states to predict remaining token counts from residual-stream states across multiple completion datasets.",{"name":81,"@type":72,"acceptedAnswer":82},"What key evidence supports the remaining-length encoding claim?",{"text":83,"@type":75},"Total response length is linearly decodable from the prompt’s last hidden state, probe directions trained on natural datasets transfer broadly, and the probe estimate shifts upward at retraction and restart moments in curated high-loss completions.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]