[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82557-en":3,"doc-seo-82557-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},82557,962075006959,"Anda","https://ap-avatar.wpscdn.com/avatar/e0002397efbe92a78e?_k=1776741047341049297",8,"Research & Report","Recovering Input Text from Hidden States Study of Gradient-Based Inversion of Decoder-Only Language Models","This paper studies hidden-state inversion for decoder-only language models, aiming to recover the original input token sequence from last-layer hidden states. Instead of one-shot reconstruction, it treats inversion as continuous embedding-space optimization using a soft proxy driven toward the leaked target without hard-token projection during search, committing tokens only at the end of the inner loop. Continuous optimization exposes internal signals such as rank trajectories and per-position loss curves, while a discrete commit-time loss enables cumulative failure detection. Results on 10-token C4 prompts show exact-match improving from 66.9% to 97.5% as the candidate window widens, indicating most errors are near-misses rather than true ambiguities.","arXiv :2607 .00852v 1 [ cs .CL] 1 Jul 2026  \nRecovering Input Text from Hidden States: Study of Gradient-Based Inversion of Decoder-Only Language Models  \nMikołaj Słowikowski 1 and Maciej Witold Majewski 1  \nAGH University of Krakow, Faculty of Physics and Applied Computer Science, al. Mickiewicza 30, 30-059 Krakow, Poland [mmajewski@agh.edu.pl](mmajewski@agh.edu.pl) , [mikslo@student.agh.edu.pl](mikslo@student.agh.edu.pl)  \nAbstract. This work studies the hidden-state inversion problem: recovering the original input token sequence of a decoder-only language model from its last-layer hidden states. Rather than treating inversion as a one-shot reconstruction, we study it as a continuous embeddingspace optimisation in which a soft proxy is driven towards the leaked target without any hard-token projection during the search, and a token is committed only once, at the end of the inner loop. This design choice has two consequences which are the main focus of this paper.  \nFirst, keeping the optimisation entirely in continuous space exposes a rich set of internal signals—rank trajectories of the ground-truth token, per-position loss curves, and a discrete loss measured at commit time.  \nSecond, the discrete loss yields allows to asses the correctness of recovery via cumulative discrete loss. We further analyse which tokens break thereconstructions and find a sharp categorical asymmetry: space-prefixed, high-frequency function words in dense regions of the embedding matrix dominate the failures, while content-bearing tokens are recovered almost perfectly. On 10-token C4 prompts the exact-match rate rises from 66.9% to 97.5%(mean similarity 0.994) as the candidate window is widened, confirming that most errors are recoverable near-misses rather than genuine ambiguities. A comparison with the released SIPIT [1] reference situates these findings: per-step hard projection is faster, but the continuous formulation is what makes the optimisation observable and its failures detectable. The results show that last-layer hidden states of GPT-2 are as sensitive as the original text.  \nKeywords: Privacy attacks · Embedding inversion · Large language models · Hidden-state inversion · Failure detection · Trustworthy AI.  \n1 Introduction  \nLarge language models (LLMs) have become central infrastructure for a growing range of applications, and their deployment increasingly moves beyond a singlemachine setting. A client device may run the early layers of a model locally and send only the resulting activations to a remote server that completes the  \n2 M. Słowikowski and M.W. Majewski  \ncomputation; participants in a federated pipeline exchange gradient updates that carry information about their private inputs; retrieval systems precompute and cache hidden-state representations of sensitive documents for later reuse. In each of these scenarios raw text never leaves the trusted perimeter—only tensors of floating-point numbers do.  \nThis separation between text and its neural representation is typically treated as a privacy boundary, under the assumption that a hidden-state vector is an opaque, high-dimensional encoding from which the original words cannot be recovered. This assumption underpins split-inference protocols [12], embedding APIs that return dense vectors without the original query [10], and federated fine-tuning that transmits gradient tensors carrying implicit information about training examples [13,16] . If the assumption of irreversibility holds, an adversary who intercepts the tensors learns nothing about the text; if it does not, the entire privacy argument collapses.  \nRecent theoretical work shows that the assumption is in fact incorrect for decoder-only transformer models under mild conditions. The SIPIT result [1] establishes that the mapping from a token sequence to its last-layer hidden states is injective: no two distinct input sequences produce the same sequence of hidden-state vectors. Injectivity implies that the hidden states","cbCaibwNDKpoCkz7","https://ap.wps.com/l/cbCaibwNDKpoCkz7","pdf",2154993,4,1,16,"English","en",105,"# Abstract\n# Introduction\n## Problem Statement\n## Injectivity and Solvability","[{\"question\":\"What problem does the paper address?\",\"answer\":\"The paper addresses hidden-state inversion: recovering the original input token sequence of a decoder-only language model from its last-layer hidden states.\"},{\"question\":\"How does the proposed gradient-based inversion method differ from one-shot reconstruction?\",\"answer\":\"It performs continuous optimization in embedding space with a soft proxy, avoids hard-token projection during the search, and commits tokens only once at the end of the inner loop.\"},{\"question\":\"What does the paper find about when recovery fails?\",\"answer\":\"Failure is dominated by space-prefixed, high-frequency function words in dense regions of the embedding matrix, while content-bearing tokens are recovered almost perfectly. Exact-match accuracy improves substantially with a wider candidate window, suggesting many errors are recoverable near-misses.\"}]",1784181521,40,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"recovering-input-text-from-hidden-states-study-of-gradient-based-inversion-of-decoder-only-language-models","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/recovering-input-text-from-hidden-states-study-of-gradient-based-inversion-of-decoder-only-language-models/82557/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper address?","Question",{"text":75,"@type":76},"The paper addresses hidden-state inversion: recovering the original input token sequence of a decoder-only language model from its last-layer hidden states.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the proposed gradient-based inversion method differ from one-shot reconstruction?",{"text":80,"@type":76},"It performs continuous optimization in embedding space with a soft proxy, avoids hard-token projection during the search, and commits tokens only once at the end of the inner loop.",{"name":82,"@type":73,"acceptedAnswer":83},"What does the paper find about when recovery fails?",{"text":84,"@type":76},"Failure is dominated by space-prefixed, high-frequency function words in dense regions of the embedding matrix, while content-bearing tokens are recovered almost perfectly. Exact-match accuracy improves substantially with a wider candidate window, suggesting many errors are recoverable near-misses.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,119,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":29,"slug":118},7,"Healthcare","healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]