[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-83375-en":3,"doc-seo-83375-105":29,"detail-sidebar-cat-0-en-105":90},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},83375,1099514068365,"Aurelia","https://ap-avatar.wpscdn.com/avatar/10000253d8d9f28188e?_k=1776742907772140068",8,"Research & Report","Prompt Compression via Activation Aggregation","Large language models process prompts by propagating activations through many layers before producing an answer. This work examines whether instruction-relevant information can be compressed into a single activation (patch) vector and reinjected into an early layer, replacing the original token sequence. A learned weighted sum of intermediate-layer activations is used for compression, preserving task-relevant content with less than 2% accuracy degradation. Results also analyze cross-layer compatibility and the semantic recoverability of a single vector.","Prompt Compression via Activation Aggregation  \nThibaud Ardoin, Semira Einsele, Evis Bregu, Gerhard Wunder  \nFreie Universit¨at Berlin  \n[thibaud. ardoin@fu-berlin. de](thibaud. ardoin@fu-berlin. de)  \narXiv :2607 .08399v 1 [ cs .CL] 9 Jul 2026  \nAbstract  \nLarge language models process prompts by propagating activations through dozens of layers before generating a response. We ask whether the task-relevant information contained in an instruction prompt can be compressed into a single activation vector and re-injected into the model, replacing the original token sequence? We show this is achievable using a learned weighted sum of activations extracted at an intermediate layer and injected at an early layer of the target LLM. The compressed vector preserves task-relevant information, incurring an accuracy drop of under 2% relative to full prompt processing. Beyond its practical implications, including reducing per-query computation for fixed instruction prompts without reprocessing the original token sequence, our analysis reveals structure in the activation space of LLMs: (i) mid-layer representations transfer meaningfully to early layers, suggesting a degree of cross-layer compatibility in how information is encoded; (ii) a single activation vector encodes a quantifiable and recoverable amount of semantic information; (iii) a weighted sum of activations is a robust representation compressor.  \n1 Introduction  \nModern LLMs are often queried with repeated prompt prefixes, such as instructions, system prompts, or few-shot examples. In naive inference, these tokens are recomputed at every call, even when the prefix remains fixed and only the user query changes. This is computationally wasteful: the same prefix is repeatedly tokenized, embedded, and propagated through the transformer layers. A natural question is whether the taskrelevant information contained in such a prefix can be pre-computed, compressed, and reused directly in activation space.  \nCurrent systems already exploit repeated prompts through mechanisms such as KVcaching (Pope et al., 2023; Kwon et al., 2023), which store intermediate states associated with a fixed  \nStep 1: Extraction  \ninformation prompt  \nt1  \n···  \ntn  \n\n|  |  |  |  |  |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |\n|  |  |  |  |  |  |  |\n| hidden states |  |  |  |  |  |  |\n| (frozen) |  |  |  |  |  |  |\n\nStep 2: Compression  \nextraction  \nabout information prompt  \nFigure 1: Overview of the proposed three-step framework: extract hidden states from an input prompt, compress them into a patch vector, and inject the patch vector into a placeholder token to answer a query without direct access to the original prompt.  \nprefix. These methods provide exact or near-exact reuse: the prefix computation is retained so that it does not need to be recomputed. In contrast, we ask a more restrictive question: can the effect of a prompt be compressed into a single promptdependent activation vector and then injected back into the model in a way that preserves its behavior?  \nBeyond the engineering motivation, this question probes a more fundamental issue: how is taskrelevant information from a prompt represented in the activation space of an LLM? Prior work on activation engineering has shown that steering vectors can influence complex model behavior through simple additions in latent space (Subramani et al. ,  \n2022) . More generally, the linearity hypothesis suggests that some high-level concept representations allow simple arithmetic operations (Mikolov et al. , 2013; Liu et al. , 2023) .  \nWith respect to information compression, task vectors (Hendel et al. , 2023) condense in-context learning examples, but they do not extend naturally to general information prompts. Recent methods have shown that long contexts or arbitrary promptscan be compressed into a small number of tokens or representations (Mu et al. , 2023; Ge et al. , 2023) . However, these strong r","cbCaieyhW6oLiHjz","https://ap.wps.com/l/cbCaieyhW6oLiHjz","pdf",605742,1,15,"English","en",105,"# Introduction\n## Activation-space prompt compression approach\n## Contributions","[{\"question\":\"What problem does activation-space prompt compression address?\",\"answer\":\"It addresses the computational waste of recomputing fixed prompt prefixes (instructions, system prompts, or few-shot examples) in every inference call, even when only the user query changes.\"},{\"question\":\"How is the prompt compressed and reused in the proposed method?\",\"answer\":\"The method extracts hidden states from an intermediate layer, compresses them into a prompt-dependent patch vector using a learned weighted sum, then injects that vector into the model via an early-layer placeholder token for a second forward pass.\"},{\"question\":\"How much accuracy loss does the compressed activation approach introduce?\",\"answer\":\"The compressed vector preserves task-relevant information with an accuracy drop of under 2% compared to full prompt processing.\"}]",1784187074,38,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":85,"head_meta":87,"extra_data":89,"updated_unix":27},"prompt-compression-via-activation-aggregation","",{"@graph":35,"@context":84},[36,53,67],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/prompt-compression-via-activation-aggregation/83375/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":61,"encodingFormat":60,"isAccessibleForFree":62,"interactionStatistic":63},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-16",true,{"@type":64,"interactionType":65,"userInteractionCount":4},"InteractionCounter",{"@type":66},"ViewAction",{"@type":68,"mainEntity":69},"FAQPage",[70,76,80],{"name":71,"@type":72,"acceptedAnswer":73},"What problem does activation-space prompt compression address?","Question",{"text":74,"@type":75},"It addresses the computational waste of recomputing fixed prompt prefixes (instructions, system prompts, or few-shot examples) in every inference call, even when only the user query changes.","Answer",{"name":77,"@type":72,"acceptedAnswer":78},"How is the prompt compressed and reused in the proposed method?",{"text":79,"@type":75},"The method extracts hidden states from an intermediate layer, compresses them into a prompt-dependent patch vector using a learned weighted sum, then injects that vector into the model via an early-layer placeholder token for a second forward pass.",{"name":81,"@type":72,"acceptedAnswer":82},"How much accuracy loss does the compressed activation approach introduce?",{"text":83,"@type":75},"The compressed vector preserves task-relevant information with an accuracy drop of under 2% compared to full prompt processing.","https://schema.org",{"og:url":51,"og:type":86,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":88,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":91},[92,96,100,104,109,114,119,122,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":93,"show_sort_weight":94,"slug":95},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":97,"show_sort_weight":98,"slug":99},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":101,"show_sort_weight":102,"slug":103},"Exam",70,"exam",{"id":105,"doc_module":4,"doc_module_name":45,"category_name":106,"show_sort_weight":107,"slug":108},5,"Comic",60,"comic",{"id":110,"doc_module":4,"doc_module_name":45,"category_name":111,"show_sort_weight":112,"slug":113},6,"Technology",50,"technology",{"id":115,"doc_module":4,"doc_module_name":45,"category_name":116,"show_sort_weight":117,"slug":118},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":120,"slug":121},30,"research-report",{"id":123,"doc_module":4,"doc_module_name":45,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":105,"slug":137},19,"General","general"]