[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84852-en":3,"doc-seo-84852-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},84852,8796095461564,"Liam","https://ap-avatar.wpscdn.com/davatar_155a257f0dc6eb9ab79c44ca47cae57d",8,"Research & Report","Akashic: A Low-Overhead LLM Inference Service with MemAttention","Akashic proposes a low-overhead memory system for LLM agent serving where long, continuously accumulated context increases prefill cost, may exceed context limits, and can dilute task evidence, harming throughput and output quality. The system uses MemAttention to organize memory into bounded semantic chunks with cross-chunk inference so cross-chunk evidence is preserved without rewriting full history. It also applies hardware-software co-designed memory placement to reduce the locality gap and retrieval fragmentation. Experiments on four benchmarks show up to 10.2-point accuracy gains, 1.21× throughput, and 1.88× higher sustainable request rates versus strong baselines.","arXiv:2607.05708v1 [cs.AI] 7 Jul 2026  \n# Akashic:A Low-Overhead LLM Inference Service withMemAttention\n\nYang Liu**  Zhaokai Luo**Huayi Jin Ruozhou He Chenchen Hong Zhiyong WangYifei Liu⁹  Yunfei Gu Chentao WuI Junhao Hu  \nXiaohongshu Inc.,China IShangHai JiaoTong University Peking University*Corresponding to:Zhaokai Luo \u003Cluozhaokai@xiaohongshu.com>  \n## Abstract\n\nRecent LLM-based agent systems continuously accumulatecontext across multi-turn interactions,tool invocations,andcross-session workflows.Replaying the full history for ev-ery request quickly becomes impractical:long contexts in-crease prefill cost,may exceed context limits,and often burytask-relevant evidence in irrelevant content,degrading bothserving efficiency and output quality.We propose Akashic,a low-overhead memory system built around MemAtten-tion,which organizes context into bounded chunks and mod-els semantic relationships across chunks,preserving cross-chunk evidence without repeatedly rewriting the full history.Akashic further applies hardware-software co-designed mem-ory placement to co-locate likely co-retrieved chunks,reduc-ing retrieval fragmentation and I/O overhead.Across fourrepresentative workloads and three model sizes,Akashicimproves task accuracy by up to 10.2 points,throughput byup to 1.21×,and sustainable request rate by up to 1.88×overstrong prior memory baselines.  \n(b)BrowseComp·OPT-30B  \nFigure 1.(a)Existing memory designs occupy differentpoints on the task-quality /serving-efficiency trade-off,whereas Akashic targets the high-quality,high-efficiencyregime.(b)On BrowseComp with OPT-30B,Akashic im-proves both accuracy and throughput over all baselines,out-performing the strongest prior method by about 2.0 pointsin accuracy and about 1.35×in throughput  \nas MemGAS summarize smaller units independently[41].They bound per-update cost,but semantically dependentevidence may be split across segments and never recoveredtogether,weakening multi-turn and long-range reasoning.  \nIn practical LLM serving,resolving this trade-off requiresaddressing two key challenges.First,context informationdensity is highly non-uniform across workloads and evenacross different phases of the same interaction(§3.1):somespans are highly compressible,whereas others are dense andexpensive to compress.A fixed global compression policy istherefore often mismatched to the actual workload.Second,agent memory is not only a semantic object but also a stor-age object.Even when retrieval selects only a few relevantmemories,end-to-end latency can remain high if those mem-ories are physically scattered across many pages or blocks.We refer to this mismatch between semantic relatedness andphysical placement as the locality gap(§3.2).  \nWe present Akashic,a low-overhead memory systemfor LLM serving that addresses these challenges throughMemAttention and hardware-softwareco-designed mem-ory placement optimization.Akashic performs chunk-granularmemory maintenance:it compacts one bounded chunk ata time,rather than repeatedly rewriting the full history.Topreserve cross-chunk evidence,Akashic uses cross-chunkinference to reconcile each new chunk with a small set of se-mantically related prior chunks before writing the result backto memory.This design bounds maintenance overhead whileretaining information that would otherwise be fragmented  \n## 1 Introduction\n\nRecent agent systems continuously accumulate input contextacross multi-turn interactions,tool invocations,and cross-session execution,and this growing context can substantiallydegrade model throughput and output quality[7,11,13,14,  \n21,25,40].A straightforward solution is to feed the full inter-action history into every request,but this approach quicklybecomes impractical:long prompts increase prefill cost,canexceed context limits,and often bury task-relevant evidencein irrelevant context,hurting both serving efficiency andoutput quality[20,29].As a result,modern agent systemsincreasingly maintain external memory and re-inject only ","cbCaivM08DSXzweZ","https://ap.wps.com/l/cbCaivM08DSXzweZ","pdf",3157583,4,1,14,"English","en",105,"# Abstract\n# 1 Introduction","[{\"question\":\"What problem does Akashic target in LLM agent memory systems?\",\"answer\":\"Akashic targets efficiency and quality degradation caused by replaying full multi-turn history, which increases prefill cost, risks context-limit overflow, and buries task-relevant evidence in irrelevant content.\"},{\"question\":\"How does MemAttention help preserve useful cross-chunk evidence?\",\"answer\":\"MemAttention maintains memory at chunk granularity and reconciles each new bounded chunk with a small set of semantically related prior chunks via cross-chunk inference before writing back.\"},{\"question\":\"What does the hardware-software memory placement optimization improve?\",\"answer\":\"It co-locates likely co-retrieved chunks and compacts misplaced stale data, reducing retrieval fragmentation and I/O overhead caused by the locality gap during end-to-end serving latency.\"}]",1784198810,35,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"akashic-a-low-overhead-llm-inference-service-with-memattention","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":20},"https://docshare.wps.com/document/akashic-a-low-overhead-llm-inference-service-with-memattention/84852/",{"url":52,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does Akashic target in LLM agent memory systems?","Question",{"text":75,"@type":76},"Akashic targets efficiency and quality degradation caused by replaying full multi-turn history, which increases prefill cost, risks context-limit overflow, and buries task-relevant evidence in irrelevant content.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does MemAttention help preserve useful cross-chunk evidence?",{"text":80,"@type":76},"MemAttention maintains memory at chunk granularity and reconciles each new bounded chunk with a small set of semantically related prior chunks via cross-chunk inference before writing back.",{"name":82,"@type":73,"acceptedAnswer":83},"What does the hardware-software memory placement optimization improve?",{"text":84,"@type":76},"It co-locates likely co-retrieved chunks and compacts misplaced stale data, reducing retrieval fragmentation and I/O overhead caused by the locality gap during end-to-end serving latency.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]