[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-82685-en":3,"doc-seo-82685-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},82685,3848291630094,"Emma Wilson","https://eur-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45",8,"Research & Report","Metronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving","Real-time interaction models such as Moshi, MiniCPM-o, and Qwen-Omni turn model serving into a periodic real-time task, ingesting streaming audio each frame and decoding against a wall-clock deadline while KV cache memory grows and remains pinned. Under sustained full-duplex load, serving can fail abruptly: per-frame latency stays low and metrics look healthy, then the engine freezes once accumulated session state exhausts the KV pool. Metronome bounds per-session resident state using an in-engine KV window, restores stability and observability, and enables online admission control to discover schedulable concurrency end-to-end on real audio across four models.","arXiv :2607 .02640v 1 [ cs . SD] 2 Jul 2026  \nMetronome: Bound the Cache, Keep the Beat for Real-Time Interaction Model Serving  \nJiaying Meng Independent Researcher  \nBojie Li Pine AI  \nAbstract  \nReal-time interaction models—Moshi, MiniCPM-o, Qwen-Omni—turn serving into a periodic real-time task: on every frame a session ingests streaming audio and must respond by a recurring wall-clock deadline, while its KV cache grows monotonically and stays pinned for the whole conversation. This regime hides a dangerous failure mode. On a real full-duplex stack, sustained load does not degrade serving gracefully: it falls off a cliff, jumping in one step from milliseconds per frame to a stalled engine when accumulated session state exhausts the KV pool. The collapse is metastable—identical five-minute runs collapse or survive on run-to-run variance—and silent: latency and deadline-miss metrics read healthy throughout.  \nWe show one move restores both stability and observability: bound each session’s resident state, and latency starts telling the truth. Metronome’s in-engine KV window eliminates the collapse (0/20 vs. 14/20 runs across two batches) and turns per-frame latency into a monotone load signal, on which an online admission controller discovers the schedulable concurrency; without the window, the identical controller over-admits into the wall. A first-order model predicts the collapse time within a few percent on the headline model, and a quality probe validates the bound’s design by ablation: the window alone is quality-free in turn-based decoding, and its few pinned attentionsink tokens are what keep free-running generation healthy. Everything is measured end-to-end on real audio, across four interaction models on one GPU.  \nCode: [https://github.com/19PINE-AI/metronome](https://github.com/19PINE-AI/metronome)[ ](https://github.com/19PINE-AI/metronome)Website: [https://01.me/research/metronome](https://01.me/research/metronome)  \nper-frame latency (ms)  \n103  \n102  \n101  \n100  \nvLLM-realtime (unbounded KV) Metronome (windowed KV)  \n2 s frame deadline (miss it and the call stutters)  \nfrozen at the wall  \nlatency cliff:  \nthe call freezes flat, a few ms: stays on beat  \n0 50 100 150 200 250 300  \nelapsed time in a single call (s), at 128 concurrent sessions  \nFigure 1. Existing serving falls off a latency cliff; Metronome stays on beat. One five-minute call at 128 sessions (Qwen3-Omni-30B): unbounded resident KV jumps in one step from a few ms to a ∼1.6 s ceiling where the stalled engine stops producing tokens; windowed KV stays flat for the whole call.  \n1 Introduction  \nA new class of models has quietly created a serving regime of its own. Kyutai Moshi [Défossez et al., 2024], MiniCPM-o-4.5 [OpenBMB, 2026], and the Qwen-Omni family [Chu et al., 2024, Xu et al., 2025] listen to streaming audio and respond continuously, with no turn boundary, and a growing line of streaming speech LLMs follows the same shape [Xie and Wu, 2024, Fang et al., 2024] . Frontier labs are converging on it: Thinking Machines Lab’s interaction models interleave 200 ms micro-turns of multimodal input and output on a persistent GPU sequence [Thinking Machines Lab, 2026] . At the serving layer these models are not chatbots. A chatbot request is ephemeral: a prompt arrives, tokens are generated, the request and its KV cache are freed. An interaction session is a periodic real-time task: on every frame—every 80 ms for Moshi, every 1–2 s for the omni models—it ingests a new audio chunk as a small prefill and decodes a short response, against a recurring wall-clock deadline, for a conversation that runs for minutes (Figure 3) .  \nThis regime sits at an unstudied intersection. LLM serving assumes requests are ephemeral: engines maximize aggregate throughput and reclaim or swap KV between requests [Yu et al., 2022, Kwon et al., 2023, Gujarati et al., 2020], parking whatever state must persist out of GPU memory during the idle gaps between turns. Classical real-t","cbCaiblV2mm6Zm53","https://ap.wps.com/l/cbCaiblV2mm6Zm53","pdf",625042,1,21,"English","en",105,"# Introduction\n# Failure Mode and Its Properties\n## Memory Saturation\n## Metastability\n## Silent Deadline-Miss Behavior\n# Metronome Approach\n## Bounded Resident State with KV Window\n## Stability and Observability\n# Experimental Setup and Evaluation\n## End-to-End Measurement on Real Audio\n## Multi-Model, Single-GPU Results","[{\"question\":\"What failure mode does the paper identify for real-time interaction model serving under sustained load?\",\"answer\":\"Serving can collapse abruptly: per-frame latency and deadline-miss signals remain healthy until the KV pool is exhausted, at which point the engine stalls and stops producing tokens.\"},{\"question\":\"Why are the collapse and overload hard to detect using standard latency metrics?\",\"answer\":\"The collapse is silent—empty frames are returned on time—so latency and deadline-miss metrics can look normal throughout, only freezing at the instant of collapse.\"},{\"question\":\"How does Metronome address the instability?\",\"answer\":\"Metronome bounds per-session resident state via an in-engine KV window, eliminating the collapse and making per-frame latency monotonic as a load signal for online admission control.\"}]",1784182279,53,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"metronome-bound-the-cache-keep-the-beat-for-real-time-interaction-model-serving","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/metronome-bound-the-cache-keep-the-beat-for-real-time-interaction-model-serving/82685/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What failure mode does the paper identify for real-time interaction model serving under sustained load?","Question",{"text":75,"@type":76},"Serving can collapse abruptly: per-frame latency and deadline-miss signals remain healthy until the KV pool is exhausted, at which point the engine stalls and stops producing tokens.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why are the collapse and overload hard to detect using standard latency metrics?",{"text":80,"@type":76},"The collapse is silent—empty frames are returned on time—so latency and deadline-miss metrics can look normal throughout, only freezing at the instant of collapse.",{"name":82,"@type":73,"acceptedAnswer":83},"How does Metronome address the instability?",{"text":84,"@type":76},"Metronome bounds per-session resident state via an in-engine KV window, eliminating the collapse and making per-frame latency monotonic as a load signal for online admission control.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]