[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85995-en":3,"doc-seo-85995-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85995,1374391975076,"Riley","https://ap-avatar.wpscdn.com/avatar/14000253ca4ec9f6853?x-image-process=image/resize,m_fixed,w_180,h_180&k=1783305029341752051",8,"Research & Report","Spectral Heat Flow for Conservative Token Condensation in Vision Language Models","Vision-Language Models (VLMs) incur high inference cost due to processing long sequences of visual tokens. Existing token pruning often fails under aggressive compression by discarding information in a way that breaks spatial coherence and reduces diversity. SpecFlow introduces a training-free conservative condensation framework that uses spectral heat flow on a kNN token graph, adaptive spatial partitioning for guaranteed coverage, and coreset sinks to aggregate discarded tokens while preserving statistical conservation. Results show improved performance across architectures and pruning ratios, with LLaVA-1.5 retaining 95.6% performance at 88.9% visual token pruning.","Spectral Heat Flow for Conservative Token Condensation in Vision-Language  \nModels  \nZhaoyang Li * 1 Yanjun Li * 1 Wangkai Li 1 Yujia Chen 1 Tianzhu Zhang 1  \narXiv :2607 . 10640v1 [ cs .CV] 12 Jul 2026  \nAbstract  \nVision-Language Models (VLMs) are costly at inference time because they must process long sequences of visual tokens. Existing token pruning methods often degrade under high compression by blindly discarding information, breaking spatial structure or collapsing diversity. We propose SpecFlow, a training-free framework that shifts the paradigm from destructive pruning to conservative condensation, strictly enforcing spatial coverage and statistical conservation to ensure stability. Treating visual tokens as nodes in akNN graph, SpecFlow (i) computes a stable importance field via spectral heat flow to preserve structural coherence, (ii) allocates budgets via adaptive spatial partitioning to guarantee coverage, and (iii) aggregates discarded information into coreset sinks to maintain statistical conservation. The method is plug-and-play, requires no fine-tuning, and is compatible with FlashAttention. Experiments confirm that our SpecFlow outperforms SOTA methods across tasks, VLM architectures, and pruning ratios. Notably, LLaVA-1.5 with SpecFlow retains 95.6% of original performance despite pruning 88.9% of visual tokens, offering an exceptional efficiency-accuracy balance. Code is available at [https://github](https://github) . com/Lzy-dot/SpecFlow.  \n1. Introduction  \nRecent Vision-language models (VLMs) (Li et al., 2025a ; Team et al., 2023 ; Liu et al., 2024c ; Chen et al., 2024c) have shown remarkable progress in multimodal understanding, delivering strong results on visual question answering (Guo  \n1 School of Information Science and Technology / National Key Laboratory of Deep Space Exploration, University of Science and Technology of China. Correspondence to: Tianzhu Zhang \u003C[tzzhang@ustc.edu.cn](tzzhang@ustc.edu.cn) >.  \nProceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) .  \nFigure 1. Global Top-K pruning vs. our proposed SpecFlow.(a) Global Top-K pruning based on [CLS] attention can yield fragmented selections due to spiky attention distributions, often creating spatial holes within objects. (b) SpecFlow diffuses attentionderived energy on a kNN token graph and applies coverage-aware regional budgeting, yielding region-coherent retained tokens under high compression.  \net al., 2023 ; Zhao et al., 2024 ; Huynh et al., 2025 ; Kuanget al., 2025), image captioning (Ghandi et al., 2023 ; Chenet al., 2024a), and video understanding (Lin et al., 2024 ; Maaz et al., 2024 ; Wang et al., 2024b).Despite this progress, efficient deployment remains challenging due to the large number of visual tokens produced by modern vision encoders. For example, LLaVA-1.5 (Liu et al., 2024a) typically encodes a 336 × 336 image into 576 patch tokens, which are then processed by the language model during theprefill stage, often resulting in substantially higher latency than text-only inference.  \nA natural approach to accelerate inference is to reduce the number of visual tokens. Existing token pruning methods (Chen et al., 2024b ; Yang et al., 2025 ; Zhang et al., 2024b ; Zou et al., 2025) commonly perform token pruning via Top-K selection using attention-derived scores. While effective in reducing computation, such point-wise ranking can be brittle under high compression because it disregards two properties of visual token sets: spatial coherence (tokens corresponding to a region tend to form contiguous structures) and contextual dependency (background and surrounding regions can be essential for reasoning) . Empirically, these limitations manifest as two recurring failure modes. First, attention scores can be highly concentrated, so Top-K selection often yields spatially fragmented token subsets, creating “holes” within objects and disrupting  \nc","cbCaimAlzTOCFWs1","https://ap.wps.com/l/cbCaimAlzTOCFWs1","pdf",5488799,2,1,19,"English","en",105,"# Introduction\n## Challenges of token pruning in VLM inference\n## SpecFlow: conservative condensation framework","[{\"question\":\"为什么现有的视觉-语言模型（VLM）在推理阶段成本较高？\",\"answer\":\"因为推理需要处理由视觉编码器产生的长序列视觉token。即使是相对固定的图像分块，也会在预填充（prefill）阶段带来显著延迟。\"},{\"question\":\"现有基于Top-K注意力的token pruning在高压缩下有哪些常见失败模式？\",\"answer\":\"一是注意力分布可能高度集中，导致保留token空间碎片化，在对象内部产生“空洞”；二是显著性驱动的剪枝可能过度保留前景而移除关键背景语境，从而破坏依赖空间关系与场景语义的任务。\"},{\"question\":\"SpecFlow如何实现“保守凝结”并保持稳定性？\",\"answer\":\"SpecFlow在token图上通过谱热流传播得到平滑且结构一致的重要性场，并使用自适应空间划分分配区域预算以保证空间覆盖；同时把被丢弃的信息聚合到coreset“汇点”中，以维持统计守恒与多样性，从而无需微调即可提升效率-精度平衡。\"}]",1784207657,48,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"spectral-heat-flow-for-conservative-token-condensation-in-vision-language-models","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,47,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":20},"https://docshare.wps.com/document/","Document",{"item":48,"name":12,"@type":43,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/spectral-heat-flow-for-conservative-token-condensation-in-vision-language-models/85995/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-23","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"为什么现有的视觉-语言模型（VLM）在推理阶段成本较高？","Question",{"text":75,"@type":76},"因为推理需要处理由视觉编码器产生的长序列视觉token。即使是相对固定的图像分块，也会在预填充（prefill）阶段带来显著延迟。","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"现有基于Top-K注意力的token pruning在高压缩下有哪些常见失败模式？",{"text":80,"@type":76},"一是注意力分布可能高度集中，导致保留token空间碎片化，在对象内部产生“空洞”；二是显著性驱动的剪枝可能过度保留前景而移除关键背景语境，从而破坏依赖空间关系与场景语义的任务。",{"name":82,"@type":73,"acceptedAnswer":83},"SpecFlow如何实现“保守凝结”并保持稳定性？",{"text":84,"@type":76},"SpecFlow在token图上通过谱热流传播得到平滑且结构一致的重要性场，并使用自适应空间划分分配区域预算以保证空间覆盖；同时把被丢弃的信息聚合到coreset“汇点”中，以维持统计守恒与多样性，从而无需微调即可提升效率-精度平衡。","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":20,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":22,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},"General","general"]