[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-84916-en":3,"doc-seo-84916-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},84916,1099514068035,"Ezra","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders","Vision-Language Models (VLMs) are increasingly used as the conditioning backbone for diffusion-based image editing, leveraging multimodal reasoning to localize objects and attributes. However, editing pipelines often lose localization accuracy in complex, multi-entity scenes. This work studies the gap by treating the VLM as a condition encoder and introducing Analysis-by-Proxy, which trains a lightweight interpretable proxy on intermediate representations via an auxiliary localization task. Results show single-pass constraints hide localization signals inside variable intermediate layers rather than the predefined conditioning layers, revealing failures of current condition-extraction strategies.","Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders  \nYoav Baron 1 Sara Dorfman 1 Roni Paiss 2 Daniel Cohen-Or 1 Or Patashnik 1  \narXiv :2607 .06445v 1 [ cs .CV] 7 Jul 2026  \nAbstract  \nVision-Language Models (VLMs) are increasingly utilized as the conditioning backbone for diffusion-based image editing due to their remarkable multimodal reasoning capabilities. While standalone VLMs demonstrate strong localization capabilities, editing pipelines frequently struggle to maintain this accuracy, particularly in complex, multi-entity scenes. In this work, we investigate this performance gap, hypothesizing that it stems from treating the VLM as a condition encoder.  \nIn this role, the model is restricted to a single forward pass, preventing the autoregressive generation process for which it was optimized, thereby failing to fully expose its capabilities. To investigate whether this spatial understanding persists when the VLM is used as a condition encoder, we introduce Analysis-by-Proxy. In this framework, we train a lightweight, interpretable proxy model on the VLM’s intermediate representations using an auxiliary localization task. By analyzing the VLM through this proxy, we uncover the specific VLM representations that encode localization information. Our findings expose a fundamental mismatch between how spatial knowledge is represented within a VLM condition encoder and how it is extracted by current editing pipelines.  \nWe reveal that under single-pass constraints, the localization signal does not reliably propagate to the predefined layer configurations commonly used for conditioning. Instead, this crucial signal remains hidden within intermediate representations, at locations that vary depending on the input prompt. Using our introduced Analysis-by-Proxy framework, we reveal the fundamental failures of existing condition extraction strategies in editing  \npipelines, opening the door to more principled  design of conditioning architectures.  \n1Tel Aviv University, Tel Aviv, Israel 2 Google DeepMind. Correspondence to: Yoav Baron \u003C[yvbrn13@gmail.com](yvbrn13@gmail.com)>.  \nProceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 . Copyright 2026 by the author(s) .  \n1. Introduction  \nVision-Language Models (VLMs) (Bai et al., 2025a ;b ; Liu et al., 2023b ;a; Team, 2024 ; Li et al., 2025) have recently emerged as powerful tools, demonstrating remarkable capabilities in parsing and reasoning over multimodal inputs. As such, they have been widely adopted as the backbone for the instruction condition in state-of-the-art diffusionbased image editing models (Wu et al., 2025) . These editing pipelines typically condition a Diffusion Transformer (DiT) (Rombach et al., 2022 ; Esser et al., 2024) on the hidden representations extracted from a VLM, making the overall edit quality critically dependent on which internal representations are selected for conditioning. These representations serve several functions within the editing process, including providing the signal for the accurate localization of the object or attribute to be edited. Although localization is only one component of the editing process, even slight failures at this stage directly result in incorrect, misplaced, or entirely hallucinated edits. The challenge of accurate localization is especially pronounced in complex, multientity scenes, where the model must determine which visual instance satisfies the textual description and distinguish it from similar surrounding objects (see Figure 2) .  \nIn this work, we investigate the behavior of the VLM when it serves as the conditioning backbone for a DiT. We characterize this paradigm as treating the VLM as a condition encoder: the model processes the input in a single forward pass without autoregressively generating text. In this setting, standard practice utilizes representations from a predefined and input-independent subset of layers for the conditi","cbCaiuLgDTRXiWwq","https://ap.wps.com/l/cbCaiuLgDTRXiWwq","pdf",12169750,1,18,"English","en",105,"# Abstract\n# Introduction\n## VLMs as conditioning backbones for diffusion editing\n## Performance gap and single-pass condition-encoder setting","[{\"question\":\"What problem does the paper address in diffusion-based image editing with VLMs?\",\"answer\":\"Editing pipelines using VLM hidden representations often mislocalize targets, especially in complex multi-entity scenes. The paper investigates why accurate localization deteriorates when the VLM is used under a condition-encoder constraint.\"},{\"question\":\"Why does using a VLM as a condition encoder cause localization failures?\",\"answer\":\"The paper hypothesizes a mismatch caused by the VLM’s pretraining for autoregressive generation. Under single forward-pass constraints, spatial knowledge may be encoded internally but not reliably propagated to the predefined layers used for conditioning.\"},{\"question\":\"How does Analysis-by-Proxy help identify where localization information is stored?\",\"answer\":\"Analysis-by-Proxy trains a lightweight interpretable proxy on the VLM’s intermediate representations using an auxiliary localization task. By analyzing the VLM through this proxy, it reveals which internal representations encode localization signals.\"}]",1784199335,45,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"analysis-by-proxy-localization-signals-in-vlms-operating-as-condition-encoders","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/analysis-by-proxy-localization-signals-in-vlms-operating-as-condition-encoders/84916/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does the paper address in diffusion-based image editing with VLMs?","Question",{"text":75,"@type":76},"Editing pipelines using VLM hidden representations often mislocalize targets, especially in complex multi-entity scenes. The paper investigates why accurate localization deteriorates when the VLM is used under a condition-encoder constraint.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why does using a VLM as a condition encoder cause localization failures?",{"text":80,"@type":76},"The paper hypothesizes a mismatch caused by the VLM’s pretraining for autoregressive generation. Under single forward-pass constraints, spatial knowledge may be encoded internally but not reliably propagated to the predefined layers used for conditioning.",{"name":82,"@type":73,"acceptedAnswer":83},"How does Analysis-by-Proxy help identify where localization information is stored?",{"text":84,"@type":76},"Analysis-by-Proxy trains a lightweight interpretable proxy on the VLM’s intermediate representations using an auxiliary localization task. By analyzing the VLM through this proxy, it reveals which internal representations encode localization signals.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]