[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85424-en":3,"doc-seo-85424-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":13,"seo_description":14,"update_tm":27,"read_time":28},85424,1099513958762,"Logic","https://ap-avatar.wpscdn.com/avatar/1000023916a998db790?x-image-process=image/resize,m_fixed,w_180,h_180&k=1782109480056885918",8,"Research & Report","AutoV Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs","AutoV introduces a lightweight visual-prompt retrieval framework for LVLMs, addressing the saturation of fixed single-prompt designs. Given an image and a textual query, AutoV automatically selects the most suitable visual prompt from a diverse candidate pool by learning instance-adaptive preferences. Training supervision is automated by evaluating each prompt candidate with a pretrained LVLM and ranking candidates by prediction loss, yielding a robust, query-aware signal without manual annotation. Experiments show gains across image understanding, captioning, grounding, and classification.","arXiv :2506 . 16112v4 [ cs .CV] 11 Jul 2026  \nAutoV: Loss-Oriented Ranking for Visual Prompt Retrieval in LVLMs  \nYuan Zhang 1 ,2, Chun-Kai Fan 1, Sicheng Yu2, Junwen Pan2, Tao Huang3, Ming Lu 1, Kuan Cheng 1, Qi She2, and Shanghang Zhang 1†  \n1 State Key Laboratory of Multimedia Information Processing,  \nSchool of Computer Science, Peking University  \n2 ByteDance  \n3 Shanghai Jiao Tong University  \n Code [https://github.com/Gumpest/AutoV](https://github.com/Gumpest/AutoV)  \nAbstract. Inspired by text prompts in large language models, visual prompts have been explored to enhance the perceptual capabilities of large vision-language models (LVLMs) . However, performance tends to saturate under single visual prompt designs, making further prompt engineering increasingly ineffective. To address this limitation, we shift from prompt engineering to prompt retrieval and propose AutoV, a lightweight framework for instance-adaptive visual prompt identification. Given an input image and a textual query, AutoV automatically locates the most suitable visual prompt from a diverse candidate pool. Training such a retrieval framework requires prompt-level supervision, yet prompt quality is inherently ambiguous and difficult to assess reliably, even for humans.  \nTo enable automatic supervision, we evaluate visual prompts using a pretrained LVLM and label them according to their prediction losses. Using the loss-oriented ranking as a robust training signal, AutoV learns to retrieve the query-aware optimal prompt for each instance without manual annotation. Experiments indicate that AutoV enhances the performance of various LVLMs on image understanding, captioning, grounding, and classification tasks. For example, AutoV improves LLaVA-OV by 10.2% on VizWiz and boosts Qwen2 .5-VL by 3.8% on MMMU, respectively.  \nKeywords: Visual Prompt · Retrieval · Loss-Oriented · LVLMs  \n1 Introduction  \nRecent advancements in large language models (LLMs) [1, 3 , 5 , 8 , 10 , 42 , 56] have spurred significant progress in large vision-language models (LVLMs), which connect visual encoders with language model decoders to effectively adapt textual reasoning capabilities to visual understanding tasks [4, 15 , 24 , 30 , 37 , 52 , 53] . Consequently, visual input plays a crucial role in LVLMs, enabling precise perception and fine-grained grounding across various multimodal scenarios.  \n† Corresponding author.  \n2 Zhang et al.  \nIs there an airplane in the image?  \n√ Easy for Human Label  \nWhat logo is printed on top right? × Hard for Human Label  \n(a) Performance Saturation. (b) Visual Prompt Labeling Difficulty.  \nFig. 1: Motivation of AutoV. (a) Performance saturation. Existing visual prompts approach benchmark ceilings, limiting further gains from prompt engineering. (b) Labeling difficulty and task diversity. Optimal prompts vary across tasks, and the crown denotes the one leading to the correct answer. While the optimal prompt is easy to identify in the top example, it is much harder to determine in the bottom one.  \nInspired by textual prompting in LLMs [28, 58 , 79], visual prompting has emerged as an effective paradigm for guiding LVLM attention [35, 51 , 60 , 71] . By injecting structured visual cues, such as blur masks, bounding circles, or attention heatmaps, visual prompts encourage models to focus on task-relevant regions. However, existing approaches predominantly rely on fixed, heuristically designed prompts. While these handcrafted strategies can yield improvements on certain benchmarks, they implicitly assume that a single prompt design generalizes across diverse visual scenes and textual queries.  \nIn practice, this assumption rarely holds. As shown in Figure 1(a), heuristic prompts tend to approach benchmark-specific performance ceilings, leaving limited room for improvement through prompt engineering alone. Moreover, prompt effectiveness varies across tasks and instances. For example, in Figure 1(b), attention-based masks may benefit OCR-sensitiv","cbCaibD1J39hrKG6","https://ap.wps.com/l/cbCaibD1J39hrKG6","pdf",9819367,1,26,"English","en",105,"# Introduction\n## Motivation and Problem Setup\n## AutoV Framework: Candidate Generation and Ranking\n## Loss-Oriented Automated Supervision","[{\"question\":\"What problem does AutoV address in visual prompting for LVLMs?\",\"answer\":\"AutoV addresses performance saturation caused by fixed, heuristically designed single visual prompts, where further prompt engineering yields limited improvements and optimal prompts vary by instance and task.\"},{\"question\":\"How does AutoV retrieve an appropriate visual prompt for a given image-query pair?\",\"answer\":\"AutoV generates multiple visual prompt candidates, encodes their representations, and uses a lightweight ranking network that integrates visual and textual information to select the highest-ranking candidate for inference.\"},{\"question\":\"How does AutoV generate supervision without manual prompt annotation?\",\"answer\":\"AutoV evaluates each prompt candidate using a pretrained LVLM and labels candidates according to their prediction losses, then trains the retrieval model with loss-oriented ranking as an automated training signal.\"}]",1784203375,66,{"code":4,"msg":30,"data":31},"ok",{"site_id":24,"language":23,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"autov-loss-oriented-ranking-for-visual-prompt-retrieval-in-lvlms","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/autov-loss-oriented-ranking-for-visual-prompt-retrieval-in-lvlms/85424/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":23,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-17","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does AutoV address in visual prompting for LVLMs?","Question",{"text":75,"@type":76},"AutoV addresses performance saturation caused by fixed, heuristically designed single visual prompts, where further prompt engineering yields limited improvements and optimal prompts vary by instance and task.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does AutoV retrieve an appropriate visual prompt for a given image-query pair?",{"text":80,"@type":76},"AutoV generates multiple visual prompt candidates, encodes their representations, and uses a lightweight ranking network that integrates visual and textual information to select the highest-ranking candidate for inference.",{"name":82,"@type":73,"acceptedAnswer":83},"How does AutoV generate supervision without manual prompt annotation?",{"text":84,"@type":76},"AutoV evaluates each prompt candidate using a pretrained LVLM and labels candidates according to their prediction losses, then trains the retrieval model with loss-oriented ranking as an automated training signal.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":45,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":45,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":45,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]