[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-seo-327152-105":3,"detail-sidebar-cat-0-en-105":81,"doc-detail-327152-en":130},{"code":4,"msg":5,"data":6},0,"ok",{"site_id":7,"language":8,"slug":9,"title":10,"keywords":11,"description":12,"schema_data":13,"social_meta":74,"head_meta":76,"extra_data":78,"updated_unix":80},105,"en","egogapbench-benchmarking-egocentric-action-selection-in-multi-agent-scenes","EGOGAPBENCH - BENCHMARKING EGOCENTRIC ACTION SELECTION IN MULTI-AGENT SCENES","","Existing egocentric benchmarks build the egocentric setting mainly from first-person-view data, making it hard to evaluate egocentric perspective understanding independently. This work separates first-person-view understanding from egocentric perspective understanding and introduces EgoGapBench, a diagnostic benchmark for Egocentric Action Selection (EAS) in multi-agent scenes. Humans perform reliably, while open-source and proprietary MLLMs choose other agents’ visible actions, and fine-tuning on EgoGapBench data improves accuracy without reaching human levels.",{"@graph":14,"@context":73},[15,34,56],{"@type":16,"itemListElement":17},"BreadcrumbList",[18,23,27,31],{"item":19,"name":20,"@type":21,"position":22},"https://docshare.wps.com","Home","ListItem",1,{"item":24,"name":25,"@type":21,"position":26},"https://docshare.wps.com/document/","Document",2,{"item":28,"name":29,"@type":21,"position":30},"https://docshare.wps.com/document/research-report/","Research & Report",3,{"item":32,"name":10,"@type":21,"position":33},"https://docshare.wps.com/document/egogapbench-benchmarking-egocentric-action-selection-in-multi-agent-scenes/327152/",4,{"url":32,"name":10,"@type":35,"image":36,"author":41,"headline":10,"publisher":44,"fileFormat":47,"inLanguage":8,"description":12,"dateModified":48,"datePublished":49,"encodingFormat":47,"isAccessibleForFree":50,"interactionStatistic":51},"DigitalDocument",{"url":37,"@type":38,"width":39,"height":40},"https://docshare.wps.com/thumbnails/egogapbench-benchmarking-egocentric-action-selection-in-multi-agent-scenes/327152.png","ImageObject",300,407,{"name":42,"@type":43},"Patrick","Person",{"url":19,"name":45,"@type":46},"DocShare","Organization","application/pdf","2026-11-07","2026-09-21",true,{"@type":52,"interactionType":53,"userInteractionCount":55},"InteractionCounter",{"@type":54},"ViewAction",9,{"@type":57,"mainEntity":58},"FAQPage",[59,65,69],{"name":60,"@type":61,"acceptedAnswer":62},"What problem does EgoGapBench target in existing egocentric benchmarks?","Question",{"text":63,"@type":64},"Existing benchmarks rely largely on first-person-view data, so model correctness can reflect visible first-person body cues rather than true egocentric perspective understanding. EgoGapBench aims to isolate the perspective ability when such cues are absent.","Answer",{"name":66,"@type":61,"acceptedAnswer":67},"How does EgoGapBench define Egocentric Action Selection (EAS)?",{"text":68,"@type":64},"EAS is defined as selecting an appropriate action from the agent’s perspective in the presence of other agents, where other agents’ actions act as plausible but perspective-inappropriate attractors.",{"name":70,"@type":61,"acceptedAnswer":71},"How do MLLMs and humans perform on EgoGapBench?",{"text":72,"@type":64},"Humans answer reliably. Open-source and proprietary MLLMs perform substantially worse and systematically select actions performed by other visible agents, while fine-tuning on EgoGapBench training data improves accuracy but does not reach human performance.","https://schema.org",{"og:url":32,"og:type":75,"og:title":10,"og:site_name":45,"og:description":12},"article",{"robots":77,"canonical":32},"index,follow",{"doc_id":79,"site_id":7},327152,1791428895,{"code":4,"msg":82,"data":83},"success",[84,88,92,96,101,106,111,115,119,122,126],{"id":22,"doc_module":4,"doc_module_name":25,"category_name":85,"show_sort_weight":86,"slug":87},"Story & Novel",90,"story-novel",{"id":26,"doc_module":4,"doc_module_name":25,"category_name":89,"show_sort_weight":90,"slug":91},"Literature",80,"literature",{"id":33,"doc_module":4,"doc_module_name":25,"category_name":93,"show_sort_weight":94,"slug":95},"Exam",70,"exam",{"id":97,"doc_module":4,"doc_module_name":25,"category_name":98,"show_sort_weight":99,"slug":100},5,"Comic",60,"comic",{"id":102,"doc_module":4,"doc_module_name":25,"category_name":103,"show_sort_weight":104,"slug":105},6,"Technology",50,"technology",{"id":107,"doc_module":4,"doc_module_name":25,"category_name":108,"show_sort_weight":109,"slug":110},7,"Healthcare",40,"healthcare",{"id":112,"doc_module":4,"doc_module_name":25,"category_name":29,"show_sort_weight":113,"slug":114},8,30,"research-report",{"id":55,"doc_module":4,"doc_module_name":25,"category_name":116,"show_sort_weight":117,"slug":118},"Religion & Spirituality",20,"religion-spirituality",{"id":117,"doc_module":4,"doc_module_name":25,"category_name":120,"show_sort_weight":117,"slug":121},"World Cup","world-cup",{"id":123,"doc_module":4,"doc_module_name":25,"category_name":124,"show_sort_weight":123,"slug":125},10,"Lifestyle","lifestyle",{"id":127,"doc_module":4,"doc_module_name":25,"category_name":128,"show_sort_weight":97,"slug":129},19,"General","general",{"code":4,"msg":82,"data":131},{"doc_id":79,"user_id":132,"nickname":42,"user_avatar":133,"doc_module":4,"category_id":112,"category_name":29,"doc_title":10,"doc_description":12,"doc_content":134,"file_id":135,"file_url":136,"file_type":137,"file_size":138,"view_count":55,"is_deleted":4,"is_public":22,"is_downloadable":22,"audit_status":22,"page_count":139,"language":140,"language_code":8,"site_id":7,"html_lang":8,"table_of_contents":141,"faqs":142,"seo_title":143,"seo_description":12,"update_tm":144,"read_time":145,"preview_page_count":4},549758146520,"https://ap-avatar.wpscdn.com/avatar/80002397d8c0411e94?_k=1775819394049821470","EGOGAPBENCH: BENCHMARKING EGOCENTRIC ACTION SELECTION IN MULTI-AGENT SCENES  \nJihyeok Jung  \nKAIST AI  \n[ji9759@kaist.ac.kr](ji9759@kaist.ac.kr)  \nJeewu Lee  \nSogang University  \n[xlxaxb@sogang.ac.kr](xlxaxb@sogang.ac.kr)  \nSanghyeop Kim  \nSogang University  \n[fgd25@sogang.ac.kr](fgd25@sogang.ac.kr)  \nChanhee Han  \nMinistry of Science and ICT [enhance63@korea.kr](enhance63@korea.kr)  \nSeong Joon Oh∗  \nKAIST AI [coallaoh@kaist.ac.kr](coallaoh@kaist.ac.kr)  \narXiv :2607 .00547v 1 [ cs .CV] 1 Jul 2026  \nABSTRACT  \nExisting egocentric benchmarks have primarily constructed the egocentric setting from first-person-view data, which makes it difficult to evaluate egocentric perspective itself in isolation. However, understanding first-person-view input and taking an egocentric perspective are separable abilities, especially when firstperson body cues are absent or when other agents are present. To isolate egocentric perspective understanding, we introduce EgoGapBench, a diagnostic benchmark for measuring action selection in multi-agent egocentric scenes. We define the ability measured by this benchmark as Egocentric Action Selection (EAS): selecting an appropriate action from the agent’s perspective in the presence of other agents. On EgoGapBench, humans answer reliably, whereas both open-source and proprietary MLLMs perform substantially worse and systematically select actions performed by other visible agents. Fine-tuning on existing egocentric data fails to close this gap and can even be detrimental. In contrast, fine-tuning on EgoGapBench training data improves accuracy but does not reach human performance.  \nThese results show that EAS is difficult to acquire from first-person-view data alone, and that MLLMs should be evaluated and trained not only for scene understanding but also for egocentric action selection. The code and benchmark are available at [https://github.com/jhCOR/EgoGapBench](https://github.com/jhCOR/EgoGapBench).  \n1 INTRODUCTION  \nExisting egocentric benchmarks (Mangalam et al., 2023; Rezaei et al., 2025; Pan et al., 2026; Jia et al., 2022) have primarily constructed the egocentric setting using first-person-view data (Grauman et al., 2022; Damen et al., 2020), making it difficult to isolate egocentric perspective itself (Chenget al., 2024; Jung et al., 2025) (Figure 2, left) . When we vertically flip images sampled from EgoThink (Cheng et al., 2024), the model returns nearly the same answer to questions asking about the camera wearer’s action, suggesting that it relies on observed first-person body cues (Figure 2, middle) . Therefore, in existing egocentric benchmarks, it is difficult to verify whether a model’s correct answer reflects a genuine understanding of egocentric perspective or merely the reading of visible body cues. However, as shown in Figure 1, current MLLMs, unlike humans, struggle to choose appropriate actions when first-person body cues are absent (Zhang et al., 2026) .  \nIn this study, we distinguish the ability to understand first-person-view input (Grauman et al., 2022) from the ability to understand an egocentric perspective. The former concerns the ability to interpret visual input captured from the field of view of an agent situated in the scene. In contrast, the latter concerns the ability to understand a scene from the observer’s standpoint, and is therefore not limited to first-person-view input but can also apply to more general input formats (Jung et al., 2025) . The two abilities usually operate together, but they can become separable when the agent’s body is not visible or when other agents are present in the scene. This separation becomes especially important  \n∗ Corresponding author.  \nFigure 1: An example item from EgoGapBench. The item asks the model to adopt a viewpoint behind home plate, corresponding to the umpire’s role, in a scene that also contains other agents. Besides the correct answer, the options include the observed action of another agent (the batter preparing to hi","cbCaivbQbL4sLFla","https://ap.wps.com/l/cbCaivbQbL4sLFla","pdf",1598443,15,"English","# Abstract\n# Introduction\n## Problem with existing egocentric benchmarks\n## Separating egocentric perspective understanding from first-person-view understanding\n## EgoGapBench design and task definition","[{\"question\":\"What problem does EgoGapBench target in existing egocentric benchmarks?\",\"answer\":\"Existing benchmarks rely largely on first-person-view data, so model correctness can reflect visible first-person body cues rather than true egocentric perspective understanding. EgoGapBench aims to isolate the perspective ability when such cues are absent.\"},{\"question\":\"How does EgoGapBench define Egocentric Action Selection (EAS)?\",\"answer\":\"EAS is defined as selecting an appropriate action from the agent’s perspective in the presence of other agents, where other agents’ actions act as plausible but perspective-inappropriate attractors.\"},{\"question\":\"How do MLLMs and humans perform on EgoGapBench?\",\"answer\":\"Humans answer reliably. Open-source and proprietary MLLMs perform substantially worse and systematically select actions performed by other visible agents, while fine-tuning on EgoGapBench training data improves accuracy but does not reach human performance.\"}]","EGOGAPBENCH - BENCHMARKING EGOCENTRIC ACTION SELECTION IN MULTI-AGENT SCENES | PDF",1789976556,38]