[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-572432-105":53,"doc-detail-572432-en":126},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":119,"head_meta":121,"extra_data":123,"updated_unix":125},105,"en","learning-to-learn-from-multimodal-experience-preprint","Learning to Learn from Multimodal Experience - Preprint","","Multimodal agents increasingly interact with real-world-like environments and generate rich trajectories spanning perception, reasoning, and action. A key challenge is converting raw multimodal trajectories into reusable experience that improves future task performance. Existing experience-driven learning methods are mainly built for textual settings and depend on manually designed memory schemas, limiting their effectiveness when signals vary by modality, time scale, and abstraction. AUTOMMEMO automatically designs memory mechanisms for multimodal learning via executable memo programs, iterative update-then-retrieve evaluation, reflection-guided mutation, and budget-aware tree search.",{"@graph":63,"@context":118},[64,80,101],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":51,"@type":70,"position":76},"https://docshare.wps.com/template/general/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/learning-to-learn-from-multimodal-experience-preprint/572432/",4,{"url":78,"name":59,"@type":81,"image":82,"author":87,"headline":59,"publisher":90,"fileFormat":93,"inLanguage":57,"description":61,"dateModified":94,"datePublished":95,"encodingFormat":93,"isAccessibleForFree":96,"interactionStatistic":97},"DigitalDocument",{"url":83,"@type":84,"width":85,"height":86},"https://docshare.wps.com/thumbnails/learning-to-learn-from-multimodal-experience-preprint/572432.png","ImageObject",442,249,{"name":88,"@type":89},"Muffet","Person",{"url":68,"name":91,"@type":92},"DocShare","Organization","application/pdf","2026-10-20","2026-10-13",true,{"@type":98,"interactionType":99,"userInteractionCount":73},"InteractionCounter",{"@type":100},"ViewAction",{"@type":102,"mainEntity":103},"FAQPage",[104,110,114],{"name":105,"@type":106,"acceptedAnswer":107},"What problem does AUTOMMEMO address in multimodal experience-driven learning?","Question",{"text":108,"@type":109},"It addresses how to turn raw multimodal interaction trajectories into reusable experience that improves future task performance, without relying on manually designed memory schemas.","Answer",{"name":111,"@type":106,"acceptedAnswer":112},"How does AUTOMMEMO represent and design memory mechanisms?",{"text":113,"@type":109},"It represents each memory mechanism as an executable memo program that specifies how past episodes are stored, organized, and retrieved, then discovers effective memo programs through iterative update-then-retrieve evaluation and guided mutation.",{"name":115,"@type":106,"acceptedAnswer":116},"What evidence supports AUTOMMEMO’s effectiveness?",{"text":117,"@type":109},"Experiments across GUI/Web navigation and multimodal visual reasoning benchmarks show consistent performance gains over no-memory agents and fixed memory baselines, with manageable token and interaction costs.","https://schema.org",{"og:url":78,"og:type":120,"og:title":59,"og:site_name":91,"og:description":61},"article",{"robots":122,"canonical":78},"index,follow",{"doc_id":124,"site_id":56},572432,1792524662,{"code":4,"msg":5,"data":127},{"doc_id":124,"user_id":128,"nickname":88,"user_avatar":129,"doc_module":9,"category_id":50,"category_name":51,"doc_title":59,"doc_description":61,"doc_content":130,"file_id":131,"file_url":132,"file_type":133,"file_size":134,"view_count":73,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":135,"language":136,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":137,"faqs":138,"seo_title":139,"seo_description":61,"update_tm":140,"read_time":141},962090768927,"https://ap-avatar.wpscdn.com/davatar_085a072bc5b1113ac321206ff7593b45","arXiv :2605 . 16857v1 [ cs .AI] 16 May 2026  \nLearning to Learn from Multimodal Experience  \nXingyu Sui1 , Weixiang Zhao1 , Yongxin Tang1  \nYanyan Zhao1 , Yang Wu2 , Dandan Tu2 , Bing Qin1  \n1Harbin Institute of Technology, 2Huawei Technologies Co., Ltd  \n{xysui, wxzhao, [yyzhao}@ir.hit.edu.cn](yyzhao}@ir.hit.edu.cn)  \nAbstract  \nMultimodal agents increasingly interact with real-world-like environments and generate rich trajectories spanning perception, reasoning, and action. A key research focus is how to transform these raw multimodal trajectories into reusable experience that can improve future task performance. Existing experience-driven learning methods are mostly developed in textual settings and typically rely on manually designed memory schemas, making them less effective for multimodal experience that requires composing heterogeneous signals across modalities, temporal scales, and abstraction levels. We propose AUTOMMEMO , a general framework that enables agents to automatically design memory mechanisms for learning from multimodal experience. AUTOMMEMO represents each memory mechanism as an executable memo program, which defines how past episodes are stored, organized, and retrieved. It discovers effective memo programs through an iterative process of update-then-retrieve evaluation, reflection-guided mutation, and budget-aware tree search. Experiments across GUI/Web navigation and multimodal visual reasoning benchmarks show that AUTOMMEMO consistently improves agent performance over no-memory agents and fixed memory baselines, while maintaining reasonable token and interaction costs. Transfer and search-process analyses further demonstrate that the learned memory designs capture reusable experience structures and progressively refine retrieval relevance. These results highlight adaptive memory design as a key optimization target for multimodal experience-driven learning.  \n1 Introduction  \nRecent advances in multimodal large language models (MLLMs) have enabled agents to actively explore and interact within environments that more closely resemble the real world [Yao et al., 2023, Durante et al., 2024, Li et al., 2025a] . Such interactions naturally generate large volumes of trajectories, capturing rich signals across perception, reasoning, and action. A central research focus, therefore, is how to transform these raw interaction trajectories into reusable experience that can continuously improve agent performance [Shinn et al., 2023, Silver and Sutton, 2025] . In this context, experience-driven learning via memory mechanisms has emerged as a promising paradigm, enabling agents to accumulate, organize, and reuse experience across tasks [Gao et al., 2025, Fang et al., 2025] . As a result, the effectiveness of an agent is no longer solely determined by its underlying model, but increasingly by how it represents and utilizes its experience [Dou et al., 2025] .  \nHowever, existing experience-driven learning paradigms have been predominantly developed in textual settings [Zhao et al., 2024, Zhang et al., 2025a, Xia et al., 2026], leaving their extension to multimodal environments largely underexplored. This creates a fundamental mismatch: in real-world scenarios, experience is inherently multimodal, arising from interactions that involve diverse signals such as vision, language, and action dynamics [Driess et al., 2023, Sarch et al., 2024] . These signals differ in modality, temporal scope, and semantic abstraction, forming a complex mixture of structured and unstructured information. Thus, designing an effective memory system for experience learning  \nPreprint.  \nrequires making a series of tightly coupled decisions, such as what to store, how to represent it, how to organize it, and when to retrieve it. For multimodal experience, these decisions become particularly challenging, as useful information may be distributed across visual observations, language-based reasoning, and action trajectories at different temporal sca","cbCaivXLPp29r2po","https://ap.wps.com/l/cbCaivXLPp29r2po","pdf",1294852,56,"English","# Abstract\n# 1 Introduction","[{\"question\":\"What problem does AUTOMMEMO address in multimodal experience-driven learning?\",\"answer\":\"It addresses how to turn raw multimodal interaction trajectories into reusable experience that improves future task performance, without relying on manually designed memory schemas.\"},{\"question\":\"How does AUTOMMEMO represent and design memory mechanisms?\",\"answer\":\"It represents each memory mechanism as an executable memo program that specifies how past episodes are stored, organized, and retrieved, then discovers effective memo programs through iterative update-then-retrieve evaluation and guided mutation.\"},{\"question\":\"What evidence supports AUTOMMEMO’s effectiveness?\",\"answer\":\"Experiments across GUI/Web navigation and multimodal visual reasoning benchmarks show consistent performance gains over no-memory agents and fixed memory baselines, with manageable token and interaction costs.\"}]","Learning to Learn from Multimodal Experience - Preprint | PDF",1791927382,20]