[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-125985-en":3,"doc-seo-125985-105":31,"detail-sidebar-cat-0-en-105":93},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":28,"seo_description":14,"update_tm":29,"read_time":30},125985,687207024478,"Liam","https://ap-avatar.wpscdn.com/davatar_a8503ba1806abce46bf441b54a3ca4cd",8,"Research & Report","Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning","Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large language models by modular, plug-and-play updates. Uploadable Machine Learning (UML) extends this idea: edge contributors train domain-specific LoRA adapters and upload them to a central pool for personalized serving under heterogeneous downstream requests. Existing LoRA composition methods lack support for UML’s dynamic, expanding LoRA pool and prompt-dependent expert needs. This work introduces RAMoLE, a retrieve-then-compose framework combining relevant adapter retrieval, on-the-fly MoLE coordination, and efficient batch inference. Experiments show consistent gains over baselines and flexible scalability.","Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning  \nZiyu Zhao, Leilei Gan, Guoyin Wang, Yuwei Hu, Tao Shen, Hongxia Yang, Member, IEEE,  \nFei Wu, Senior Member, IEEE, Kun Kuang†  \narXiv :2406 . 16989v2 [ cs .LG] 16 Jul 2024  \nAbstract—Low-Rank Adaptation (LoRA) offers an effective solution for fine-tuning large language models (LLMs). Its modular nature allows the integration of diverse domain-specific LoRAsto enhance LLM capabilities. The emergence of open-source platforms like Huggingface has given rise to a new computational paradigm termed Uploadable Machine Learning (UML). In this framework, edge-side contributors train specialized LoRA adapters and then upload them to a central platform that leverages their plug-and-play nature to provide personalized services for downstream heterogeneous requests. Previous research on LoRA composition has centered on isolated tasks or static adapter selection during training, which does not adequately address the dynamic and expanding pool of LoRAs in UML nor sufficiently meet the needs of downstream heterogeneous requests. To bridge this gap, we propose Retrieval-Augmented Mixture of LoRA Experts (RAMoLE), a retrieve-then-compose framework that adaptively retrieves and composes multiple LoRAs according to the input prompts. RAMoLE contains three main components: firstly, identifying and retrieving LoRAs relevant to the given input through LoraRetriever; secondly, coordinating various dynamically retrieved LoRAs through a carefully designed onthe-fly MoLE mechanism; and thirdly, developing efficient batch inference to accommodate heterogeneous requests. The experimental results indicate that RAMoLE consistently outperforms the baselines, emphasizing its effectiveness and flexible scalability.  \nIndex Terms—Data mining, text mining, natural language processing, and retrieval models.  \nI. INTRODUCTION  \nREcently, leveraging the principles  \nLarge Language Models (LLMs)  \nof scaling laws [1], like openAI’s GPT-  \n3.5 [2], GPT-4 [3], Meta’s LLama [4] and many other Foundation Models [5]–[8] have achieved notable success across various natural language processing (NLP) tasks [9], [10] . However, these models, primarily trained on diverse internet datasets, sometimes struggle in specialized areas. To address this, numerous studies [9],[11],[12] have explored fine-tuning LLMs using domain-specific data, thereby adapting them to fulfill specific needs. Due to the prohibitively high computation costs for fine-tuning LLMs on specific domains, thereis a growing shift towards Parameter-Efficient Fine-Tuning  \n(PEFT) [13]–[15], which only updates a small fraction of the †Corresponding author.  \nZ. Zhao, L. Gan, Y. Hu, T. Shen F. Wu and K. Kuang are with the College of Computer Science and Technology, Zhejiang University, China. (Email: [benzhao.styx@gmail.com](benzhao.styx@gmail.com); [leileigan@zju.edu.cn](leileigan@zju.edu.cn); [hyw-luv@zju.edu.cn](hyw-luv@zju.edu.cn); [tao.shen@zju.edu.cn](tao.shen@zju.edu.cn); [wufei@zju.edu.cn](wufei@zju.edu.cn); [kunkuang@zju.edu.cn](kunkuang@zju.edu.cn)).  \nG. Wang and H. Yang are with ByteDance Inc., Seattle, USA. (E-mail: [guoyinwang.duke@gmail.com](guoyinwang.duke@gmail.com); [hx.yang@bytedance.com](hx.yang@bytedance.com)).  \nFigure 1 . Illustration of UML. a) LoRAs from various domains and tasks aimed at enhancing specific capabilities of the LLM can be uploaded to or updated to the LoRA pool. b) The multi-LoRA serving framework aims to leverage the plug-and-play nature of LoRAs to offer comprehensive services. c) The downstream tasks, presented in a mixed-task form, require personalized expert routing.  \nmodel’s parameters or integrates new trainable parameters that augment the model’s capabilities.  \nAs interest grows in open-source communities like Hugging Face 1 , ModelScope2 , and Civit AI3 , there is an increasing trend towards sharing numerous PEFT modules trained on abroad spectrum of domains and tasks across these platfo","cbCaifgDlyFWPTcS","https://ap.wps.com/l/cbCaifgDlyFWPTcS","pdf",3056754,7,1,17,"English","en",105,"# Introduction\n## Uploadable Machine Learning (UML)\n## LoRA and Parameter-Efficient Fine-Tuning (PEFT)\n## Challenges in Multi-LoRA Serving\n## Proposed Approach: RAMoLE","[{\"question\":\"What problem does RAMoLE address in Uploadable Machine Learning (UML)?\",\"answer\":\"It addresses the lack of prompt-adaptive LoRA composition for UML, where the LoRA pool is dynamic and grows over time. Prior methods do not adequately handle heterogeneous downstream requests with a continually expanding set of adapters.\"},{\"question\":\"How does RAMoLE work at a high level?\",\"answer\":\"RAMoLE retrieves relevant LoRAs for the input prompt, coordinates multiple dynamically retrieved LoRAs using an on-the-fly MoLE mechanism, and performs efficient batch inference for heterogeneous requests.\"},{\"question\":\"Why are LoRA adapters suitable for UML scenarios?\",\"answer\":\"LoRA is modular and plug-and-play, allowing domain-specific adapters to be trained efficiently with reduced parameter updates and then loaded dynamically by a centralized serving platform.\"}]","Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning | PDF",1785902394,43,{"code":4,"msg":32,"data":33},"ok",{"site_id":25,"language":24,"slug":34,"title":13,"keywords":35,"description":14,"schema_data":36,"social_meta":88,"head_meta":90,"extra_data":92,"updated_unix":29},"retrieval-augmented-mixture-of-lora-experts-for-uploadable-machine-learning","",{"@graph":37,"@context":87},[38,55,70],{"@type":39,"itemListElement":40},"BreadcrumbList",[41,45,49,52],{"item":42,"name":43,"@type":44,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":46,"name":47,"@type":44,"position":48},"https://docshare.wps.com/document/","Document",2,{"item":50,"name":12,"@type":44,"position":51},"https://docshare.wps.com/document/research-report/",3,{"item":53,"name":13,"@type":44,"position":54},"https://docshare.wps.com/document/retrieval-augmented-mixture-of-lora-experts-for-uploadable-machine-learning/125985/",4,{"url":53,"name":13,"@type":56,"author":57,"headline":13,"publisher":59,"fileFormat":62,"inLanguage":24,"description":14,"dateModified":63,"datePublished":64,"encodingFormat":62,"isAccessibleForFree":65,"interactionStatistic":66},"DigitalDocument",{"name":9,"@type":58},"Person",{"url":42,"name":60,"@type":61},"DocShare","Organization","application/pdf","2026-08-22","2026-08-05",true,{"@type":67,"interactionType":68,"userInteractionCount":20},"InteractionCounter",{"@type":69},"ViewAction",{"@type":71,"mainEntity":72},"FAQPage",[73,79,83],{"name":74,"@type":75,"acceptedAnswer":76},"What problem does RAMoLE address in Uploadable Machine Learning (UML)?","Question",{"text":77,"@type":78},"It addresses the lack of prompt-adaptive LoRA composition for UML, where the LoRA pool is dynamic and grows over time. Prior methods do not adequately handle heterogeneous downstream requests with a continually expanding set of adapters.","Answer",{"name":80,"@type":75,"acceptedAnswer":81},"How does RAMoLE work at a high level?",{"text":82,"@type":78},"RAMoLE retrieves relevant LoRAs for the input prompt, coordinates multiple dynamically retrieved LoRAs using an on-the-fly MoLE mechanism, and performs efficient batch inference for heterogeneous requests.",{"name":84,"@type":75,"acceptedAnswer":85},"Why are LoRA adapters suitable for UML scenarios?",{"text":86,"@type":78},"LoRA is modular and plug-and-play, allowing domain-specific adapters to be trained efficiently with reduced parameter updates and then loaded dynamically by a centralized serving platform.","https://schema.org",{"og:url":53,"og:type":89,"og:title":13,"og:site_name":60,"og:description":14},"article",{"robots":91,"canonical":53},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":94},[95,99,103,107,112,117,121,124,129,132,136],{"id":21,"doc_module":4,"doc_module_name":47,"category_name":96,"show_sort_weight":97,"slug":98},"Story & Novel",90,"story-novel",{"id":48,"doc_module":4,"doc_module_name":47,"category_name":100,"show_sort_weight":101,"slug":102},"Literature",80,"literature",{"id":54,"doc_module":4,"doc_module_name":47,"category_name":104,"show_sort_weight":105,"slug":106},"Exam",70,"exam",{"id":108,"doc_module":4,"doc_module_name":47,"category_name":109,"show_sort_weight":110,"slug":111},5,"Comic",60,"comic",{"id":113,"doc_module":4,"doc_module_name":47,"category_name":114,"show_sort_weight":115,"slug":116},6,"Technology",50,"technology",{"id":20,"doc_module":4,"doc_module_name":47,"category_name":118,"show_sort_weight":119,"slug":120},"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":47,"category_name":12,"show_sort_weight":122,"slug":123},30,"research-report",{"id":125,"doc_module":4,"doc_module_name":47,"category_name":126,"show_sort_weight":127,"slug":128},9,"Religion & Spirituality",20,"religion-spirituality",{"id":127,"doc_module":4,"doc_module_name":47,"category_name":130,"show_sort_weight":127,"slug":131},"World Cup","world-cup",{"id":133,"doc_module":4,"doc_module_name":47,"category_name":134,"show_sort_weight":133,"slug":135},10,"Lifestyle","lifestyle",{"id":137,"doc_module":4,"doc_module_name":47,"category_name":138,"show_sort_weight":108,"slug":139},19,"General","general"]