[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-125825-en":3,"doc-seo-125825-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},125825,549758252649,"Ivy","https://ap-avatar.wpscdn.com/avatar/8000253669c5317157?_k=1778319167496531819",8,"Research & Report","Multi-tenant Machine Learning Model Serving Systems on GPU Clusters - A dissertation","GPUs are costly and scarce, making efficient machine-learning inference a critical systems challenge. In multi-tenant deployments, pooling a shared GPU cluster can serve multiple models with far fewer resources than independently allocating k GPUs per model. This dissertation optimizes cluster-wide GPU utilization by addressing batching efficiency under latency constraints, GPU consolidation for bursty requests, and GPU cluster auto-scaling. It presents Symphony for optimized DNN serving and Punica for cost-effective multi-tenant LoRA LLM serving.","Multi-tenant Machine Learning Model Serving Systems on  \nGPU Clusters  \nLequn Chen  \nA dissertation  \nsubmitted in partial fulfillment of the  \nrequirements for the degree of  \nDoctor of Philosophy  \nUniversity of Washington  \n2024  \nReading Committee:  \nArvind Krishnamurthy, Chair  \nRatul Mahajan  \nLuis Ceze  \nProgram Authorized to Offer Degree:  \nComputer Science & Engineering  \n©Copyright 2024 Lequn Chen  \nUniversity of Washington  \nAbstract  \nMulti-tenant Machine Learning Model Serving Systems on GPU Clusters  \nLequn Chen  \nChair of the Supervisory Committee:  \nArvind Krishnamurthy  \nComputer Science & Engineering  \nIn an era where GPUs are both costly and scarce, efficiently serving machine learning models has become a critical challenge. Assuming that serving one model requires k GPUs, serving n models would seemingly require kn GPUs. In the multi-tenant setting, we can pool the whole cluster’s GPUs to serve the n models collectively, thus requiring far fewer GPUs. This talk addresses how to optimize cluster-wide GPU utilization in a multi-tenant setting. Key challenges addressed include: (1) batching efficiency under latency constraints, (2) bursty requests and GPU consolidation,(3) GPU cluster auto-scaling.  \nThis dissertation discusses two projects that address the above research problems. The first project, Symphony, focuses on serving DNN models. With a novel Deferred Batch Scheduling algorithm and a system design supporting it, Symphony makes high-quality batching decisions and enables robust auto-scaling. Symphony achieves 6x goodput given the same number of GPUs, saves 60% GPUs when serving the same request rate, and is capable to handle 15 million requests per second. The second project, Punica, creates anew paradigm of serving multiple LoRA fine-tuned large language models at the cost of one. Punica improves throughput by 12x without latency sacrifice.  \nTABLE OF CONTENTS  \nPage  \nList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii  \nList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v  \nChapter 1: Introduction ................................ 1  \n1.1 Overview ..................................... 1  \n1.2 Thesis Contributions ............................... 3  \n1.3 Published Materials ................................ 4  \n1.4 Thesis Outline ................................... 4  \nChapter 2: Background ................................. 5  \n2.1 Deep neural network model serving ....................... 6  \n2.2 Large lanuage model serving .......................... 7  \nChapter 3: Symphony: Optimized DNN Serving using Deferred Batch Scheduling 11  \n3.1 Background .................................... 13  \n3.2 Deferred Batch Scheduling ............................ 16  \n3.3 System Design and Implementation ....................... 25  \n3.4 Evaluation ..................................... 30  \n3.5 Discussion ..................................... 41  \n3.6 Summary ..................................... 42  \nChapter 4: Punica: Multi-Tenant LoRA Serving .................... 43  \n4.1 Background .................................... 45  \n4.2 Punica Overview ................................. 46  \n4.3 Segmented Gather Matrix-Vector Multiplication ................ 48  \n4.4 Punica in Detail .................................. 51  \n4.5 Implementation .................................. 55  \n4.6 Evaluation ..................................... 57  \n4.7 Related Work ................................... 65  \n4.8 Summary ..................................... 66  \nChapter 5: Conclusion ................................. 67  \n5.1 Future Work .................................... 67  \nBibliography ........................................ 70  \nAppendix A: Appendix for Symphony .......................... 82  \nA.1 Global Partitioning Algorithm .......................... 82  \nA.2 Networking Performance of Our Testbed .................... 85  \nA.3 Model Zoo Det","cbCaiu94GLTQy44E","https://ap.wps.com/l/cbCaiu94GLTQy44E","pdf",1599796,1,98,"English","en",105,"# Chapter 1: Introduction\n## 1.1 Overview\n## 1.2 Thesis Contributions\n## 1.3 Published Materials\n## 1.4 Thesis Outline\n# Chapter 2: Background\n## 2.1 Deep neural network model serving\n## 2.2 Large language model serving\n# Chapter 3: Symphony: Optimized DNN Serving using Deferred Batch Scheduling\n## 3.1 Background\n## 3.2 Deferred Batch Scheduling\n## 3.3 System Design and Implementation\n## 3.4 Evaluation\n## 3.5 Discussion\n## 3.6 Summary\n# Chapter 4: Punica: Multi-Tenant LoRA Serving\n## 4.1 Background\n## 4.2 Punica Overview\n## 4.3 Segmented Gather Matrix-Vector Multiplication\n## 4.4 Punica in Detail\n## 4.5 Implementation\n## 4.6 Evaluation\n## 4.7 Related Work\n## 4.8 Summary\n# Chapter 5: Conclusion\n## 5.1 Future Work\n# Bibliography\n# Appendix A: Appendix for Symphony\n## A.1 Global Partitioning Algorithm\n## A.2 Networking Performance of Our Testbed\n## A.3 Model Zoo Details\n## A.4 Scheduling Algorithm Pseudocode","[{\"question\":\"Why does multi-tenant ML model serving reduce GPU requirements?\",\"answer\":\"Instead of dedicating k GPUs per model, multi-tenant serving pools the entire GPU cluster to serve n models collectively, which can require far fewer GPUs than kn.\"},{\"question\":\"What main cluster utilization problems does the dissertation focus on?\",\"answer\":\"It targets batching efficiency under latency constraints, handling bursty requests via GPU consolidation, and enabling GPU cluster auto-scaling.\"},{\"question\":\"What are the two projects introduced, and what do they optimize?\",\"answer\":\"Symphony optimizes DNN serving with Deferred Batch Scheduling and robust auto-scaling. Punica introduces a multi-tenant LoRA serving approach that improves throughput significantly without latency sacrifice.\"}]","Multi-tenant Machine Learning Model Serving Systems on GPU Clusters - A dissertation | PDF",1785901423,247,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"multi-tenant-machine-learning-model-serving-systems-on-gpu-clusters-a-dissertation","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/multi-tenant-machine-learning-model-serving-systems-on-gpu-clusters-a-dissertation/125825/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-05",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why does multi-tenant ML model serving reduce GPU requirements?","Question",{"text":75,"@type":76},"Instead of dedicating k GPUs per model, multi-tenant serving pools the entire GPU cluster to serve n models collectively, which can require far fewer GPUs than kn.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What main cluster utilization problems does the dissertation focus on?",{"text":80,"@type":76},"It targets batching efficiency under latency constraints, handling bursty requests via GPU consolidation, and enabling GPU cluster auto-scaling.",{"name":82,"@type":73,"acceptedAnswer":83},"What are the two projects introduced, and what do they optimize?",{"text":84,"@type":76},"Symphony optimizes DNN serving with Deferred Batch Scheduling and robust auto-scaling. Punica introduces a multi-tenant LoRA serving approach that improves throughput significantly without latency sacrifice.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]