Device-Cloud Collaborative LLM Inference with Multi-Modal, Multi-Task, Multi-Turn Conversations

Jul 16, 2026

Upload by Logic

Recent large language models (LLMs) enable multitask solving via multi-modal inputs and multi-turn dialogue, but deployment is harder than for traditional ML due to their size. On-device inference is limited by compute, memory, and energy, while cloud inference may violate real-time needs and adds communication and usage costs. The paper proposes TMO, a device-cloud LLM system using Three-Way Offloading across multi-modal data, multi-task handling, and multi-turn context. A resource-constrained reinforcement learning strategy selects inference location and modalities to maximize long-term reward under constraints, and introduces the M4A1 evaluation dataset, improving latency, cost, and response quality.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages16

Reading Time40 min

Introduction

Research Questions

Device-Cloud Collaborative LLM Inference with Multi-Modal, Multi-Task, Multi-Turn Conversations Liangqi Yuan, Graduate Student Member, IEEE, Dong-Jun Han, Member, IEEE, Shiqiang Wang, Fellow, IEEE, and Christopher G. Brinton, Senior Member, IEEE arXiv :2502 . 1 1007v 5 [ cs .LG] 10 Jul 2026 Abstract—Compared to traditional machine learning models, recent large language models (LLMs) can exhibit multitask-solving capabilities through multi-modal data sources and multi-turn conversations. These unique characteristics of LLMs, together with their large model size, make their deployment more challenging. Specifically, (i) deploying LLMs on devices faces computational, memory, and energy resource issues, while (ii) deploying them in the cloud cannot guarantee real-time service and incurs communication/usage costs. In this paper, we design TMO, a device-cloud LLM inference system with Three-M Offloading: Multi-modal, Multi-task, and Multi-turn conversation. TMO incorporates (i) a lightweight on-device LLM that can process simple tasks at high speed and (ii) a largescale cloud LLM that can handle multi-modal data sources. We develop a resource-constrained reinforcement learning (RCRL) strategy for TMO that optimizes the inference location (i.e., device [vs. cloud](vs. cloud)) and multi-modal data sources to use for each task in multi-turn conversations, aiming to maximize the longterm reward (response quality, latency, and usage cost) while adhering to resource constraints. We also contribute M4A1, a new dataset we curated across multiple modalities, tasks, conversation turns, and LLM configurations, enabling evaluation of offloading decisions. We demonstrate the effectiveness of TMO compared to several exploration-decision and LLM-as-Router baselines, showing significant improvements in latency, cost, and response quality. Our code and dataset are available at [https://github.com/liangqiyuan/TMO](https://github.com/liangqiyuan/TMO). Index Terms—Large Language Model, Reinforcement Learning, Multi-modal, Collaborative Inference, Resource Constraint. I. INTRODUCTION Large language models (LLMs) have demonstrated remarkable general-purpose task-solving capabilities across various fields and have been gradually incorporated into daily life [2]–[5] . The development of LLMs is thriving, yet deploying This work was supported in part by the National Science Foundation (NSF) under grant CPS-2313109, by the Office of Naval Research (ONR) under grant N00014-23-C-1016, by the Air Force Office of Scientific Research (AFOSR) under grant FA9550-24-1-0083, by the Institute of Information & communications Technology Planning & Evaluation (IITP) from the Korea government (MSIT) (No. RS-2024-00457882, AI Research Hub Project), and by NVIDIA’s Academic Grant Program. An earlier version of this paper was presented at the Twenty-sixth International Symposium on Theory, Algorithmic Foundations, and Protocol Design for Mobile Networks and Mobile Computing (MobiHoc 2025) [1], where it received the Best Paper Award Runner-up. L. Yuan and C. G. Brinton are with the School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA. E-mail: [liangqiy@purdue.edu](liangqiy@purdue.edu); [cgb@purdue.edu](cgb@purdue.edu) D.-J. Han is with the Department of Computer Science and Engineering, Yonsei University, Seoul, [South Korea. E-mail: djh@yonsei.ac.kr](South Korea. E-mail: djh@yonsei.ac.kr) S. Wang is with the Department of Computer Science, University of Exeter, EX4 4RN, UK. E-mail: [shiqiang.wang@ieee.org](shiqiang.wang@ieee.org) Fig. 1. Application Scenario for the TMO System in Kitchen Activity Assistance. Given a user prompt, the decision engine selects between the on-device and cloud LLMs and chooses the relevant data modalities based on the current task and conversation history. For the query “Where did I put my dishes?”, first-person and overhead views are uploaded to the cloud LLM, which the text-o

FAQs

What deployment challenges does the paper identify for large language models?

On-device inference faces computational, memory, and energy limits, while cloud inference cannot guarantee real-time service and increases communication and usage costs.

What is TMO and how does it work?

TMO is a device-cloud LLM inference system that performs Three-Way Offloading across multi-modal data, multi-task handling, and multi-turn conversations, using a lightweight on-device LLM for simple tasks and a large-scale cloud LLM for richer multi-modal processing.

How does the system choose where to run inference and which modalities to use?

It uses a resource-constrained reinforcement learning strategy to optimize inference location (device vs. cloud) and the relevant multi-modal data sources per task across multi-turn conversation history, maximizing long-term reward under resource constraints.

Jul 16, 2026

Upload by Logic