Voltron Enabling Elastic Multi-Device Execution of LLM Inference for Empowered Edge Intelligence

Jul 16, 2026

Upload by Skyler

Large language models (LLMs) power intelligent services, but centralized inference in data centers introduces response latency from communication overhead and creates privacy leakage risks. Deploying LLM inference on edge devices reduces latency and can improve personalization, yet single-device memory limits restrict the achievable model size and accuracy. Voltron proposes elastic multi-device on-device inference that leverages multiple nearby user-edge devices to run larger models while adapting to heterogeneous, real-world edge conditions, achieving up to 16.5% higher accuracy while meeting user QoS requirements.

LanguageEnglish

CategoryTechnology

FormatPDF

Pages16

Reading Time40 min

Abstract

Introduction

Centralized LLM inference limitations

Motivation for edge execution

Challenges in multi-device edge inference

Heterogeneity and runtime variance

Voltron: Enabling Elastic Multi-Device Execution of LLM Inference for Empowered Edge Intelligence Chanwoo Cho [chanwoo_cho@korea.ac.kr](chanwoo_cho@korea.ac.kr)[ ](chanwoo_cho@korea.ac.kr)Korea University Seoul, Republic of Korea Wooseok Kim [ws8462@korea.ac.kr](ws8462@korea.ac.kr) Korea University Seoul, Republic of Korea Yonglak Son [yonglak_son@korea.ac.kr](yonglak_son@korea.ac.kr) Korea University Seoul, Republic of Korea Young Seo Lee∗ [yslee@ssu.ac.kr](yslee@ssu.ac.kr)[ ](yslee@ssu.ac.kr)Soongsil University Seoul, Republic of Korea Young Geun Kim∗ [younggeun_kim@korea.ac.kr](younggeun_kim@korea.ac.kr)[ ](younggeun_kim@korea.ac.kr)Korea University Seoul, Republic of Korea arXiv :2607 .07046v 1 [ cs .DC] 8 Jul 2026 Abstract Large language models (LLMs) are widely used in intelligent services due to their remarkable capability in generative tasks. Typically, LLM-based services process the inference requests ofthe users in a centralized data center. Unfortunately, such centralized execution has limitations for end-users, such as increased response latency with communication overhead and privacy leakage risk. To alleviate the aforementioned limitations, there have been increasing pushes to execute LLM inference locally on user-end devices. However, the limited resources of a single edge device impose restrictions on achievable accuracy of LLMs. To overcome the issue, we first propose to leverage multiple user-end devices available atthe edge for LLM inference, enabling the execution of larger models. Specifically, we propose Voltron, a novel on-device LLM inference framework that elastically utilizes multiple user-end devices for LLM inference execution while adapting to diverse real-world edge environments. In our evaluation, Voltron achieves up to 16.5% higher accuracy than state-ofthe-art LLMs that can be executed on a single edge device, satisfying user QoS requirements. 1 Introduction Large language models (LLMs) have shown a remarkable capability in generative tasks such as question answering, text generation, and summarization. Many intelligent services are utilizing the capability of LLMs as a key functionality. For example, personal assistants (e.g., Siri [7] and Alexa [6]) use LLMs to improve their conversational abilities, AI chatbots (e.g., ChatGPT [11] and Gemini [18]) provide information through question-and-answer interactions using LLMs, and personal agents (e.g., OpenClaw [54], ManusAI [26]) rely on LLMs to autonomously plan and execute complex user tasks. ∗ Co-corresponding author. Typically, LLM-based services batch inference requests from tens of millions of users in a centralized data center [34, 59, 60] . To accelerate the large volume of inference requests, state-of-the-art inference serving frameworks such as vLLM [34] are used to maximize the size of batched LLM inference requests. However, such centralized execution has several limitations for end-users. First, the response latency can be prolonged by communication overhead depending on wireless network conditions and congestion in core network, which adversely impacts a user QoS — the communication overhead can further be increased depending on the subscription option of the service [19, 48] . Second, the inference requests may contain user private data, posing a security concern [61] . To alleviate the aforementioned limitations, there have been increasing pushes to execute LLM inference locally on user-end devices by leveraging the available computational resources recently featured in mobile SoCs — on-device execution ofLLM inference is expected to open up new personalized applications, such as smart home agent [6], personal AI robot [1], etc. However, it is still not feasible to execute large models whose required memory is beyond the device memory capacity. To enable on-device LLM, many researchers have focused on designing smaller LLMs (sLLMs)(e.g., MobileLLM [39] and TinyLlama [65]) or applying model comp

FAQs

Why is centralized LLM inference insufficient for end users?

Centralized execution increases response latency due to communication overhead and can also raise privacy leakage concerns when inference requests include user private data.

What problem does Voltron address with single-device edge execution?

Voltron targets the memory constraint of a single edge device, which limits the feasible model size and thus accuracy when running LLM inference locally.

How does Voltron enable better accuracy at the edge?

Voltron uses an elastic multi-device on-device inference framework that leverages multiple nearby edge devices to execute larger LLMs and adapt to diverse edge environments, improving accuracy by up to 16.5% over single-device baselines.

Jul 16, 2026

Upload by Skyler