Workload-Driven Optimization for On-Device Real-Time Subtitle Translation

Jul 16, 2026

Upload by Aria

This report studies on-device English-to-Traditional-Chinese subtitle translation for Taiwan under short inputs, short outputs, batch-size-one inference, low latency, and privacy constraints. These conditions reduce the value of optimizations for long-context or high-throughput serving. Starting from LMT-60-0.6B, the work finds vocabulary projection becomes a key decode-time cost post-quantization, then replaces the 151k vocabulary with a 64k subtitle-domain tokenizer, migrates embeddings, and performs embedding calibration plus supervised fine-tuning.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Abstract

Introduction

Task and Design Goals

Cue-level context-aware translation

On-device constraints

Method

Workload profile and decode cost

64k-vocabulary subtitle tokenizer

arXiv :2607 .09957v 1 [ cs .CL] 10 Jul 2026 Workload-Driven Optimization for On-Device Real-Time Subtitle Translation Tsz-To Wong National Yang Ming Chiao Tung University, Hsinchu, Taiwan[tsztowong. cs13@nycu. edu. tw](tsztowong. cs13@nycu. edu. tw) Abstract This report studies on-device English-to-Traditional-Chinese subtitle translation for Taiwan under short inputs, short outputs, batch-size-one inference, low latency, and privacy constraints. These conditions limit the value of optimizations designed for long-context or high-throughput language-model serving. Starting from LMT-60-0.6B [2], preliminary profiling suggests that vocabulary projection becomes a more important decode-time cost after GGUF quantization reduces the relative cost of Transformer blocks. We replace the original 151k-token vocabulary with a 64k-token subtitle-domain tokenizer, migrate the embedding space, and adapt the model through embedding calibration followed by full supervised fine-tuning. On a fixed 500-example subset of the OpenSubtitles2024 test set, the LocalSubs achieves a 59.2% tie-excluded win rate against Google Translate under GPT-4o pairwise judging. Performance is strongest on short cues and declines as cue length increases. Preliminary Apple M2 Metal measurements on a 64k-vocabulary model show a 1.63 × speedup over a 151k-vocabulary profiling baseline. The raw benchmark configuration is incomplete, so the latency result is treated as preliminary. 1 Introduction Real-time subtitle translation imposes a different set of constraints from conventional machine translation and general-purpose language-model serving. Each request typically contains one current subtitle cue and only a small amount of preceding context. Both the input and output are short, inference is effectively performed at batch size one, and the translation must be produced before the subtitle display window expires. For privacy-sensitive applications, the complete pipeline must also run on local consumer hardware while generating natural Traditional Chinese as used in Taiwan. These characteristics change the inference bottleneck. Techniques designed for long contexts, repeated prefixes, or large batches offer limited benefit when prompts are short and requests arrive sequentially. In this setting, fixed per-request overhead, decode-time computation, vocabulary projection, and output token count can have a greater effect on end-to-end latency. A large multilingual tokenizer may further increase the output-projection cost while providing inefficient tokenization for domain-specific Chinese subtitle text. This work investigates a workload-driven optimization path based on LMT-60-0.6B [2] . The original 151k vocabulary is replaced with a 64k subtitle-domain tokenizer trained on English and Taiwan Traditional Chinese subtitle text. Because tokenizer replacement changes token identities and invalidates the original embedding matrix, the model is adapted through embedding migration, an embedding-calibration stage, and full supervised fine-tuning. The resulting system is evaluated using pairwise preference judgments against a fixed Google Translate anchor, while deployment performance is examined through preliminary Apple M2 Metal measurements. The main contributions are: • A 64k-vocabulary subtitle tokenizer that reduces the output-projection dimension and improves Chinese token density. • An embedding migration and two-stage adaptation procedure for replacing the tokenizer of apretrained translation model. • A translation-quality evaluation based on pairwise preference judgments, including context ablation and cue-length analysis. 2 Task and Design Goals 2.1 Cue-level context-aware translation The model receives up to three previous English subtitle cues as context and one current cue asthe translation target. It must output only the Taiwan Traditional Chinese translation of the current cue. Context is used for disambiguation and tone continuit

FAQs

Why do conventional LLM serving optimizations offer limited benefit for real-time subtitle translation?

Real-time subtitle translation uses short prompts and outputs, effectively batch size one, and requires completion within the subtitle display window. In this setting, fixed per-request overhead and decode-time costs (especially vocabulary projection and output tokens) dominate end-to-end latency.

What main bottleneck does the report identify after GGUF quantization?

After GGUF Q5 K M quantization reduces the relative cost of Transformer blocks, the output projection (vocabulary projection) becomes a more important decode-time component because each decode step computes logits over the full vocabulary.

How does the proposed system adapt the model when replacing the vocabulary with a 64k subtitle tokenizer?

Replacing the tokenizer changes token identities and invalidates the original embedding matrix, so the report adapts via embedding migration, an embedding-calibration stage, and then full supervised fine-tuning using the subtitle domain.

Jul 16, 2026

Upload by Aria