When Machines Speak - A Unified Generative Framework for Integrating Machine-Native Symbols into Pretrained Large Language Models

Sep 15, 2026

Uploaded by Riley

Many AI systems encode entities, behaviors, and structured information using discrete machine-native symbols rather than natural language. This leaves a gap between language modeling and structured prediction because machine-native codes are outside the token space of pretrained LLMs. UniLang introduces a unified generative framework that extends pretrained LLMs to treat machine-native symbols as first-class generative units alongside text tokens. The approach expands vocabulary and embeddings and jointly models both token types with a single autoregressive objective. Experiments on sequential recommendation and legal precedent prediction show consistent improvements over strong baselines, indicating a path to unified generative modeling beyond language.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Abstract

Introduction

arXiv:2608.19529v1]cs.CL] 20 Aug 2026 # When Machines Speak:A Unified GenerativeFramework for Integrating Machine-Native Symbolsinto Pretrained Large Language Models Su Yan Rakesh Iyer Google Inc. Google Inc. sueyan@google.com rni@google.com ## Abstract Many real-world AI systems represent entities,behaviors,and strucured informa-tion using discrete machine-native symbols rather than natural language.Whilethese representations are compact and preserve task-relevant structure,they lieoutside the linguistic token space of pretrained large language models (LLMs),creating a fundamental divide between language modeling and structured predic-tion.We introduce UniLang,a unified generative framework that bridges thisdivide by extending pretrained LLMs to treat machine-native symbols as first-classgenerative units alongside natural-language tokens.UniLang expands the LLM'svocabulary and embedding space with grounded machine-native representations,enabling textual and symbolic tokens to be jointly modeled and generated under asingle autoregressive objective.This unified interface allows pretrained LLMs todirectly operate on machine-native representations without requiring them to beverbalized as natural language or relying on task-specific architectures.We evaluateUniLang on two structurally distinct tasks,sequential recommendation and legalprecedent prediction,spanning different domains and types of structured prediction.Across both tasks,UniLang consistently outperforms strong baselines,demonstrat-ing a path toward extending pretrained LLMs beyond language and using themas a common generative modeling backbone for heterogeneous machine-nativerepresentations. ## 1 Introduction Large language models(LLMs)have emerged as a general modeling framework across a wide range ofapplications.Their success stems from a unified autoregressive formulation in which diverse natural-language tasks are expressed through a common vocabulary and generation objective.However,thisunified interface is fundamentally linguistic:pretrained LLMs operate over natural-language tokens. Many modern AI systems,in contrast,represent information using machine-native symbolic repre-sentations rather than natural language.These representations increasingly take the form of discretecodes produced by vector quantization or related techniques,representing complex entities such ascompressed audio signals |37l,semantic identifiers in recommendation systems I926,and struc-tured relational information in graph learning I20].Such representations preserve explicit discretestructure and offer computational efficiency that natural language cannot easily replicate.However,they remain outside the linguistic token space of pretrained LLMs,making them difficult for LLMsto natively model and generate. This mismatch creates a methodological divide.Existing approaches largely follow one of twoparadigms.The first treats natural language as a universal interface,converting structured informa-tion into textual descriptions so that pretrained LLMs can process it I5.1316,19,38,39].Whileeffective,verbalization may obscure native structure,introduce representational ambiguity,and sacrifice machine-level precision.The second operates directly on machine-native representationsusing task-specific models [14,26,36.40].Although these approaches preserve structural fidelity,they do not directly leverage the linguistic and world knowledge encoded in a pretrained LLM withinthe same generative space. At a deeper level,the limitation is representational.Pretrained LLMs lack a unified interface thatallows machine-native symbols to function as first-class generative units alongside natural-languagetokens.Without such an interface,structured symbolic prediction and language modeling remainseparate paradigms. This observation motivates a simple question: Can pretrained LLMs be extended to directly model and generate machine-nativesymbols,rather than forcing symbolic information

FAQs

What problem does UniLang address in pretrained LLMs?

UniLang addresses the mismatch between pretrained LLM token spaces and discrete machine-native symbolic representations, which are difficult to model or generate natively within LLMs. It aims to bridge language modeling and structured prediction under one framework.

How does UniLang integrate machine-native symbols into an LLM?

UniLang extends the LLM vocabulary and embedding space and grounds machine-native symbols into the pretrained representation space. It then enables joint generation of natural-language tokens and machine-native symbols using a single autoregressive objective.

Which tasks are used to evaluate UniLang, and what is the outcome?

UniLang is evaluated on sequential recommendation and legal precedent prediction, chosen for structural heterogeneity. Results show it consistently outperforms strong baselines while using the same modeling framework across both tasks.

Sep 15, 2026

Uploaded by Riley