GemNav Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model

Jul 16, 2026

Upload by Sage

GemNav presents a data-efficient visual robot navigation policy that adapts a frozen Multimodal Large Language Model (MLLM) for short-to-medium horizon waypoint navigation using Low-Rank Adaptation (LoRA) only on the language tower. The method avoids a dedicated visual encoder and continuous regression head by using a single discrete token vocabulary shared by waypoints and categorical navigation signals. A soft-decoded auxiliary loss restores metric waypoint structure. Trained on an 8.7-hour open corpus, it transfers zero-shot across four unseen real environments, reaching goals within 0.25–0.42 m in 20 trials, with limited gains from short image histories.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

Introduction

Waypoint navigation setting

GemNav approach

Contributions and evaluation

arXiv :2607 .06882v 1 [ cs .RO] 8 Jul 2026 GemNav: Discrete-Token Visual Robot Navigation using a Multimodal Large Language Model Peter BohmSaimunur Rahman, Abdelwahed Khamis, Sagun Man Singh Shrestha, Chris McCool, Peyman Moghadam CSIRO Technology, Australia Abstract: Visual navigation policies built on large pretrained models have so far followed a common recipe: a dedicated visual encoder, a bespoke action head, and training on thousands of hours of cross-embodiment datasets. We ask whether this recipe is necessary. In this paper, we introduce GemNav, a visual robot navigation policy that adapts a frozen Multimodal Large Language Model (MLLM) for short-to-medium horizon waypoint navigation using Low-Rank Adaptation (LoRA) on the language tower alone, with no auxiliary visual encoder and no continuous regression head. Waypoints and categorical navigation signals share a single discrete token vocabulary generated by the language-model head, and a soft-decoded auxiliary loss recovers the metric structure that pure cross-entropy training discards. On a single 8.7-hour open corpus, roughly three orders of magnitude smaller than competing training sets, the policy transfers zero-shot to four physically distinct unseen environments and stops within 0.25–0.42 m of the goal across 20 real-world trials covering an open carpark, an obstacle carpark, a long outdoor chemical yard, and an indoor warehouse. Conditioning on short image histories improves offline metrics but yields no robot benefit, pointing to a ceiling on what temporal context adds once pretrained vision features are in place. These results indicate that discrete-token adaptation of frozen MLLMs can provide a data-efficient, deployable alternative for foundation model robot navigation. Keywords: Visual Navigation, Data-Efficient Robot Learning, Multimodal Large Language Model (MLLM) 1 Introduction Robot navigation has traditionally been built around a tightly coupled stack of perception, state estimation, and control, tuned for a particular robot, sensor setup, or environment [1] . Large pretrained vision-language-action models have shifted this framing, treating navigation as a cross-embodiment learning problem: a single policy trained on hundreds to thousands of hours of trajectories pooled across robots, environments, and sensor configurations [2, 3, 4, 5, 6] . Recent visual navigation policies built on large pretrained models follow a common recipe: a strong visual backbone (often a separate dedicated encoder) is paired with a continuous-action regression head and supervised on a union of multiple navigation datasets collected across embodiments [5, 4, 7] . Yet the scalability of this approach raises a deeper question about where the true source of generalization lies. These policies achieve expansive transferability, but their success has come bundled with an ever-growing need for large demonstration data, specialized action heads, and visual encoders trained from scratch on robotic data. Recent benchmarking further suggests that the visual encoder is the dominant bottleneck for downstream VLA performance [8, 9], pointing toward a trajectory of increasing visual scale as the primary source for improvement. In this paper, we take a different view. The pretraining of modern multimodal large language models [10, 11, 12, 13] has ∗ Corresponding author: [peter.bohm@csiro.au](peter.bohm@csiro.au) already produced visual representations of remarkable breadth and robustness; the question we pose is whether those representations, held frozen, are sufficient for robot navigation, and if the gap between a general-purpose Multimodal Large Language Model (MLLM) and a deployable navigation policy can be closed by rethinking the action representation rather than by scaling perception further. We ground this question in short-to-medium-horizon waypoint navigation: a higher-level planner supplies the next local goal as a 2D pose, a goal image, or both, and the

FAQs

What core idea does GemNav introduce for visual robot navigation?

GemNav adapts a frozen multimodal large language model to waypoint navigation by predicting discrete action tokens, instead of relying on a dedicated visual encoder and a continuous regression head.

How does GemNav represent waypoints and navigation signals?

Waypoints and categorical decisions (e.g., goal reached/unreachable) share a unified discrete token vocabulary produced by the language-model head in a single token sequence.

What evidence is provided for transfer and real-world performance?

Trained on a single 8.7-hour open dataset, the policy transfers zero-shot to four unseen, physically distinct environments and stops within 0.25–0.42 m of the goal across 20 real-world trials.

Jul 16, 2026

Upload by Sage