Task Planning for Mobile Manipulation in Retail Stores using Foundation Models with Iterative Re-planning

Jul 16, 2026

Upload by Noah

Automation for retail, warehousing, and logistics can increase throughput, reduce costs, and limit disruption from labor shortages, but existing approaches often assume structured back-room environments. This work applies foundation-model methods to variable, human-centric retail shelves by introducing an LLM- and VLM-based task-planning system for restocking. The framework uses user prompts and execution-feedback-driven iterative re-planning for error correction, validated end to end in a PyBullet simulation for pick-and-place tasks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages3

Reading Time8 min

Introduction

Method

Task Planning for Mobile Manipulation in Retail Stores using Foundation Models with Iterative Re-planning Vismay Vakharia∗ , Sanjana Garai, Rolif Lima, Nijil George, Vighnesh Vatsal, Kaushik Das arXiv :2607 .09962v1 [ cs .RO] 10 Jul 2026 Abstract—Automation in industries such as retail, warehousing and logistics presents opportunities for greater throughput, cost reduction and mitigation of disruptions from labour shortages. Previously, such efforts have focused on back-room operations involving packing and sorting in relatively structured environments. With advances in robotic mobile manipulation hardware and foundation models, automation can now be applied to more variable and human-centric environments such as retail store shelves. In this work, we present a task-planning approach using Large Language Models (LLMs) and VisionLanguage Models (VLMs) to address the restocking problem in retail scenarios such as supermarkets. We demonstrate this system on a custom omnidirectional mobile manipulation platform, with user-driven prompts and a feedback-based iterative replanning approach for error correction. The end-to-end system is validated in a PyBullet simulation environment for pick-andplace tasks. I. INTRODUCTION The modern retail industry is rapidly adopting roboticsenabled solutions, driven by the competitive nature of the sector and labour shortages, particularly in developed countries. Automating a retail store using mobile robots presents significant challenges, as evidenced by international robotics competitions such as the ”Amazon Picking Challenge” [1] and ”Future Convenience Store Challenge” [2] . The robots tasked with order picking, restocking, and organizing must decompose high-level goals into sequenced sub-tasks and generate efficient motion plans – all while navigating realworld uncertainty [3] . Traditionally, this has been achieved using Task and Motion Planning (TAMP) [4], [5] frameworks, that rely on symbolic reasoning that is manually integrated with continuous motion control. However, these methods are often domainspecific and rigid in their operation. Recent work shows that Large Language Models (LLMs) can transform TAMP by replacing rigid rule-based systems with flexible, general-purpose reasoning [6], [7] . Pretrained on vast internet-scale text corpora using masked language modelling and autoregressive prediction objectives, LLMs can infer structured task sequences without the need for fine-tuning [8], adapt to environmental context, and even recover from failures without domain-specific engineering or retraining [9] . In this study, we explore this new paradigm using a custom-built, omnidirectional, dual-arm mobile manipulator [10],[11] . The system is tasked with ‘order picking’ in a simulated retail environment, where it must retrieve items The authors are with TCS Research, Tata Consultancy Services Ltd., Bengaluru - 560066, Karnataka, India. ∗Corresponding author, e-mail: [vismay.vakharia@tcs.com](vismay.vakharia@tcs.com) Fig. 1: Framework Architecture from a given list in their respective quantities efficiently based on store layout. Our framework combines an LLM for high-level logical reasoning with a VLM for spatial understanding, using execution feedback to continually refine its plans. The proposed framework is tested in a PyBullet simulation environment. II. METHOD The proposed framework is initialized by the user inputting the query in natural language (English) to the LLM. Along with the query, a planogram containing the description of the environment and robot-specific information consisting of the feasible symbolic actions and their respective parameter sets (table I) are provided to the LLM. The first stage of real-world task execution involves creating a plan that outlines a sequence of actions. The LLM is prompted to generate a task sequence from the robot’s feasible actions to ensure that all the tasks are within the scope of the robot’s capabilities whi

FAQs

What problem does the proposed system address in retail stores?

It addresses restocking via mobile manipulation in retail scenarios such as supermarkets, including efficient retrieval and organization of items on store shelves.

How do LLMs and VLMs work together in the framework?

An LLM performs high-level logical task sequencing using feasible symbolic actions and an environment planogram, while a VLM supports spatial understanding during scanning to identify objects and select grasp-related details.

How is iterative re-planning used to handle errors?

Execution feedback is used to continually refine task plans, enabling the system to recover from failures and correct errors during operation.

Jul 16, 2026

Upload by Noah