From GUI Tests to Conversational Interaction: A New Perspective on App-Specific Voice Assistants

Jul 16, 2026

Mobile voice assistants are often implemented as system-level services that do not align with application-specific behavior, forcing developers to manually recreate app logic for app-level voice interaction and increasing cost. The work proposes an LLM-driven method that reuses GUI test code as behavior-preserving, executable specifications. Large language models reinterpret GUI tests into app-specific voice artifacts, grounding assistants in existing application logic. A prototype on Android (AppVA) shows preliminary results across five open-source applications, suggesting GUI tests can support synthesized app voice assistants beyond testing.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages5

Reading Time13 min

Abstract

Introduction

From GUI Tests to Conversational Interaction: A New Perspective on App-Specific Voice Assistants Xue Qin [xue.qin@villanova.edu](xue.qin@villanova.edu)[ ](xue.qin@villanova.edu)Villanova University Villanova, Pennsylvania, USA Sumesh Surendran Letha [ssuren01@villanova.edu](ssuren01@villanova.edu)[ ](ssuren01@villanova.edu)Villanova University Villanova, Pennsylvania, USA arXiv :2607 . 1 1387v 1 [ cs . SE] 13 Jul 2026 Abstract Voice assistants are widely deployed on mobile platforms, yet most are designed as system-level services that remain poorly aligned with application-specific behavior. As a result, enabling voice interaction at the app level requires developers to manually reimplement application logic, leading to high development and maintenance costs. We propose an LLM-driven approach to automating the development of app-specific voice assistants by repurposing GUI test code, which encodes behavior-preserving, executable specifications of application functionality. In this paper, we present a perspective in which large language models reinterpret GUI tests as bridges between application behavior and conversational interaction. By transforming test methods into app-specific VA artifacts, such as voice intents, capability descriptions, and executable action plans, our approach grounds voice assistants directly in existing application logic rather than external specifications. We illustrate this vision through AppVA, a research prototype on Android. Our preliminary results across five open-source applications suggest that GUI test code can be reused beyond testing, enabling the synthesis of app-specific voice assistants and highlighting a broader research direction at the intersection of software testing, interaction design, and LLM-enabled automation. CCS Concepts • Software and its engineering → Software notations and tools. Keywords Voice Assistant, Test Code Reusing, Code Generation, Accessibility Development, LLM ACM Reference Format: Xue Qin and Sumesh Surendran Letha. 2026. From GUI Tests to Conversational Interaction: A New Perspective on App-Specific Voice Assistants. In 34th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE Companion ’26), July 05–09, 2026, Montreal, QC, Canada. ACM, New York, NY, USA, 5 pages. [https://doi.org/10.1145/3803437.3805573](https://doi.org/10.1145/3803437.3805573) This work is licensed under a Creative Commons Attribution-NonCommercialNoDerivatives 4 .0 International License. FSE Companion ’26, Montreal, QC, Canada © 2026 Copyright held by the owner/author(s) . ACM ISBN 979-8-4007-2636-1/2026/07 [https://doi.org/10.1145/3803437.3805573](https://doi.org/10.1145/3803437.3805573) 1 Introduction Voice assistants (VA) are becoming a dominant interaction modality on mobile devices, with over 150 million users [11] in the U.S. alone. As user expectations shift toward conversational interaction [18], developers face increasing pressure to integrate voice capabilities into millions of existing mobile applications. A common industrial approach to adding voice-assistant (VA) capabilities to mobile applications relies on embedded, generalpurpose VA services. Platforms such as Amazon Alexa require developers to reconstruct application functionality as external “skills” [2] triggered by predefined voice commands. This approach forces developers to maintain a parallel interaction model that mirrors application logic, leading to growing divergence between voice workflows and application behavior as apps evolve. Prior studies report declining adoption and persistent challenges [3–5], reflecting a deeper limitation: system-level VAs are not grounded in applicationspecific workflows. In contrast, research in the Human–Computer Interaction (HCI) community has explored Voice User Interfaces (VUIs) for mobile apps. A VUI enables interaction through natural language commands, e.g., saying “start

FAQs

为什么传统移动端语音助手在应用层难以直接落地？

传统方案多依赖系统级通用语音服务，需要开发者把应用能力重构成外部“技能”，并维持平行的交互模型。随着应用演进，这会造成语音工作流与应用行为逐步分离。

文中提出的核心思路是什么？

作者提出用基于LLM的自动化方式复用GUI测试代码。GUI测试代码被视为行为保持、可执行的应用规格说明，LLM将测试方法转化为与应用对齐的语音意图、能力描述与可执行行动计划等语音助手工件。

AppVA原型如何验证该愿景？

研究实现了Android上的AppVA原型，并在五个开源应用上进行了初步实验。结果显示GUI测试代码不仅能用于测试，也可用于合成应用特定语音助手，从而指向软件测试、交互设计与LLM自动化结合的更广研究方向。

Jul 16, 2026

Upload by Genevieve