ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories

Jul 16, 2026

Upload by Aurora

Training capable OS agents requires data that unifies structured user intent, multi-turn task delegation, and grounded tool execution, yet existing datasets lack these properties. The proposed ISE (Intent → Simulate → Execute) introduces a three-stage synthesis paradigm addressing the gaps jointly. It builds 43,956 deduplicated structured intents, generates 23,132 role-locked multi-turn trajectories, and performs live isolated tool execution to capture genuine failure–recovery dynamics. Fine-tuning improves ClawEval pass@1 on Qwen3-8B from 19.3 to 37.7, releasing code and data.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages13

Reading Time33 min

Introduction

Abstract

Related Work

ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories Siyuan Luo 1,* Nairong Zheng 1,* Lin Zhou2,† Tiankuo Yao2,† Shengyou Yuan2,† Haojia Yu2 Cong Pang2 Jiapeng Luo2 Lewei Lu2,‡ 1University of Electronic Science and Technology of China 2 SenseTime Research [valierelane@gmail.com](valierelane@gmail.com) [luotto@sensetime.com](luotto@sensetime.com) arXiv :2606 . 1 1520v 3 [ cs .CL] 13 Jul 2026 Abstract Training capable OS agents requires data that simultaneously captures structured user intents, multi-turn task delegation, and grounded tool execution—properties absent from existing datasets. We propose ISE (Intent → Simulate → Execute), a three-stage synthesis paradigm that addresses these gaps jointly. Stage 1 constructs ∼50,000 structured intents via a 4D framework (Persona × Domain × Task × Complexity); after deduplication the pool contains 43,956 unique intents and attains a Vendi Score of 61.57 over the entire pool on mpnet-base-v2 embeddings (cosine kernel, q= 1) . Stage 2 drives multi-turn user–agent interaction through a role-locked user simulator that grounds each user turn in actual execution outcomes, producing 23,132 complete trajectories averaging 8.12 user turns and 68.24 total dialogue turns. Stage 3 executes every tool call in a live, isolated OS workspace, yielding authentic failure–recovery dynamics rather than simulated responses. Fine-tuning on ISETrace lifts ClawEval pass@1 from 19.3 to 37.7 on Qwen3-8B (agent tool-use tasks, common-denominator protocol), surpassing both a GPT-4o zero-shot reference and a 4 ×-larger Qwen3-32B base; a Stage 2 ablation indicates multi-turn simulation contributes a substantial share of the gain. We release all code and data at [https:](https:)//[github.com/Valiere01/ISE-Trace](github.com/Valiere01/ISE-Trace). 1 Introduction Large language model agents are increasingly deployed in stateful operating-system environments, yet the training data used to teach them still underrepresents four properties of real use: user intents * Co-first authors. † Core contributors. ‡ Corresponding author. are implicit and underspecified, actions have external side effects, users react to partial progress and failure, and successful completion is often verifiable only through environment state. Despite rapid progress in large language models, agents still fail on more than half of realistic multi-turn OS tasks (Yao et al., 2024), and the bottleneck isnot model capacity—it is training data. A closer look at current synthesis pipelines reveals three systematic structural gaps. Gap 1 (Intent-first bias): Most pipelines start from a list of available APIs or tools—e.g., the 16k+ REST endpoints on RapidAPI or a curated SDK catalog (Qin et al., 2023; Liu et al., 2024)—and back-derive tasks from each tool (“get_weather(city)” →“What’s the weather in Tokyo?”) . The resulting task distribution therefore mirrors the catalog rather than what users actually want; long-tail and crosstool intents are systematically under-represented. The natural alternative—asking an LLM to freegenerate user tasks—fares no better: instructiontuned LLMs exhibit a well-documented mode collapse toward the high-frequency phrasings they have seen most often (Wang et al., 2022) (algorithmic puzzles, generic email templates, customerservice openers), producing tasks that look diverse on the surface but cluster in a narrow region of intent space. Gap 2 (Single-turn bias): Nearly all OS agent datasets are single-turn (Sun et al., 2024; Xu et al., 2024), failing to capture the multi-turn task delegation, correction, and verification cycles central to real agent interactions. Even pipelines with user simulators (Prabhakar et al., 2025; Chen et al., 2026b) suffer from role drift—instruction-tuned LLMs gradually adopt assistant-style language—and state hallucination—simulators issue follow-up requests based on assumed states that diverge from actual OS state (Zhou et al., 2026) . Gap 3 (Simula

FAQs

What is ISE and what problem does it address for OS-agent training?

ISE is a three-stage synthesis paradigm (Intent → Simulate → Execute) designed to produce training data that captures structured user intents, multi-turn task delegation, and grounded tool execution—capabilities missing from existing datasets.

How are intents and trajectories constructed in ISE?

Stage 1 builds a large pool of structured intents using a 4D framework and deduplicates to 43,956 unique intents. Stage 2 uses a role-locked user simulator to generate 23,132 complete multi-turn trajectories with an average of 8.12 user turns.

How does ISE ensure tool execution is grounded and why does it matter?

Stage 3 executes every tool call in a live, isolated OS workspace, producing authentic failure–recovery dynamics instead of simulated responses. This grounding is used to improve evaluation performance after fine-tuning.

Jul 16, 2026

Upload by Aurora