Inducing Task Models from Computer-Use Traces - Research paper on TASK MODEL INDUCTION (TMI)

Sep 15, 2026

Uploaded by Sage

Naturalistic computer-use traces, including passively recorded screenshots and mouse or keyboard actions, enable deriving symbolic, auditable, and reusable task models for everyday work. Such models are increasingly needed as computer-use agents enter real organizational settings, but induction is difficult because only low-level events are observed and work is multi-threaded with interleaved goals. The proposed TASK MODEL INDUCTION (TMI) discovers latent tasks and induces for each a hierarchical objective model plus a control-flow procedure model, achieving strong recovery and step reconstruction.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages23

Reading Time58 min

AccessFree to download

Abstract

Introduction

Challenges in modeling computer-use traces

Prior approaches and limitations

Proposed task model definition and method (TMI)

Inducing Task Models from Computer-Use Traces Yucheng Jiang 1 Zora Zhiruo Wang2 Ruishi Chen 1 Diyi Yang 1 1 Stanford University 2 Carnegie Mellon University {yuchengj,[diyiy}@cs.stanford.edu](diyiy}@cs.stanford.edu) arXiv : 2608 .203 19v 1 [cs .CL] 20 Aug 2026 Abstract Naturalistic computer-use traces, passively recorded screenshots and mouse or keyboard actions, are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done. Such models matter as computer-use agents enter real work, where agents need to learn how tasks are actually performed, and organizations need to audit and reuse that knowledge. However, inducing such task models is challenging, as activity is observed only as low-level events and realworld work is multi-threaded with interleaved goals. Existing methods assume a given task ora single workflow, and produce step-level summaries rather than structured task models. We introduce TASK MODEL INDUCTION (TMI), which (i) discovers the latent tasks in an unconstrained trace, disentangling concurrent activity, and (ii) for each latent task, induces a task model pairing a hierarchical objective model of recursive goal decomposition with a procedure model of the control flow that organized the execution. Intrinsically, on controlled human and agent trajectories, TMI recovers interleaved tasks with 0.974 agreement against ground-truth groupings and reconstructs 74.9% of the observed execution steps, far more than the strongest workflow induction baseline. Extrinsically, skills derived from TMI’s task models improve held-out task accuracy by 30.0% over the strongest baseline. 1 1 Introduction Naturalistic computer-use activity traces, passively recorded sequences of screenshots and mouse or keyboard events, are a valuable resource for deriving symbolic, auditable, and reusable models of how everyday work is done (Shaikh et al., 2025 ; Wang et al., 2025) . Most of this work is never documented, and the expertise it encodes remains 1Our codebase is available at [https://github.com/](https://github.com/)[ ](https://github.com/)Yucheng-Jiang/task-model-induction. Figure 1: Task model induction from computer-use traces. Screenshots and input events are grounded into semantic actions, whose interleaved spans are assigned to latent tasks. Each task is represented as a hierarchy pairing objectives with control-flow operators. tacit. Recovering the objectives a user pursued and the procedures they followed turns a recording into an explicit account of what was accomplished and how, which people can audit for systematic patterns in human and AI work (Wang et al., 2025), reuse as documentation, and transfer to new practitioners. Beyond human use, such traces support agent learning from human activity without costly annotation (Lu et al., 2025 ; Song et al., 2026), and personalization, where activity-based user models let systems anticipate user goals (Shaikh et al., 2026) . Modeling these traces, however, poses three challenges. At the signal level, raw events such as cursor movements, key presses, and screen pixels carry little semantic meaning, and bridging perceptual observations to semantic intent requires substantial interpretation (Yang et al., 2023 ; You et al., 2024 ; Zheng et al., 2024 ; Shaw et al., 2023) . At the structural level, naturalistic work is multithreaded, with users switching among unrelated goals within a single session and pursuing interleaved sub-objectives within a single task (Czerwinski et al., 2004 ; González and Mark, 2004 ; Market al., 2008 ; Adamczyk and Bailey, 2004 ; Iqbal and Horvitz, 2007 ; Salvucci et al., 2009) . At the representational level, a trace captures the executed path but does not directly reveal either the goal hierarchy that motivated each step or the control flow that organized the execution (Stanton, 2006 ; Card, 2018 ; Diaper and Stanton, 2003) . Prior approaches address these challenges only partially. LLM s

FAQs

What problem does TASK MODEL INDUCTION (TMI) address?

TMI aims to induce structured task models from naturalistic computer-use traces, where goals are latent and activity is multi-threaded with interleaved objectives.

How does TMI represent each induced task?

For each latent task, TMI pairs a hierarchical objective model (recursive goal decomposition) with a procedure model describing execution control flow via structured sequencing and iteration operators.

What evidence supports TMI’s effectiveness?

TMI recovers interleaved tasks with high agreement versus ground-truth groupings, reconstructs a large fraction of observed execution steps, and improves held-out task accuracy by 30.0% over the strongest baseline using skills derived from its task models.

Sep 15, 2026

Uploaded by Sage