Knowing-how & Knowing-that: A New Task for Machine Reading Comprehension of User Manuals

Oct 22, 2026

Uploaded by jayni

Machine reading comprehension of user manuals supports customer service, yet current models struggle with complex questions that require more than direct fact lookup. The work introduces the Knowing-how & Knowing-that task, requiring answers to factoid-style, procedure-style, and inconsistent questions. It proposes representing steps and facts jointly in a heterogeneous graph (TARA) for unified inference. A heuristic system parses manuals into TARAs and builds an annotated dataset to benchmark real-world question answering.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

AccessFree to download

Introduction

Motivation and customer-service setting

Limitations of existing MRC models

Proposed task and representation overview

Knowing-how & Knowing-that: A New Task for Machine Reading Comprehension of User Manuals Hongru Liang1 Jia Liu2 Weihong Du1 dingnan jin2 Wenqiang Lei 1 ∗ Zujie Wen2 Jiancheng Lv 1 1 College of Computer Science, Sichuan University 2Ant Group, China {lianghongru, wenqianglei, [lvjiancheng}@scu.edu.cn](lvjiancheng}@scu.edu.cn)[ ](lvjiancheng}@scu.edu.cn)[duweihong@stu.scu.edu.cn](duweihong@stu.scu.edu.cn) {jianiu.lj, dingnan.jdn, [zujie.wzj}@antgroup.com](zujie.wzj}@antgroup.com) Abstract The machine reading comprehension (MRC) of user manuals has huge potential in customer service. However, current methods have trouble answering complex questions. Therefore, we introduce the Knowing-how & Knowingthat task that requires the model to answer factoid-style, procedure-style, and inconsistent questions about user manuals. We resolve this task by jointly representing the sTeps and fActs in a gRAph (TARA), which supports a unified inference of various questions. Towards a systematical benchmarking study, we design a heuristic method to automatically parse user manuals into TARAs and build an annotated dataset to test the model’s ability in answering real-world questions. Empirical results demonstrate that representing user manuals as TARAsis a desired solution for the MRC of user manuals. An in-depth investigation of TARA further sheds light on the issues and broader impacts of future representations of user manuals. We hope our work can move the MRC of user manuals to a more complex and realistic stage. 1 Introduction User manuals are supposed to be helpful in using products, getting involved in promotions, or other goals of interest (Ryle, 2009 ; Chu et al., 2017 ; Bombieri et al., 2021) . Though well-designed and instructive, they are seldom read by users because“Life is too short to read manuals”(Blackler et al., 2016) . Towards high user satisfaction, professional customer service representatives (CSRs) are hired to do the reading and answer user questions about the manuals. The machine reading comprehension (MRC) of user manuals thus has huge potential, as it would not only reduce high labor costs but enable the service ready for customers 24/7 . Let’s see how a CSR approaches various questions given the user manual 1 in Figure 1. Specifi- ∗ Corresponding author 1We make a few modifications (e.g., removing the application name) to this real-world user manual for anonymity. Figure 1: A user manual and three QA cases about it cally, she can answer Q1 by looking up the arguments of facts, i.e., the value ( 100% ) of the hit rate. For Q2, she needs to reason about the steps in the procedure as the answer is not explicitly exhibited in the user manual—the user manual directs the user to “Sign in” and “Scan”, and eventually to the “payment page”. After that, the subsequent description does not direct the user anywhere else. A user can infer from the procedure that one can scratch the card after immediately getting it on the payment page. Therefore, the location to scratch the card is on the payment page . The most difficult one is Q3, which raises a case inconsistent with the descriptions in the user manual—going along with the user manual (S-4), the user didn’t get the scratch card. We call this type of question an inconsistent question. A possible response to 10550 Findings of the Association for Computational Linguistics: ACL 2023 , pages 10550–10564 July 9-14, 2023 ©2023 Association for Computational Linguistics the inconsistent question may be an answer (R1) or a high-utility question leading to an answer (R2-R4) . Towards such responses, the CSR needs to reason about not only the steps but the facts binding the steps—the inconsistency may be caused by incorrect operations in the previous step (e.g., the user didn’t pay by scanning the QR code ) orunsatisfiable constraints in the facts (e.g., the user already had 10 scratch cards ) . Therefore, aligned with the CSR, a practical MRC model must also

FAQs

What problem does the document address in machine reading comprehension of user manuals?

Existing methods struggle to answer complex questions from manuals, especially procedure-style and inconsistent questions that cannot be resolved by simple fact lookup.

What is the Knowing-how & Knowing-that task?

Given a user manual, the model must answer factoid-style, procedure-style, or inconsistent questions about the manual.

How does the proposed TARA approach resolve different question types?

TARA jointly represents steps and facts in a heterogeneous graph, enabling unified inference across step and fact reasoning for various question categories.

Oct 22, 2026

Uploaded by jayni