Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection

Jul 16, 2026

Upload by Aurora

LLMs and agents can generate web scrapers from natural-language requirements, but free-form generation often breaks due to dependency errors, broken selectors, schema mismatches, and heterogeneous page structures. A constrained, verifiable agent framework shifts LLM output to typed JSON collector configurations, using a six-type taxonomy, template and utility-function constraints, static Airflow DAG execution, rule-based quality checks, and structured feedback correction. Experiments on 138 tasks show strong requirement-to-typing support and deterministic instantiation needs completion of source, field, and execution constraints.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages15

Reading Time38 min

Introduction

Related Work

Open-Web Data Collection

LLMs, Code Generation, and Constraints

arXiv :2607 .00035v 1 [ cs .AI] 25 Jun 2026 Making Failure Safe: A Constrained, Verifiable Agent Framework for Open-Web Data Collection Bo Chen Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China [chenbo01@ict.ac.cn](chenbo01@ict.ac.cn) Abstract. LLMs and agents can generate web scrapers from naturallanguage requirements, but direct generation remains unreliable because of dependency errors, broken selectors, schema mismatches, and heterogeneous page structures. We propose a constrained, verifiable agent framework that shifts LLM output from free-form code to typed JSON collector configurations, combining a six-type collector taxonomy, template and utility-function constraints, static Airflow DAG execution, rule-based quality checking, and structured feedback correction. Experiments on 138 tasks show that the taxonomy supports description-based requirement typing, while confirming that stable instantiation requires completing source, field, and execution constraints beyond the initial description. On 80 independently source-verified tasks, the framework runs with zero execution-stage LLM tokens and the lowest average wall-clock time, trading moderate one-shot quality for a reusable, deterministic, and verifiable execution path suited to repeated scheduled collection. These results position the framework as a reusable, low-cost, and verifiable execution path for repeated open-web data collection. Keywords: Data collection framework · Agent · Constrained generation · Data quality validation 1 Introduction Open-web data—publicly accessible information from news, government notices, e-commerce, and academic publications—drives growing demand for automated collection at scale. Traditional pipelines depend on manual effort: requirement analysis, site-structure inspection, scraper coding, and validation, suffering from long development cycles and low reusability. Recent LLM and agent advances offer a new path: understanding naturallanguage requirements, generating scraper code, and executing validation tasks. However, direct application proves unreliable. The core tension is between the structural brittleness of heterogeneous web pages and the stochastic inconsistency of LLM code generation. Without collector-type constraints, agents deviate in task decomposition; LLM-generated scrapers may contain dependency errors, broken selectors, and inconsistent output schemas. 2 B. Chen To address this, we argue that the key is not to pursue perfect agent reasoning, but to make the collection pipeline verifiable—so that every stage’s inputsand outputs can be checked, measured, and audited. We achieve this through three pillars: (1) typed modeling of collection tasks with explicit functional boundaries; (2) template and utility-function constraints that shift the agent from free-form code generation to configuration generation; and (3) small-sample validation and quality feedback before scale execution, forming a “generate– execute–check–fix” closed loop. Our contributions are: 1. A six-type collector taxonomy (search, list, detail, API, interactive, file) with explicit functional boundaries and composition rules. 2. A constrained agent framework organizing requirement understanding, instantiation, validation, quality checking, and feedback correction into a verifiable closed loop, generating JSON configurations under template, slot, and Schema constraints—not free-form code. 3. Multi-level experimental validation covering requirement-type classification, collector instantiation, quality–cost comparison against runtime LLM baselines on 80 verified tasks, feedback correction, and Airflow compatibility testing. The research is organized around three questions: RQ1: How to model collection requirements as typed tasks; RQ2: How to instantiate collectors under constraints for validatable, reusable, schedulable configurations; RQ3: How to intercept low-quality configurations before scale exec

FAQs

为什么直接用 LLM/Agent 生成网页爬虫在实践中不可靠？

直接生成容易出现依赖错误、选择器失效、输出模式不匹配以及网页结构差异带来的不一致，从而导致整体流程脆弱且难以稳定运行。

框架如何实现“可验证”的数据采集执行路径？

通过将自由代码生成替换为带约束的类型化 JSON collector 配置，并结合模板/工具函数约束、静态 Airflow DAG 执行、规则化质量检查以及结构化反馈闭环。

实验结果表明该框架在什么方面更有优势？

在 138 个任务上，六类 taxonomy 支持基于描述的需求类型化；在 80 个源已独立验证的任务上，框架可在执行阶段实现零 LLM token 使用，并获得最低平均 wall-clock 时间，同时保持可复用与可审计的确定性执行路径。

Jul 16, 2026

Upload by Aurora