SchemaRAG: Dynamic Large Schema Reduction for LLM-driven Structured Information Extraction

Jul 16, 2026

Extracting structured data from unstructured text with large language models (LLMs) becomes difficult when target schemas are large, complex, and hierarchical. Including the full schema in prompts can raise cost and latency, worsen lost-in-the-middle effects, and hit context length limits. SchemaRAG introduces a retrieval-augmented generation framework that dynamically prunes the output schema space using schema metadata and optional few-shot examples. Evaluations on healthcare and e-commerce datasets show up to an 8.8% micro-F1 gain, 47% lower latency, and 48% reduced token costs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

Introduction

Related Work

SchemaRAG: Dynamic Large Schema Reduction for LLM-driven Structured Information Extraction Sin Yu Bonnie Ho* Arlie Coles* Erik Larsson Eric Marshall Nathan Bodenstab Paul Vozila Microsoft arXiv :2607 .00008v 1 [ cs .IR] 4 May 2026 Abstract Extracting structured data from unstructured text using large language models (LLMs) becomes challenging when the target schemas are large and complex. In such cases, including the full schema in the prompt increases cost and latency, risks lost-in-the-middle performance degradation, and can exceed context length limits. We propose SchemaRAG, a retrievalaugmented generation (RAG) framework that dynamically prunes the output schema space for schema-conditioned information extraction tasks by leveraging schema metadata and fewshot examples (when available) . We evaluate SchemaRAG on real-world healthcare ande-commerce datasets. Our results show that SchemaRAG can achieve up to an 8.8% increase in micro-F 1 , a 47% reduction in latency, and a 48% reduction in token costs, demonstrating its practicality for large-schema extraction. 1 Introduction Structured information extraction (IE) pairs values from unstructured text with schema-defined keys. In healthcare, this includes medical attribute extraction from unstructured clinical notes (Jiang et al., 2011 ; Agrawal et al., 2022) and accelerated human annotation of medical narrative data (Goel et al., 2023) . Similarly, e-commerce applications focus on product attribute value extraction from unstructured text descriptions with and without supplementary multimodal data (Brinkmann et al., 2024 ; Zhu et al., 2020) . Further applications of structured IE exist for problems that can be cast as form-filling, where models need to populate complex, hierarchical schemas from unstructured text. These forms could have numerous rows, and each row could have complex constraints on potential values (e.g., required *Equal contribution. Corresponding authors: [siho@microsoft.com](siho@microsoft.com), [arliecoles@microsoft.com](arliecoles@microsoft.com) types including strings, numbers, single-select radio buttons, multi-select checkboxes, or other custom types) . In this work, we consider two such use cases: 1) populating electronic health records (EHRs) from medical narratives and 2) filling a product metadata form from e-commerce product description text. Instruction-tuned large language models (LLMs) are well-suited for these tasks due to their broad semantic knowledge (Singhal et al., 2023) and ability to follow structured prompt instructions (Liu et al., 2024b) . However, naively injecting large schemas into the prompt can exceed context limits or introduce many irrelevant rows. This leads to lost-in-the-middle effects (Liu et al., 2024a) and performance degradation as the model struggles to isolate pertinent rows. The resulting latency and inference costs can preclude real-time applications. To mitigate these issues, we propose SchemaRAG, a retrieval-augmented generation approach that retrieves and prunes the output schema space. By leveraging schema metadata and, when available, limited labeled data, SchemaRAG dynamically selects a relevant schema subset for just-intime in-context prompted LLM use (Figure 1) . A well-selected reduced schema shrinks prompt size, decreases the chance of the LLM filling in irrelevant rows, and improves overall extraction performance. The framework is schema-agnostic and supports arbitrary levels of hierarchy. Because the embeddings driving the retrieval can be computed offline, the system is efficient and easily integrated into existing pipelines. Crucially, SchemaRAG operates as a training-free solution, requiring no pretraining or fine-tuning. We explicitly focus on schemaconditioned extraction using hosted LLMs under real-world deployment scenarios including large, complex schemas and strict resource budgets. By utilizing prompting rather than retraining, our approach allows for rapid iteratio

FAQs

Why does injecting a large schema into an LLM prompt hurt structured information extraction?

Large schemas increase prompt length, cost, and latency, can trigger context length limits, and can cause lost-in-the-middle performance degradation by distracting the model from relevant rows or fields.

What is SchemaRAG and how does it address large-schema extraction?

SchemaRAG is a retrieval-augmented generation framework that dynamically selects and prunes a reduced subset of the output schema, guided by schema metadata and optional few-shot examples, before running a just-in-time prompted LLM extraction.

What results does SchemaRAG achieve on real-world datasets?

On healthcare and e-commerce datasets, SchemaRAG reports up to an 8.8% increase in micro-F1, along with up to 47% latency reduction and 48% token-cost reduction, though outcomes vary by dataset.

Jul 16, 2026

Upload by Emma Wilson