[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"detail-sidebar-cat-1-en-105":3,"doc-seo-562611-105":53,"doc-detail-562611-en":118},{"code":4,"msg":5,"data":6},0,"success",[7,14,19,24,29,34,39,44,49],{"id":8,"doc_module":9,"doc_module_name":10,"category_name":11,"show_sort_weight":12,"slug":13},11,1,"Template","Presentations",90,"presentations",{"id":15,"doc_module":9,"doc_module_name":10,"category_name":16,"show_sort_weight":17,"slug":18},12,"Resumes",80,"resumes",{"id":20,"doc_module":9,"doc_module_name":10,"category_name":21,"show_sort_weight":22,"slug":23},14,"Invoices",70,"invoices",{"id":25,"doc_module":9,"doc_module_name":10,"category_name":26,"show_sort_weight":27,"slug":28},15,"Posters",60,"posters",{"id":30,"doc_module":9,"doc_module_name":10,"category_name":31,"show_sort_weight":32,"slug":33},16,"Social Media",50,"social-media",{"id":35,"doc_module":9,"doc_module_name":10,"category_name":36,"show_sort_weight":37,"slug":38},17,"Forms",40,"forms",{"id":40,"doc_module":9,"doc_module_name":10,"category_name":41,"show_sort_weight":42,"slug":43},18,"Letters",30,"letters",{"id":45,"doc_module":9,"doc_module_name":10,"category_name":46,"show_sort_weight":47,"slug":48},21,"Paper Templates",5,"papers-templates",{"id":50,"doc_module":9,"doc_module_name":10,"category_name":51,"show_sort_weight":4,"slug":52},158,"General","general-158",{"code":4,"msg":54,"data":55},"ok",{"site_id":56,"language":57,"slug":58,"title":59,"keywords":60,"description":61,"schema_data":62,"social_meta":111,"head_meta":113,"extra_data":115,"updated_unix":117},105,"en","recap-auditing-retrieval-coverage-in-production-rag-pipelines","Re:CAP – Auditing Retrieval Coverage in Production RAG Pipelines","","Retrieval-augmented generation (RAG) is difficult to monitor in production because non-stationary, continuously re-indexed corpora lack exhaustive relevance labels. The method Re:CAP (REtrieval Coverage Audit by iterative Probing) audits a deployed RAG pipeline by probing for evidence of missing documents rather than enumerating all relevant ones. It iteratively builds a topic registry, generates gap questions, retrieves candidates, and uses an LLM judge to keep only novel information. Across four benchmarks, Re:CAP recovers 9–29% of gold labels beyond flat BM25 top-500, reaching 48% on TREC-COVID. It improves MuSiQue recall by +12.9pp under under half the document budget and yields reproducible end-to-end recall within ±1% across independent runs.",{"@graph":63,"@context":110},[64,80,101],{"@type":65,"itemListElement":66},"BreadcrumbList",[67,71,74,77],{"item":68,"name":69,"@type":70,"position":9},"https://docshare.wps.com","Home","ListItem",{"item":72,"name":10,"@type":70,"position":73},"https://docshare.wps.com/template/",2,{"item":75,"name":51,"@type":70,"position":76},"https://docshare.wps.com/template/general/",3,{"item":78,"name":59,"@type":70,"position":79},"https://docshare.wps.com/template/recap-auditing-retrieval-coverage-in-production-rag-pipelines/562611/",4,{"url":78,"name":59,"@type":81,"image":82,"author":87,"headline":59,"publisher":90,"fileFormat":93,"inLanguage":57,"description":61,"dateModified":94,"datePublished":95,"encodingFormat":93,"isAccessibleForFree":96,"interactionStatistic":97},"DigitalDocument",{"url":83,"@type":84,"width":85,"height":86},"https://docshare.wps.com/thumbnails/recap-auditing-retrieval-coverage-in-production-rag-pipelines/562611.png","ImageObject",442,249,{"name":88,"@type":89},"Blue Pony","Person",{"url":68,"name":91,"@type":92},"DocShare","Organization","application/pdf","2026-11-03","2026-10-12",true,{"@type":98,"interactionType":99,"userInteractionCount":47},"InteractionCounter",{"@type":100},"ViewAction",{"@type":102,"mainEntity":103},"FAQPage",[104],{"name":105,"@type":106,"acceptedAnswer":107},"What improvements does Re:CAP achieve over BM25 and hybrid baselines?","Question",{"text":108,"@type":109},"On four public benchmarks, Re:CAP recovers 9–29% of gold labels unreachable by flat BM25 top-500, and reaches 48% on TREC-COVID. On MuSiQue, it improves recall by +12.9pp versus flat hybrid top-500 while using less than half the document budget, and on TREC-COVID it recovers additional gold docs judged to add new information by human annotators.","Answer","https://schema.org",{"og:url":78,"og:type":112,"og:title":59,"og:site_name":91,"og:description":61},"article",{"robots":114,"canonical":78},"index,follow",{"doc_id":116,"site_id":56},562611,1792546150,{"code":4,"msg":5,"data":119},{"doc_id":116,"user_id":120,"nickname":88,"user_avatar":121,"doc_module":9,"category_id":50,"category_name":51,"doc_title":59,"doc_description":61,"doc_content":122,"file_id":123,"file_url":124,"file_type":125,"file_size":126,"view_count":47,"is_deleted":4,"is_public":9,"is_downloadable":9,"audit_status":9,"page_count":127,"language":128,"language_code":57,"site_id":56,"html_lang":57,"table_of_contents":129,"faqs":130,"seo_title":131,"seo_description":61,"update_tm":132,"read_time":15},962090760266,"https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c","Re:CAP – Auditing Retrieval Coverage in Production RAG Pipelines  \nAviral Joshi, Hanoz Bhathena, Max Nelson, Saket Sharma  \nMachine Learning Center of Excellence, JPMorgan Chase & Co.  \n{aviral.joshi, hanoz.bhathena, max .nelson, [saket.sharma}@jpmchase.com](saket.sharma}@jpmchase.com)  \narXiv : 2609 .24 122v2 [cs .CL] 22 Sep 2026  \nAbstract  \nRetrieval-augmented generation (RAG) is hard to monitor in production: exhaustive relevance labels do not exist for non-stationary multimillion-passage corpora that re-index in realtime. As a result, retrieval quality is generally understudied and often deprioritised in favour of generation-oriented metrics. In this work, we propose auditing retrieval coverage by probing for evidence of missing documents rather than enumerating every relevant one. Our method Re:CAP (REtrieval Coverage Audit by iterative Probing) is a reference-free audit loop applied to a deployed RAG pipeline’s initial answer and retrieved context: it identifies the topics already covered, generates probing questions for plausibly missing topics, retrieves candidate documents, and applies an LLM-as-judge to retain only those that introduce previously-unretrieved information. On four public benchmarks, Re:CAP recovers 9– 29% of gold labels that flat BM25 top-500 cannot reach, rising to 48% on TREC-COVID. On MuSiQue Re:CAP beats flat hybrid top-500 by +12 .9pp on recall at less than half the document budget. An ensemble BM25, dense, and hybrid baseline (top-500 each) still leaves out 21.2% of gold docs on TREC-COVID that Re:CAP recovers; human annotators judge that 78.9% of those structurally distinct documents add new information to the baseline answer (Fleiss κ = 0 .79, n = 123), and 73.9% on live production traffic (n = 180) . End-to-end recall is reproducible to within ±1% across three independent runs, making Re:CAP a stable instrument for periodic retrieval audits.  \n1 Introduction  \nRetrieval-augmented generation (RAG) is a standard component of enterprise knowledge assistants over proprietary corpora such as news feeds, internal research, and regulatory filings (Lewis et al., 2020 ; Gao et al., 2023) . Popular RAG evaluation frameworks—RAGAS (Es et al., 2023) and  \nARES (Saad-Falcon et al., 2024)—score answerside signals (faithfulness, answer relevance) and the relevance of the retrieved context, but cannot detect what the retriever failed to surface. As a result, an answer that is fluent and faithful to its retrieved context can still be substantively incomplete, and these metrics do not register the omission. For instance, a research assistant asked about a recent central-bank policy announcement may ground its answer in the policy statement and the lead wire story while overlooking the updated economic projections, the press-conference transcript, recorded dissents, and post-meeting analyst commentary—all present in the same corpus. We refer to this gap between the documents the retriever surfaces and the documents the corpus contains as the retrieval coverage problem, and we observe it most acutely in non-stationary corpora with multiple sources of truth. Figure 1 illustrates the problem: retrieving more deeply extends the region a query already reaches, so gold that lies outside it stays unreachable at any affordable depth.  \nClassical recall-based evaluation addresses retrieval coverage in principle but is impractical for large production deployments. Modern production indices contain 108 or more passages (Karpukhinet al., 2020 ; Bajaj et al., 2018), and the underlying corpus is re-indexed continuously as documents are added, edited, or withdrawn. Exhaustive perquery relevance judgements are prohibitive at this scale, and TREC-style pooling (Voorhees, 2000 ; Buckley et al., 2007) requires multiple participating systems and fresh labels after every change to the index, embedder, or ranker. Proprietary deployments typically have no per-query relevance labels at all, leaving operators reliant on end-to-end answ","cbCaikl7Yus9YCMw","https://ap.wps.com/l/cbCaikl7Yus9YCMw","pdf",977728,34,"English","# Abstract\n# 1 Introduction\n## Problem: retrieval coverage blind spot\n## Need for reference-free auditing\n## Reframing as gap probing\n# Method Overview: Re:CAP","[{\"question\":\"What improvements does Re:CAP achieve over BM25 and hybrid baselines?\",\"answer\":\"On four public benchmarks, Re:CAP recovers 9–29% of gold labels unreachable by flat BM25 top-500, and reaches 48% on TREC-COVID. On MuSiQue, it improves recall by +12.9pp versus flat hybrid top-500 while using less than half the document budget, and on TREC-COVID it recovers additional gold docs judged to add new information by human annotators.\"}]","Re:CAP – Auditing Retrieval Coverage in Production RAG Pipelines | PDF",1791809088]