[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-117850-en":3,"doc-seo-117850-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},117850,962075006959,"Anda","https://ap-avatar.wpscdn.com/avatar/e0002397efbe92a78e?_k=1776741047341049297",6,"Technology","MDB - Interactively Querying Datasets and Models - Abstract","MDB is a debugging framework designed to systematically diagnose failures that arise across the machine learning pipeline. It enables interactive, reusable queries over a database containing datasets and model predictions by combining functional programming with relational algebra. MDB targets error discovery and model-behavior characterization at scale, including object detection, bias discovery, image classification, and data imputation across self-driving videos, large language models, and medical records. Experiments show up to 10x faster, 40% shorter queries, and a user study confirms developers can express complex model errors.","MDB: Interactively Querying Datasets and Models  \nAaditya Naik  \nUniversity of Pennsylvania [asnaik@seas.upenn.edu](asnaik@seas.upenn.edu)  \nAdam Stein  \nUniversity of Pennsylvania [steinad@seas.upenn.edu](steinad@seas.upenn.edu)  \narXiv :2308 .06686v1 [ cs .DB] 13 Aug 2023  \nYinjun Wu  \nUniversity of Pennsylvania [wuyinjun@seas.upenn.edu](wuyinjun@seas.upenn.edu)  \nMayur Naik  \nUniversity of Pennsylvania [mhnaik@seas.upenn.edu](mhnaik@seas.upenn.edu)  \nEric Wong  \nUniversity of Pennsylvania [exwong@seas.upenn.edu](exwong@seas.upenn.edu)  \nAbstract  \nAs models are trained and deployed, developers need to be able to systematically debug errors that emerge in the machine learning pipeline. We present MDB, a debugging framework for interactively querying datasets and models. MDB integrates functional programming with relational algebra to build expressive queries over a database of datasets and model predictions. Queries are reusable and easily modified, enabling debuggers to rapidly iterate and refine queries to discover and characterize errors and model behaviors. We evaluate MDB on object detection, bias discovery, image classification, and data imputation tasks across self-driving videos, large language models, and medical records. Our experiments show that MDB enables up to 10x faster and 40% shorter queries than other baselines. In a user study, we find developers can successfully construct complex queries that describe errors of machine learning models.  \n1 Introduction  \nAs machine learning models continue to improve, they also continue to fail in unexpected and harmful ways. In 2018, a fatality caused by a self-driving vehicle Wakabayashi [2018] made it clear that such failures must be taken seriously. Other examples include state-of-the-art vision models performing worse on people with darker skin Wilson et al. [2019], language models producing text containing sterotypes against Muslims Abid et al. [2021], and medical diagnosis models degrading in performance when used in new hospitals Zech et al. [2018] . As models are deployed in increasingly critical applications like self-driving and medical diagnostics, identifying and avoiding such behaviors is crucial to ensure performance, reliability, and trustworthiness.  \nBefore rectifying these “bugs” in the machine learning pipeline, one must first identify them. Ideally, such bugs would be identified before deployment, but there are several challenges in doing so. As an example, consider Figure 1a, which depicts three consecutive frames of a video from the Cityscapes self-driving dataset Cordts et al. [2016] . A state-of-the-art vision model OneFormer Jain et al. [2022a] fails to predict a pedestrian in the third frame. This is clearly an undesirable error, but how can one characterize this bug?  \nA typical strategy to represent a bug is to construct a group of examples that contain a similar pattern Kang et al. [2018] . Importantly, such patterns should generalize to new instances of the same bug. However, the example in Figure 1a represents only a single instance—additional examples are needed to create a complete and generalizable bug description. This leads to the first challenge: due to the scale and complexity of modern datasets, manually sifting through individual samples to find such instances is infeasible. Instead, we would like a simple description that can capture these instances automatically at scale, such as “the pedestrian suddenly disappears after previously  \nPreprint. Under review.  \n\n| i. Task\u003Cbr>Object Detection in Video Data |\n| --- |\n| ii. Bug\u003Cbr>An object occurring in three consecutive frames fails to be detected consistently. |\n| iii. Query\u003Cbr>q = (Query('temporal_consistency')\u003Cbr>.register( 'outp' )\u003Cbr>.join( 'outp', key=fid_plus1, fkey=fid)\u003Cbr>.join( 'outp', key=fid_plus2, fkey=fid)\u003Cbr>.cols(match_three_bboxes)\u003Cbr>.cols(zip_adj_preds) .flatten()\u003Cbr>.filter(lambda sid, frames, lb, bx:( lb[0] == lb[1]\u003Cbr>and lb[2] == \"No Match\"\u003Cbr>and in_center(bx[","cbCaielXBK3M7RQS","https://ap.wps.com/l/cbCaielXBK3M7RQS","pdf",3648726,1,28,"English","en",105,"# Introduction\n## Debugging ML pipeline failures\n## Challenges in characterizing bugs at scale\n## MDB framework overview\n## Evaluation across domains","[{\"question\":\"What problem does MDB address in machine learning development?\",\"answer\":\"MDB focuses on systematically debugging errors that emerge in the machine learning pipeline after training and deployment, including failures that can be harmful in critical applications.\"},{\"question\":\"How does MDB enable interactive and reusable debugging queries?\",\"answer\":\"MDB integrates functional programming with relational algebra to build expressive queries over a database of datasets and model predictions, so queries can be reused and easily modified.\"},{\"question\":\"What kinds of tasks and domains does MDB evaluate on?\",\"answer\":\"MDB is evaluated on object detection, bias discovery, image classification, and data imputation across self-driving videos, large language models, and medical records.\"}]","MDB - Interactively Querying Datasets and Models - Abstract | PDF",1785679988,71,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"mdb-interactively-querying-datasets-and-models-abstract","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/technology/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/mdb-interactively-querying-datasets-and-models-abstract/117850/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem does MDB address in machine learning development?","Question",{"text":75,"@type":76},"MDB focuses on systematically debugging errors that emerge in the machine learning pipeline after training and deployment, including failures that can be harmful in critical applications.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does MDB enable interactive and reusable debugging queries?",{"text":80,"@type":76},"MDB integrates functional programming with relational algebra to build expressive queries over a database of datasets and model predictions, so queries can be reused and easily modified.",{"name":82,"@type":73,"acceptedAnswer":83},"What kinds of tasks and domains does MDB evaluate on?",{"text":84,"@type":76},"MDB is evaluated on object detection, bias discovery, image classification, and data imputation across self-driving videos, large language models, and medical records.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,113,118,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":111,"slug":112},50,"technology",{"id":114,"doc_module":4,"doc_module_name":46,"category_name":115,"show_sort_weight":116,"slug":117},7,"Healthcare",40,"healthcare",{"id":119,"doc_module":4,"doc_module_name":46,"category_name":120,"show_sort_weight":121,"slug":122},8,"Research & Report",30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]