[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-85246-en":3,"doc-seo-85246-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":21,"is_downloadable":21,"audit_status":21,"page_count":22,"language":23,"language_code":24,"site_id":25,"html_lang":24,"table_of_contents":26,"faqs":27,"seo_title":13,"seo_description":14,"update_tm":28,"read_time":29},85246,1374391974564,"Clementine","https://ap-avatar.wpscdn.com/avatar/14000253aa45c000a9e?x-image-process=image/resize,m_fixed,w_180,h_180&k=1779874745381141002",8,"Research & Report","Lightning Fast Matching Dependency Discovery with Desbordante","Matching dependencies generalize functional dependencies by letting users define custom similarity functions for attribute matching, supporting key data quality tasks such as entity resolution, deduplication, data integration, and schema matching. Discovery is computationally intensive, limiting real-world use. The paper presents optimization techniques for HyMD, including a new sampling method, faster generalization lookup, and an improved dependency representation. Implemented in Desbordante, the optimizations achieve over 40× average speedup and up to 170× in some cases, with bidirectional Python integration for custom similarity functions.","Lightning Fast Matching Dependency Discovery  \nwith Desbordante  \nAlexey Shlyonskikh, Michael Sinelnikov, Daniil Nikolaev, Yurii Litvinov, George Chernishev  \nSaint-Petersburg University  \nSaint-Petersburg, Russia  \n{shlyonskikh.alexey, michael.a.sinelnikov, [daniil.vl.nikolaev](daniil.vl.nikolaev}@gmail.com)[}](daniil.vl.nikolaev}@gmail.com)[@gmail.com](daniil.vl.nikolaev}@gmail.com), {y.litvinov, [g.chernyshev](g.chernyshev}@spbu.ru)[}](g.chernyshev}@spbu.ru)[@spbu.ru](g.chernyshev}@spbu.ru)  \narXiv :2607 . 10771v1 [ cs .DB] 12 Jul 2026  \nAbstract—Matching dependency is a generalization of the functional dependency concept, which allows users to apply custom similarity functions for matching individual attributes. Matching dependencies have a wide range of applications for solving various data quality problems, such as entity resolution, data deduplication, data integration, schema matching, and many more. However, their discovery is a very computationally intensive problem, which limits their practical application.  \nIn this paper, we describe a number of optimization techniques for HyMD — currently the state-of-the-art algorithm for the discovery of matching dependencies. These optimizations belong to both technical and scientific domains. The most important of them are: 1) a new sampling technique, 2) a faster generalization lookup technique, and 3) an improved representation of a dependency. The first one aims to raise the efficiency of inference from record pairs, while the last two are designed to speed up lattice-related operations.  \nTo evaluate our optimizations, we implemented our version of HyMD in Desbordante, an open-source high-performance data profiler. Experiments demonstrated that they allow for a speedup of more than 40x over the state-of-the-art implementation on average, reaching a speedup greater than 170x in some cases.  \nFinally, the improved version of HyMD is ready to use by anyone. It comes with bidirectional Python integration, which allows calling the C++ algorithm implementation from Python programs while allowing users to supply their custom matching functions.  \nI. INTRODUCTION  \nData profiling [1] involves extracting various types of information from data. Such information can range from simple statistics (e.g. the number of columns in a table) to complex facts, indicating the presence of various patterns in data. We refer to the first type of extraction [2] as naive profiling and to the second one as science-intensive profiling.  \nScience-intensive data profiling uses computationally costly algorithms to extract sophisticated patterns from the data, such as functional dependencies [3], inclusion dependencies [4]), association rules [5], algebraic constraints [6], inferred semantic data types [7], and others. These patterns represent knowledge about data and can be used to perform various tasks in many domains: data quality, data integration, database management, query optimization, database reverse engineering, and many more.  \nOne of the well-known patterns is matching dependency (MD), which is defined on tables. Informally, it states that for all record pairs where some attribute values are “similar”, some other attributes are also “similar”.  \nTABLE I  \nAIRLINE FLIGHTS  \n id  Source  From  To  Distance (km)   \n\n| 1 | ac1 | Saint-Petersburg | Helsinki | 315 |\n| --- | --- | --- | --- | --- |\n| 2 | ac2 | St-Petersburg | Helsinki | 301 |\n| 3 | ac2 | Moscow | St-Petersburg | 650 |\n| 4 | ac2 | Moscow | St-Petersburg | 638 |\n| 5 | ac1 | Moscow | Saint-Petersburg | 670 |\n| 6 | ac1 | Moscow | Yekaterinburg | 1417 |\n\nConsider an example presented in Table I. It lists airline routes, which ended up in a single table after data from two airline carriers (ac1 and ac2) was merged. This data contains the following MD:  \nFrom0.75To0.75 → Distance0.9 .  \nThe columns involved have different data types — string and integer. Therefore different metrics should be used to specify similarity. For From and To we ","cbCaikzLjKoRf4Ao","https://ap.wps.com/l/cbCaikzLjKoRf4Ao","pdf",421997,3,1,12,"English","en",105,"# Abstract\n# Introduction\n## Data profiling and science-intensive patterns\n## Matching dependencies and motivating example","[{\"question\":\"What problem do matching dependencies address in data quality?\",\"answer\":\"They capture rules that when certain attributes in record pairs are similar, other attributes should also be similar. This supports tasks like record matching, entity resolution, deduplication, and data integration.\"},{\"question\":\"Why is matching dependency discovery computationally challenging?\",\"answer\":\"Algorithms for discovery have exponential time complexity. This makes optimizations necessary for the approach to be practically usable.\"},{\"question\":\"Which optimizations are proposed for HyMD and what impact do they have?\",\"answer\":\"The paper introduces a new sampling technique, a faster generalization lookup technique, and an improved representation of a dependency. Experiments in Desbordante show more than 40× average speedup and over 170× in some cases.\"}]",1784202035,30,{"code":4,"msg":31,"data":32},"ok",{"site_id":25,"language":24,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"lightning-fast-matching-dependency-discovery-with-desbordante","",{"@graph":36,"@context":85},[37,53,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,50],{"item":41,"name":42,"@type":43,"position":21},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":20},"https://docshare.wps.com/document/research-report/",{"item":51,"name":13,"@type":43,"position":52},"https://docshare.wps.com/document/lightning-fast-matching-dependency-discovery-with-desbordante/85246/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":24,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":41,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-07-24","2026-07-16",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What problem do matching dependencies address in data quality?","Question",{"text":75,"@type":76},"They capture rules that when certain attributes in record pairs are similar, other attributes should also be similar. This supports tasks like record matching, entity resolution, deduplication, and data integration.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"Why is matching dependency discovery computationally challenging?",{"text":80,"@type":76},"Algorithms for discovery have exponential time complexity. This makes optimizations necessary for the approach to be practically usable.",{"name":82,"@type":73,"acceptedAnswer":83},"Which optimizations are proposed for HyMD and what impact do they have?",{"text":84,"@type":76},"The paper introduces a new sampling technique, a faster generalization lookup technique, and an improved representation of a dependency. Experiments in Desbordante show more than 40× average speedup and over 170× in some cases.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":25},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,122,127,130,134],{"id":21,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":29,"slug":121},"research-report",{"id":123,"doc_module":4,"doc_module_name":46,"category_name":124,"show_sort_weight":125,"slug":126},9,"Religion & Spirituality",20,"religion-spirituality",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":128,"show_sort_weight":125,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":46,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":46,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]