[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-116844-en":3,"doc-seo-116844-105":29,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":11,"language":21,"language_code":22,"site_id":23,"html_lang":22,"table_of_contents":24,"faqs":25,"seo_title":26,"seo_description":14,"update_tm":27,"read_time":28},116844,1099513958607,"Jiven","https://ap-avatar.wpscdn.com/avatar/100002390cf8733938c?x-image-process=image/resize,m_fixed,w_180,h_180&k=1778829742770036399",8,"Research & Report","Dataset Management Platform for Machine Learning - Defensive Publications Series - 21 February 2023","Dataset quality directly affects how well machine learning models are trained and evaluated, making dataset management a critical capability. The disclosure presents a platform that combines dataset management with dataset transformation, reducing manual effort involved in dataset versioning and preparation. A storage engine serves as a source of truth, providing versioning and access control. A transformation mechanism generates dataset snapshots for different training, evaluation, and orchestration workflows, including lineage tracking and automation support.","Technical Disclosure Commons  \nDefensive Publications Series  \nFebruary 2023  \nDataset Management Platform for Machine Learning n/a  \nFollow this and additional works at: [https://www.tdcommons.org/dpubs_series](https://www.tdcommons.org/dpubs_series)  \nRecommended Citation  \nn/a, \"Dataset Management Platform for Machine Learning\", Technical Disclosure Commons,(February 21, 2023)  \n[https://www.tdcommons.org/dpubs_series/5690](https://www.tdcommons.org/dpubs_series/5690)  \nThis work is licensed under a Creative Commons Attribution 4.0 License.  \nThis Article is brought to you for free and open access by Technical Disclosure Commons. It has been accepted for inclusion in Defensive Publications Series by an authorized administrator of Technical Disclosure Commons.  \nDataset Management Platform for Machine Learning  \nABSTRACT  \nThe quality of the data in a dataset can have a substantial impact on the performance of a machine learning model that is trained and/or evaluated using the dataset. Effective dataset management, including tasks such as data cleanup, versioning, access control, dataset  \ntransformation, automation, integrity and security, etc., can help improve the efficiency and  \nspeed of the machine learning process. Currently, engineers spend a substantial amount of  \nmanual effort and time to manage dataset versions or to prepare datasets for machine learning  \ntasks. This disclosure describes a platform to manage and use datasets effectively. The  \ntechniques integrate dataset management and dataset transformation mechanisms. A storage  \nengine is described that acts as a source of truth for all data and handles versioning, access  \ncontrol etc. The dataset transformation mechanism is a key part to generate a dataset (snapshot)  \nto serve different purposes. The described techniques can support different workflows,  \npipelines, or data orchestration needs, e.g., for training and/or evaluation of machine learning  \nmodels.  \nKEYWORDS  \n● Dataset management  \n● Dataset transformation  \n● Training data  \n● Data versioning  \n● Data pipeline  \n● Data orchestration  \n● Machine learning  \nPublished by Technical Disclosure Commons, 2023 2  \nBACKGROUND  \nIn machine learning, a dataset is a collection of data that is used to train and evaluate a model. The quality of the data in a dataset can have a substantial impact on the performance of a machine learning model. Effective dataset management, including tasks such as data cleanup, versioning, access control, dataset transformation, automation, integrity and security, etc., is critical for machine learning researchers and engineers. Effective dataset management can also help improve the efficiency and speed of the machine learning process. However, there are no good tools or processes to manage a dataset, e.g., dataset versions, access control for the dataset, etc., or to transform a dataset. Engineers spend a substantial amount of manual effort and time to manage dataset versions or to prepare datasets for machine learning tasks.  \nWhile there are a few related products, they all fall short. DVC [1] is a data version control system, but it focuses on model development and deployment, rather than dataset  \nmanagement or transformation. Deeplake [2, 3] provides the ability to manage datasets, but  \ndoes not have support for data transformation at a large scale. TFDS [4] does not have  \nversioning and access control. git [5] is a well-known version control solution that is suitable for code or text, but not for large objects. Thus, git is not suitable for machine learning datasets,  \nwhich are usually very large and often include non-text data.  \nDESCRIPTION  \nThis disclosure describes a platform to manage and use datasets effectively. The techniques integrate dataset management (version, access control, etc.) and dataset transformation mechanisms. A storage engine is described that acts as a source of truth for all data and handles versioning, access control etc. The dataset tr","cbCaijEFpxo9wxhK","https://ap.wps.com/l/cbCaijEFpxo9wxhK","pdf",251811,1,"English","en",105,"# Abstract\n# Background\n# Description\n## Data repository and transformation pipelines\n## Architecture of the platform\n## Pipeline design and maintainability","[{\"question\":\"Why does dataset quality matter for machine learning performance?\",\"answer\":\"Dataset quality can substantially impact the performance of a model trained and/or evaluated using that dataset.\"},{\"question\":\"What core capabilities does the platform provide?\",\"answer\":\"It integrates dataset management (versioning, access control) with dataset transformation to generate dataset snapshots for different purposes.\"},{\"question\":\"How do the transformation pipelines work in the proposed approach?\",\"answer\":\"New data is ingested into a data repository via one pipeline, while other pipelines generate snapshots for specific goals; snapshots can be committed for future use and triggered manually or automatically.\"}]","Dataset Management Platform for Machine Learning - Defensive Publications Series - 21 February 2023 | PDF",1785672040,20,{"code":4,"msg":30,"data":31},"ok",{"site_id":23,"language":22,"slug":32,"title":13,"keywords":33,"description":14,"schema_data":34,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":27},"dataset-management-platform-for-machine-learning-defensive-publications-series-21-february-2023","",{"@graph":35,"@context":85},[36,53,68],{"@type":37,"itemListElement":38},"BreadcrumbList",[39,43,47,50],{"item":40,"name":41,"@type":42,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":44,"name":45,"@type":42,"position":46},"https://docshare.wps.com/document/","Document",2,{"item":48,"name":12,"@type":42,"position":49},"https://docshare.wps.com/document/research-report/",3,{"item":51,"name":13,"@type":42,"position":52},"https://docshare.wps.com/document/dataset-management-platform-for-machine-learning-defensive-publications-series-21-february-2023/116844/",4,{"url":51,"name":13,"@type":54,"author":55,"headline":13,"publisher":57,"fileFormat":60,"inLanguage":22,"description":14,"dateModified":61,"datePublished":62,"encodingFormat":60,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":56},"Person",{"url":40,"name":58,"@type":59},"DocShare","Organization","application/pdf","2026-08-05","2026-08-02",true,{"@type":65,"interactionType":66,"userInteractionCount":20},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"Why does dataset quality matter for machine learning performance?","Question",{"text":75,"@type":76},"Dataset quality can substantially impact the performance of a model trained and/or evaluated using that dataset.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What core capabilities does the platform provide?",{"text":80,"@type":76},"It integrates dataset management (versioning, access control) with dataset transformation to generate dataset snapshots for different purposes.",{"name":82,"@type":73,"acceptedAnswer":83},"How do the transformation pipelines work in the proposed approach?",{"text":84,"@type":76},"New data is ingested into a data repository via one pipeline, while other pipelines generate snapshots for specific goals; snapshots can be committed for future use and triggered manually or automatically.","https://schema.org",{"og:url":51,"og:type":87,"og:title":13,"og:site_name":58,"og:description":14},"article",{"robots":89,"canonical":51},"index,follow",{"doc_id":7,"site_id":23},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,127,130,134],{"id":20,"doc_module":4,"doc_module_name":45,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":46,"doc_module":4,"doc_module_name":45,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":52,"doc_module":4,"doc_module_name":45,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":45,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":45,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":45,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":45,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":45,"category_name":125,"show_sort_weight":28,"slug":126},9,"Religion & Spirituality","religion-spirituality",{"id":28,"doc_module":4,"doc_module_name":45,"category_name":128,"show_sort_weight":28,"slug":129},"World Cup","world-cup",{"id":131,"doc_module":4,"doc_module_name":45,"category_name":132,"show_sort_weight":131,"slug":133},10,"Lifestyle","lifestyle",{"id":135,"doc_module":4,"doc_module_name":45,"category_name":136,"show_sort_weight":106,"slug":137},19,"General","general"]