[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-119591-en":3,"doc-seo-119591-105":30,"detail-sidebar-cat-0-en-105":92},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":20,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},119591,1374391974585,"Genevieve","https://ap-avatar.wpscdn.com/davatar_276721f389ce27ea32af1340a28f341c",8,"Research & Report","Between Randomness and Arbitrariness: Some Lessons for Reliable Machine Learning at Scale","Reliable machine learning requires rigorous measurement, yet reliability of measurement is fundamentally difficult. This dissertation develops criteria for meaningful metrics and for methodologies that enable dependable, efficient evaluation at scale, addressing reproducibility, scalability, uncertainty quantification, and epistemological concerns. It studies arbitrariness in hyperparameter optimization and social prediction, connects non-determinism-driven arbitrariness to legal ideas of arbitrariness and due process, and controls randomness in uncertainty estimation to balance scalability with reliability. It also presents evaluation methods for generative AI, linking memorization measurement and open-licensed data diffusion training to U.S. copyright law.","arXiv :2406 .09548v2 [ cs .LG] 12 Aug 2024  \nBetween Randomness and Arbitrariness: Some Lessons for Reliable Machine Learning at Scale  \nA. Feder Cooper  \nJune 3, 2024  \nA Dissertation  \nPresented to the Faculty of the Graduate School  \nof Cornell University  \nin Partial Fulfillment of the Requirements for the  \nDegree of Doctor of Philosophy  \n© 2024 A. Feder Cooper. All rights reserved.  \nBETWEEN RANDOMNESS AND ARBITRARINESS:  \nSOME LESSONS FOR RELIABLE MACHINE LEARNING AT SCALE  \nA. Feder Cooper, Ph.D.  \nCornell University 2024  \nTo develop rigorous knowledge about ML models—and the systems in which they are embedded—we need reliable measurements. But reliable measurement is fundamentally challenging, and touches on issues of reproducibility, scalability, uncertainty quantification, epistemology, and more. This dissertation addresses criteria needed to take reliability seriously: both criteria for designing meaningful metrics, and for methodologies that ensure that we can dependably and efficiently measure these metrics at scale and in practice. In doing so, this dissertation articulatesa research vision for a new field of scholarship at the intersection of machine learning, law, and policy. Within this frame, we cover topics that fit under three different themes.  \nFirst, we quantify and mitigate sources of arbitrariness in machine learning, with respect to hyperparameter optimization and social prediction contexts. We clarify important connections between machine-learning arbitrariness, rooted in non-determinism, with legal notions of arbitrariness that implicate legal rules and due process.  \nSecond, we tame randomness in uncertainty estimation and optimization algorithms, in order to achieve scalability without sacrificing reliability. We discuss how across computing, and particularly in machine learning, scalability and reliability are typically in trade-off. Analogous trade-offs in law and policy make this type of trade-off a useful abstraction for communicating about machinelearning capabilities and risks to policymakers and other non-expert stakeholders.  \nThird, we provide methods for evaluating generative-AI systems, with specific focuses on quantifying memorization in language models and training latent diffusion models on open-licensed data. These contributions have urgent and significant connections to U.S. copyright law. We provide an abridged discussion of landmark legal scholarship that details the complicated relationships between generative-AI supply chain and copyright.  \nBy making contributions in these three themes, this dissertation serves as an empirical proof by example that research on reliable measurement for machine learning is intimately and inescapably bound up with research in law and policy. These different disciplines pose similar research questions about reliable measurement in machine learning. They are, in fact, two complementary sides of the same research vision, which, broadly construed, aims to construct machine-learning systems that cohere with broader societal values.  \nBiographical Sketch  \nA. Feder Cooper was born and raised in New York, NY, and obtained his B.A. in Computer Science and Archaeology from Columbia University in 2014 . Prior to a research career, Cooper worked for several years as a software engineer. In 2018, he began his Ph.D. in Computer Science at Cornell University. His doctoral work was chaired by Professor Christopher De Sa, with additional advising by James Grimmelmann, Jon Kleinberg, [and Adrian Sampson. His Ph.D. work](and Adrian Sampson. His Ph.D. work), broadly construed, studies reliable measurement and evaluation of machine learning, covering both computer science aspects of this work as well as their associated ethical, legal, and policy dimensions.1  \nHis contributions span uncertainty estimation, privacy and security of generative-AI systems, distributed training, hyperparameter optimization, and model selection. His work has been recognized by spotlig","cbCaimNchshiAdld","https://ap.wps.com/l/cbCaimNchshiAdld","pdf",40771799,1,360,"English","en",105,"# Introduction\n## Reliable measurement for machine learning systems\n## Themes: arbitrariness, randomness, and generative-AI evaluation\n# Research contributions\n## Mitigating arbitrariness in optimization and prediction\n## Taming randomness for scalable uncertainty estimation\n## Evaluating generative-AI systems and legal connections\n# Dissertation background and author information","[{\"question\":\"What problem does the dissertation focus on?\",\"answer\":\"It focuses on making machine learning measurement truly reliable, including how to design meaningful metrics and how to measure them dependably and efficiently at scale in practice.\"},{\"question\":\"How does the dissertation address arbitrariness in machine learning?\",\"answer\":\"It quantifies and mitigates sources of arbitrariness, especially in hyperparameter optimization and social prediction contexts, and links non-determinism-based arbitrariness to legal notions tied to rules and due process.\"},{\"question\":\"What evaluation methods are proposed for generative-AI systems?\",\"answer\":\"It provides methods for evaluating generative-AI systems, including quantifying memorization in language models and training latent diffusion models on open-licensed data, with discussed connections to U.S. copyright law.\"}]","Between Randomness and Arbitrariness: Some Lessons for Reliable Machine Learning at Scale | PDF",1785725174,907,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":87,"head_meta":89,"extra_data":91,"updated_unix":28},"between-randomness-and-arbitrariness-some-lessons-for-reliable-machine-learning-at-scale","",{"@graph":36,"@context":86},[37,54,69],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/between-randomness-and-arbitrariness-some-lessons-for-reliable-machine-learning-at-scale/119591/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":63,"encodingFormat":61,"isAccessibleForFree":64,"interactionStatistic":65},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04","2026-08-03",true,{"@type":66,"interactionType":67,"userInteractionCount":20},"InteractionCounter",{"@type":68},"ViewAction",{"@type":70,"mainEntity":71},"FAQPage",[72,78,82],{"name":73,"@type":74,"acceptedAnswer":75},"What problem does the dissertation focus on?","Question",{"text":76,"@type":77},"It focuses on making machine learning measurement truly reliable, including how to design meaningful metrics and how to measure them dependably and efficiently at scale in practice.","Answer",{"name":79,"@type":74,"acceptedAnswer":80},"How does the dissertation address arbitrariness in machine learning?",{"text":81,"@type":77},"It quantifies and mitigates sources of arbitrariness, especially in hyperparameter optimization and social prediction contexts, and links non-determinism-based arbitrariness to legal notions tied to rules and due process.",{"name":83,"@type":74,"acceptedAnswer":84},"What evaluation methods are proposed for generative-AI systems?",{"text":85,"@type":77},"It provides methods for evaluating generative-AI systems, including quantifying memorization in language models and training latent diffusion models on open-licensed data, with discussed connections to U.S. copyright law.","https://schema.org",{"og:url":52,"og:type":88,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":90,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":93},[94,98,102,106,111,116,121,124,129,132,136],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":95,"show_sort_weight":96,"slug":97},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":99,"show_sort_weight":100,"slug":101},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":103,"show_sort_weight":104,"slug":105},"Exam",70,"exam",{"id":107,"doc_module":4,"doc_module_name":46,"category_name":108,"show_sort_weight":109,"slug":110},5,"Comic",60,"comic",{"id":112,"doc_module":4,"doc_module_name":46,"category_name":113,"show_sort_weight":114,"slug":115},6,"Technology",50,"technology",{"id":117,"doc_module":4,"doc_module_name":46,"category_name":118,"show_sort_weight":119,"slug":120},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":122,"slug":123},30,"research-report",{"id":125,"doc_module":4,"doc_module_name":46,"category_name":126,"show_sort_weight":127,"slug":128},9,"Religion & Spirituality",20,"religion-spirituality",{"id":127,"doc_module":4,"doc_module_name":46,"category_name":130,"show_sort_weight":127,"slug":131},"World Cup","world-cup",{"id":133,"doc_module":4,"doc_module_name":46,"category_name":134,"show_sort_weight":133,"slug":135},10,"Lifestyle","lifestyle",{"id":137,"doc_module":4,"doc_module_name":46,"category_name":138,"show_sort_weight":107,"slug":139},19,"General","general"]