[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-121894-en":3,"doc-seo-121894-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},121894,4398048949847,"Eliana","https://ap-avatar.wpscdn.com/avatar/400002536579ef2da7f?_k=1778318612642679267",8,"Research & Report","Building reliable machine learning systems for neuroscience","Neuroscience generates rapidly growing datasets that enable fundamental studies of brain function, behavior, and the origins of disorders, but the field needs reproducible, reliable, and accessible machine-learning systems to turn data scale into trustworthy insights. This dissertation leverages existing data and domain expertise to build more reliable ML methods for animal behavior analysis. It improves pose estimation with weak supervision using spatial-temporal structure, and evaluates deep ensembling, showing limitations for reliability under distribution shift and in long-tail settings, then outlines challenges and opportunities for next-generation systems.","Building reliable machine learning systems for neuroscience  \nE. Kelly Buchanan  \nSubmitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy under the Executive Committee  \nof the Graduate School of Arts and Sciences  \nCOLUMBIA UNIVERSITY  \n© 2024  \nEstefany Kelly Buchanan All Rights Reserved  \nAbstract  \nBuilding reliable machine learning systems for neuroscience  \nE. Kelly Buchanan  \nNeuroscience as a field is collecting more data than at any other time in history. The scale of this data allows us to ask fundamental questions about the mechanisms of brain function, the basis of behavior, and the development of disorders. Our ambitious goals as well as the abundance of data being recorded call for reproducible, reliable, and accessible systems to push the field forward. While we have made great strides in building reproducible and accessible machine learning (ML) systems for neuroscience, reliability remains a major issue.  \nIn this dissertation, we show that we can leverage existing data and domain expert knowledge to build more reliable ML systems to study animal behavior. First, we consider animal pose estimation, a crucial component in many scientific investigations. Typical transfer learning ML methods for behavioral tracking treat each video frame and object to be tracked independently. We improve on this by leveraging the rich spatial and temporal structures pervasive in behavioral videos. Our resulting weakly supervised models achieve significantly more robust tracking. Our tools allow us to achieve improved results when we have imperfect, limited data while requiring users to label fewer training frames and speeding up training. We can more accurately process raw video data and learn interpretable units of behavior. In turn, these improvements enhance performance on downstream applications.  \nNext, we consider a ubiquitous approach to (attempt to) improve the reliability of ML methods, namely combining the predictions of multiple models, also known as deep ensembling. Ensembles  \nof classical ML predictors, such as random forests, improve metrics such as accuracy by  \nwell-understood mechanisms such as improving diversity. However, in the case of deep ensembles, there is an open methodological question as to whether, given the choice between a deep ensemble and a single neural network with similar accuracy, one model is truly preferable over the other. Via careful experiments across a range of benchmark datasets and deep learning models, we demonstrate limitations to the purported benefits of deep ensembles. Our results challenge common assumptions regarding the effectiveness of deep ensembles and the “diversity” principles underpinning their success, especially with regards to important metrics for reliability, such as out-of-distribution (OOD) performance and effective robustness. We conduct additional studies of the effects of using deep ensembles when certain groups in the dataset are underrepresented (so-called “long tail” data), a setting whose importance in neuroscience applications is revealed by our aforementioned work.  \nAltogether, our results demonstrate the essential importance of both holistic systems work and fundamental methodological work to understand the best ways to apply the benefits of modern machine learning to the unique challenges of neuroscience data analysis pipelines. To conclude the dissertation, we outline challenges and opportunities in building next-generation ML systems.  \nTable of Contents  \nAcknowledgments ........................................ xx  \nDedication ............................................ xxiii  \nChapter 1: Introduction .................................... 1  \n1.1 Motivation ....................................... 2  \n1.2 Key Contributions ................................... 5  \n1.2.1 Building reliable tools to study animal behavior ............... 6  \n1.2.2 Limitations of reliability methods under distribution shift .......... 7  \n","cbCaihSHWfSg8n9U","https://ap.wps.com/l/cbCaihSHWfSg8n9U","pdf",20961905,1,257,"English","en",105,"# Table of Contents\n## Chapter 1: Introduction\n## 1.1 Motivation\n## 1.2 Key Contributions\n## 1.2.1 Building reliable tools to study animal behavior\n## 1.2.2 Limitations of reliability methods under distribution shift\n## 1.3 List of Publications\n## Chapter 2: Animal Pose estimation when labeled data is scarce\n## 2.1 Introduction\n## 2.2 Deep Graph Pose: a class of semi-supervised pose estimation algorithms\n## 2.3 Structured variational inference\n## 2.3.1 Conceptual comparison against fully-supervised approaches\n## 2.4 Benefits of semi-supervised models over supervised counterparts\n## 2.5 Benefits of having more robust models in downstream analysis\n## 2.6 Conclusion and Future directions\n## Chapter 3: Animal Pose segmentation when labeled data is scarce\n## 3.1 Introduction\n## 3.2 Methods\n## 3.3 Results\n## 3.4 Discussion and Future Directions\n## Chapter 4: Chasing the Tails: The Effects of Ensembling in Long Tail Data\n## 4.1 Introduction\n## 4.2 Setup\n## 4.3 Experiments\n## 4.4 Discussion\n## 4.5 Conclusion\n## Chapter 5: Deep Ensembles work but are they necessary?\n## 5.1 Introduction\n## 5.2 Related work\n## 5.3 Setup\n## 5.4 Hypothesis: ensemble diversity is responsible for improved UQ","[{\"question\":\"What reliability problem motivates this dissertation in neuroscience machine learning?\",\"answer\":\"Reliability remains a major issue even as reproducible and accessible ML systems for neuroscience improve. The growing scale of data requires systems that produce dependable results for downstream scientific analysis.\"},{\"question\":\"How does the work improve animal pose estimation when labeled data is scarce?\",\"answer\":\"It leverages spatial and temporal structures present in behavioral videos rather than treating each frame and tracked object independently. The resulting weakly supervised models are more robust, need fewer labeled frames, and improve downstream application performance.\"},{\"question\":\"What do the dissertation results show about deep ensembling for reliability?\",\"answer\":\"Across benchmark datasets and deep learning models, the dissertation demonstrates limitations of the commonly claimed benefits of deep ensembles. The findings challenge assumptions, particularly for reliability-relevant metrics such as out-of-distribution performance and robustness, and also examine long-tail underrepresentation effects.\"}]","Building reliable machine learning systems for neuroscience | PDF",1785807608,648,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"building-reliable-machine-learning-systems-for-neuroscience","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/building-reliable-machine-learning-systems-for-neuroscience/121894/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-04",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What reliability problem motivates this dissertation in neuroscience machine learning?","Question",{"text":75,"@type":76},"Reliability remains a major issue even as reproducible and accessible ML systems for neuroscience improve. The growing scale of data requires systems that produce dependable results for downstream scientific analysis.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"How does the work improve animal pose estimation when labeled data is scarce?",{"text":80,"@type":76},"It leverages spatial and temporal structures present in behavioral videos rather than treating each frame and tracked object independently. The resulting weakly supervised models are more robust, need fewer labeled frames, and improve downstream application performance.",{"name":82,"@type":73,"acceptedAnswer":83},"What do the dissertation results show about deep ensembling for reliability?",{"text":84,"@type":76},"Across benchmark datasets and deep learning models, the dissertation demonstrates limitations of the commonly claimed benefits of deep ensembles. The findings challenge assumptions, particularly for reliability-relevant metrics such as out-of-distribution performance and robustness, and also examine long-tail underrepresentation effects.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]