Re-examining Metrics for Success in Machine Learning - from Fairness and Interpretability to Protein Design

Aug 5, 2026

Upload by Aria

Quantitative metrics and the datasets used to evaluate them drive rapid advances in machine learning by setting priorities and enabling efficient discovery of models. Yet metrics must reflect real-world goals so that measurable gains transfer to practical tasks, an external-validity challenge that evolves as new issues surface. This dissertation develops tests for representation similarity metrics, new data for fair classification, and protein-language-model metrics capturing evolutionary-taxa bias, including a mitigation method.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages144

Reading Time6 h 3 min

AccessFree to download

Introduction

Evaluating Representation Similarity Metrics

Problem Setup: Metrics and Models

Warm-up: Intuitive Tests for Sensitivity and Speciﬁcity

Rigorously Evaluating Dissimilarity Metrics

Assessing Fair Machine Learning with New Datasets

Archaeology of UCI Adult: Origin, Impact, Limitations

New datasets for algorithmic fairness

Identifying Biases in Protein Language Models

Bias mitigation

UC Berkeley UC Berkeley Electronic Theses and Dissertations Title Re-examining Metrics for Success in Machine Learning, from Fairness and Interpretability to Protein Design Permalink [https://escholarship.org/uc/item/4637w8xs](https://escholarship.org/uc/item/4637w8xs) Author Ding, Frances Publication Date 2024 Peer reviewed|Thesis/dissertation [eScholarship.org](eScholarship.org) Powered by the California Digital Library University of California Re-examining Metrics for Success in Machine Learning, from Fairness and Interpretability to Protein Design By Frances Ding A dissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy in Computer Science in the Graduate Division of the University of California, Berkeley Committee in charge: Assistant Professor Jacob Steinhardt, Co-chair Associate Professor Moritz Hardt, Co-chair Professor Yun Song Professor Jeremy Reiter Summer 2024 Re-examining Metrics for Success in Machine Learning, from Fairness and Interpretability to Protein Design Copyright 2024 By Frances Ding 1 Abstract Re-examining Metrics for Success in Machine Learning, from Fairness and Interpretability to Protein Design By Frances Ding Doctor of Philosophy in Computer Science University of California, Berkeley Assistant Professor Jacob Steinhardt, Co-chair Associate Professor Moritz Hardt, Co-chair Quantitative metrics, along with datasets to assess them with, are key ingredients that have fueled rapid progress in machine learning (ML) in recent years. These metrics, datasets, and benchmarks deﬁne priorities and facilitate eﬃcient discovery of model designs that make progress on those priorities. Ideally, metrics track real world goals, such that improvement on them translates to improvement in related, real tasks. Creating metrics that achieve this external validity is an ever-present challenge in ML. Thus, the science of metrics is an iterative one, as identifying and resolving one issue allows other, more subtle ones, to become apparent. In this thesis, we describe a series of works that highlight limitations in metrics across diﬀerent subﬁelds of ML and design new metrics to ﬁll these gaps. We ﬁrst examine representation similarity metrics used in the interpretability subﬁeld to compare neural network representations. We show that current popular metrics often disagree on fundamental observations, making it unclear which one we should believe. We develop practical, statistically grounded tests to evaluate these metrics and ﬁnd diﬀerent weaknesses in each. We next examine metrics and benchmarks for fair classiﬁcation. We highlight idiosyncrasies in the popular UCI Adult dataset that limit its external validity, and we contribute a suite of new datasets derived from US Census surveys that extend the existing data ecosystem for research on fair machine learning. Finally, we examine the subﬁeld of protein modeling with ML. We develop metrics to quantify a novel type of bias present in popular protein language models–bias towards sequences from certain evolutionary taxa. We additionally introduce a method to mitigate this bias. Across these works in diverse subﬁelds, we demonstrate the challenges and opportunities present in developing metrics that advance technical capabilities in alignment with real world needs. i To my parents, Christine and George. ii Contents Contents ii List of Figures iv List of Tables vi 1 Introduction 1 2 Evaluating Representation Similarity Metrics 4 2.1 Introduction .................................... 4 2.2 Problem Setup: Metrics and Models ....................... 5 2.3 Warm-up: Intuitive Tests for Sensitivity and Speciﬁcity ............ 7 2.4 Rigorously Evaluating Dissimilarity Metrics .................. 10 2.5 Discussion ..................................... 16 2.6 Supplementary Materials ............................. 18 3 As

FAQs

Why are metrics and benchmarks central to progress in machine learning?

They define priorities and enable efficient discovery of model designs that advance those priorities. The dissertation emphasizes that the ultimate goal is external validity: metric improvements should translate to improvements on real tasks.

What limitations are found in representation similarity metrics for interpretability?

The work shows popular metrics can disagree on fundamental observations, making it unclear which metric to trust. It then introduces practical, statistically grounded tests to evaluate these metrics and uncover distinct weaknesses.

How does the dissertation address fairness in machine learning evaluation?

It highlights limitations in the UCI Adult dataset that restrict external validity, and it contributes new datasets derived from US Census surveys to extend research on fair machine learning.

What bias is identified in protein language models, and how is it mitigated?

The dissertation develops metrics for a bias toward sequences from certain evolutionary taxa in popular protein language models. It also introduces a method to mitigate this bias and evaluate its impact on protein design.

Aug 5, 2026

Upload by Aria