From Public Health to AI Safety - Improving Machine Learning Approaches by Collecting, Selecting, or Reducing the Need for High-Quality Data

Aug 2, 2026

Upload by Asher

This thesis advances multiple application areas spanning public health and AI safety through novel data-centric machine learning methods. It introduces a training-data selection approach, rigorous large-scale manual collection processes, synthetic data generation when data is scarce, and highly diverse test datasets to measure generalisability. For AI misuse and misalignment, it builds a detector for LLM “lies,” using LLM prompting to produce lie-labeled training data and studying generalisation to unseen settings. It also addresses low-quality abundant data via automatic selection during training, improving speed, accuracy, and compute efficiency. Finally, it collects carefully validated high-quality COVID-19 datasets and fits Bayesian models to estimate NPI effects with robust validation that supports policy decisions.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages330

Reading Time13 h 52 min

AccessFree to download

Abstract

AI misuse and misalignment: lie detection data

Low-quality abundant data: automatic selection

High-stakes public health: COVID-19 NPI data collection and Bayesian modelling

Overall data-centric contributions

From Public Health to AI Safety: Improving Machine Learning Approaches by Collecting, Selecting, or Reducing the Need for High-Quality Data Jan Brauner Department of Computer Science University of Oxford A thesis submitted for the degree of Doctor of Philosophy Hilary Term 2024 2 Abstract In this thesis, we make advances in several application areas, from public health to AI safety. In the process, we develop novel machine learning (ML) methods to tackle various challenges. In each application area, we take a data-centric approach: we develop a method to automatically select high-quality training data, devise and execute rigorous processes for large-scale manual data collection, generate synthetic training data when data is otherwise hard to obtain, and collate highly diverse test data to evaluate the generalisability of our method. First, we tackle AI misuse and misalignment by building a detector for “lies”—which we define as particular types of falsehoods—output by large language models (LLMs) . This endeavour is complicated by the difficulty in acquiring appropriate training data. In the case of misuse, an attacker might use a wide range of types of LLM or lie-generation methods; obtaining a sufficiently diverse dataset to cover all situations is challenging. In the case of misalignment, the lies can be subtle, such as convincing but false text that plays to human biases; identifying a sufficient number of examples to build a dataset from can be difficult. Our approach offers a potential remedy: We generate training data by simply prompting an LLM with instructions to lie, making it easy to collect a large dataset. We then collate a large, diverse test dataset and extensively study the generalisation of our lie detector to previously unseen settings. We find that it generalises surprisingly well. In the second part of the thesis, we deal with a situation where we have abundant training data available, but some of it is of low quality. This is common in modern ML, as deep learning models require vast amounts of data, often scraped from the internet. We develop a method to automatically select the most useful data at any given point in training, enabling us to train neural networks in fewer steps, to higher accuracy, and with lower computational costs. In the third part of the thesis, a necessity for exceptionally high dataquality arises due to the high stakes of the modelling outcomes. During the COVID-19 pandemic, governments worldwide needed information on which nonpharmaceutical interventions (NPIs) could most effectively curb virus transmission. While this question could be tackled with modelling approaches, existing data on the timing and nature of interventions across countries was incomplete and inaccurate. To guide high-stakes policy decisions, we need carefully validated, high-quality data. We collect two datasets with extensive quality-control measures as the basis of our modelling studies. We then fit multiple Bayesian models to infer the effect of various government interventions on viral transmission. Our effectiveness estimates are backed with rigorous model validation experiments, a step often lacking in other work on this topic. The resulting NPI effectiveness estimates informed policy decisions in various countries. Overall, we make progress in different application areas by leveraging different data-centric approaches: generating synthetic training data, collating diverse test data to evaluate our method, selecting the most useful data when data is abundant, and collecting high-quality data when existing datasets are insufficient. DRAFT Printed on August 3, 2024 From Public Health to AI Safety: Improving Machine Learning Approaches by Collecting, Selecting, or Reducing the Need for High-Quality Data Jan Brauner Department of Computer Science University of Oxford A thesis submitted for the degree of Doctor of Philosophy Hilary Term 2024 Acknowledgements I thank: My

FAQs

How does the thesis improve machine learning performance across different application areas?

It applies a data-centric strategy: selecting high-quality training data, running large-scale manual collection, generating synthetic data when necessary, and evaluating with diverse test datasets.

What is the role of the lie detector in the AI safety part of the thesis?

The thesis builds a detector for specific types of falsehoods generated by large language models and uses LLM prompting to create training data for that detector.

How does the thesis handle public health questions with limited or unreliable datasets?

It creates two COVID-19 datasets using extensive quality-control measures, then fits multiple Bayesian models and validates them rigorously to infer the effects of government interventions.

Aug 2, 2026

Upload by Asher