Variational Autoencoders for Supervision, Calibration and Multimodal Learning - Thesis

Aug 2, 2026

Uploaded by Violet

Learning representations of data is a central goal in machine learning for efficient downstream use such as classification and object detection. Interpretability is also essential, enabling fine-grained intervention and reasoning over input characteristics. This thesis uses Variational Autoencoders to incorporate label information into latent learning, build shared representations for multimodal data, and calibrate predictions from neural classifiers. It develops methods for structured latent spaces with available labels, handles missing labels to reduce data requirements, and learns symmetric mutual supervision for cross-modal generation. Finally, it produces reliable confidence estimates via fast post-hoc calibration.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages179

Reading Time7 h 31 min

AccessFree to download

1 Introduction

1.1 Preamble

1.2 Variational Autoencoder

1.3 Variational Autoencoders for Supervision, Calibration and Multimodal Learning

1.3.1 Capturing Label Characteristics in VAEs

1.3.2 Multi-Modal learning through Mutual Supervision

1.3.3 Sample-dependent Temperature Scaling for Improved Calibration

2 Literature Review and Background

2.1 Capturing Label Characteristics in VAEs

2.2 Learning Multimodal VAEs through Mutual Supervision

2.3 Sample-dependent Temperature Scaling for Improved Calibration

Variational Autoencoders for Supervision, Calibration and Multimodal Learning Thomas W. Joy St Catherine’s College University of Oxford A thesis submitted for the degree of Doctor of Philosophy Trinity 2022 Acknowledgements I would like to extend my deepest gratitude to my supervisor Phil and my cosupervisor Sid. Their consistent and unwavering support has been invaluable throughout my PhD, I would like to thank them for believing in me and providing me with unbridled opportunities to undertake research, allowing me to develop both professionally and personally. I would also like to thank Puneet, for his inspirational mentorship and guidance throughout my PhD. I would also like to extend my gratitude to Pawan, who supervised me throughout my Masters degree and set me on a path towards a PhD. His support and patience provided me with the confidence and vision to pursue a path in research. I would also like to thank my undergraduate tutors, Byron Byrne and David Gillespie for their academic and administrative support during my applications. Furthermore, I would also to thank the staff at St Catherines College for being the unsung heroes and heroines of the university. I would also like to particularly thank Joanna Zapisek for her continued support and reassurance throughout the process. From a personal point of view, I wold like to thank my family for believing in meand providing essential emotional support. I would also like to thank my friends for their empathy during the difficult times and covering the Uber when I was financially insolvent. Finally, to anyone else who was part of the past four years, no matter how insignificant, I would like to say thank you for being a part of this capricious but immensely gratifying period of my life. Abstract Learning representations of data has long been a desirable goal in machine learning. Constructing such representations enables downstream tasks such as classification or object detection to be preformed efficiently. Furthermore, it is desirable to have these representations be constructed in such a way so they are interpretable, which allows for fine grained intervention and reasoning on characteristics of the input. Other tasks may include, cross-generation between modalities, or calibrating predictions such that their confidence matches their accuracy. An effective way to learn representations is through a Variational Autoencoder (VAE), which performs variational inference on the latent variables of the observable input. In this thesis we show how the VAE, can be utilised to: incorporate label information into the learning process; learn shared-representations of multimodal data; and calibrate predictions of existing neural classifiers. Data sources are often accompanied by additional label information, which may indicate the presence of a characteristic in the input. A question naturally arises as to whether the additional label information can be used to structure the representation such that it provides a notion of interoperability about the characteristic; such as “to what extent is the person smiling?”. The first contribution of this thesis is to address the aforementioned problem and propose a method which successfully uses label information to structure the latent space. Furthermore, this allows us to perform additional tasks such as fine grained interventions; classification; and conditional generations. Moreover, we are also successfully able to handle the case when label information is missing, drastically reducing the data burden when training these models. Rather than being presented with labels, we sometimes instead observe another unstructured observation of the same object, e .g. a caption of an image. In this scenario, the objective changes slightly to one where the model is able to learn shared-representations of data, allowing it to perform cross-generations between modalities. The second contribution of this theses addresses this problem.

FAQs

How does this thesis use variational autoencoders to improve learning representations?

It employs Variational Autoencoders to perform variational inference on latent variables and to structure representations for downstream tasks. The work tailors VAE usage for supervision, multimodal representation learning, and prediction calibration.

What is the first main contribution related to supervision and labels?

The thesis proposes a method that uses available label information to structure the latent space so it supports further tasks such as fine-grained interventions and conditional generation. It also addresses missing-label settings to reduce training data burden.

How does the thesis handle multimodal learning when labels are not available?

When labels are not provided, it learns shared representations by introducing mutual supervision between modalities and a bi-directional objective that enforces symmetry. This enables cross-generation and robustness when some modalities are missing during training.

Aug 2, 2026

Uploaded by Violet