End-to-End Learning - Audio-Visual Human-Centric Video Understanding

Aug 2, 2026

Uploaded by Violet

Machine learning has advanced rapidly through deep neural networks that learn semantic representations end-to-end from large labeled datasets. End-to-end learning depends on usable input data, target outputs, and an objective function aligning predictions to targets. This thesis addresses challenges in formatting and scaling these components. It develops Smooth-AP to optimize a differentiable approximation of Average Precision for image retrieval, and a self-supervised strategy for end-to-end image editing when target data is scarce. It also uses audio-visual signals for human-centric video understanding, including transfer from speaker verification and methods that fuse identity-discriminating modalities for scene understanding, labelling, and identity clustering.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages152

Reading Time6 h 23 min

AccessFree to download

Introduction and Background

Motivation

Key Ideas and Contributions

Enabling

End-to-End Learning, and Audio-Visual Human-Centric Video Understanding Andrew Brown Linacre College University of Oxford Supervisor: Professor Andrew Zisserman A thesis presented for the degree of Doctor of Philosophy Michaelmas 2022 Acknowledgements I am very grateful for the guidance and support of many important people throughout this journey. First, to my supervisor Andrew Zisserman. Your creative, exciting approach to problem-solving has been inspiring, and has shaped me into the researcher that I am today. It has been a privilege to work with you for the last ﬁve years. Thank you for your guidance and for seeing the potential in a Matlabwielding ﬂedgling all of those years ago. Thank you to the collaborators that I have been lucky enough to work with. Weidi Xie, who taught me the value and importance of solving hard problems. Thank you for guiding me through my ﬁrst research paper, for providing clarity with your vast knowledge of the ﬁeld, and for answering my hundreds of questions without hesitation, even as a new and tired parent. To Andrea Vedaldi, for being so generous with his insights and time. Thank you for showing me the importance of detail, and for the iconic COVID-19 memories of having equations explained to me on virtual whiteboards written on from Venetian public transport. To Arsha Nagrani, who has been an invaluable mentor to me throughout my time at VGG and guided me up the many steep learning curves. To Ernesto Coto, thank you for your unwavering patience in helping me with the many engineering mysteries encountered on this journey, such as compiling Caffe. To all my collaborators: Vicky Kalogeiton, Joon Son Chung, Max Bain, Jaesung Huh, Samuel Albanie, Yang Liu, Bruno Korbar, Cheng-Yang Fu, Omkar Parkhi, and Tamara Berg. Thank you sharing your ideas and time with me, and for making research meetings so exciting and inspiring. To all the members of the VGG family: Tengda Han, Chuhan Zhang, Yuki Asano, Olivia Wiles, Sagar Vaze, Shangzhe Wu, Mandela Patrick, Gul Varol, Lili Momeni, Rhydian Windsor, Daffy Afouras, Tomas Jakab, Xu Ji, Christian Rupprecht, Erika Lu, and João Henriques. Thank you for creating such a welcoming and supportive environment, for the lunches at St. Anne’s, and the good evenings at the Royal Oak. I feel very privileged to have learnt from all of you, and I am excited to see you all grow tobe giants in the research ﬁeld in the coming years. To my family (Jo, Phil, Will, Katie, and Candice), thank you for supporting me when I raised the prospect of spending yet another few years in education. To my friends in London who were still there for me after I disappeared off the face of the earth for some time. You all mean the world to me and Iam so lucky to have you all in my life. Finally, to Maya. Thank you for your endless encouragement, support, and understanding. Thank you for believing in my dream before I did, and for subsequently telling a young and lost undergraduate what was coming if he instead decided to go down a career-path in the City. Thank you for pushing me to follow this dream, and for encouraging me every time I fell along the way. I could not have done this without her, and I am so excited for our next journey. This is dedicated to her. Abstract The ﬁeld of machine learning has seen tremendous progress in the last decade, largely due to the advent of deep neural networks. When trained on large-scale labelled datasets, these machine learning algorithms can learn powerful semantic representations directly from the input data, end-to-end. End-to-end learning requires the availability of three core components: useful input data, target outputs, and an objective function for measuring how well the model’s predictions match the target outputs. In this thesis, we explore and overcome a series of challenges as related to assembling these three components in the sufﬁcient format and scale for end-to-end learning. The ﬁrst key idea presented in th

FAQs

什么是端到端学习，并且它需要哪些关键组成部分？

端到端学习通过深度网络直接从输入数据学习语义表示。它需要可用的输入数据、目标输出以及衡量预测与目标匹配程度的目标函数。

Smooth-AP在图像检索任务中解决了什么难点？

图像检索常用的基于排序的Average Precision度量不可微。论文提出Smooth-AP，通过优化Average Precision的平滑近似来实现端到端训练。

论文如何在缺少足够规模目标数据的情况下进行图像编辑的端到端学习？

论文提出自监督方法，通过对现成图像数据进行增强来模拟目标数据，从而在数据不足的情况下获得相对既有工作的收益。

Aug 2, 2026

Uploaded by Violet