Cross-Modal Learning from Visual Information for Activity Recognition on Inertial Sensors - 博士论文致谢与摘要

Aug 5, 2026

Upload by Ivy

The lack of large-scale, labeled datasets limits robust, generalized human activity recognition (HAR) from wearable inertial sensors. Inertial data collection is costly and annotation is time-consuming and error-prone, making public datasets small in subjects, classes, recorded hours, and environmental diversity. Models trained on these datasets fail to cover real-world activity variations across populations, slowing wearable sensing progress. This thesis leverages the visual modality as a source domain for cross-modal learning to transfer data and knowledge into inertial HAR, enabling scalable learning and improved generalization.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages148

Reading Time6 h 13 min

AccessFree to download

Acknowledgements

Abstract

Contents

1 Introduction

1.1 Research Questions and Contributions

1.2 Publications

2 Background

2.1 Activity Recognition Using Wearable Inertial Sensors

2.2 Learning Across Modalities

2.3 Summary

3 Generating Virtual Inertial Data from Videos

3.1 Introduction

3.2 IMUTube

3.3 Experiments

3.4 Discussion

3.5 Related Work

3.6 Summary

Cross-Modal Learning from Visual Information for Activity Recognition on Inertial Sensors 􀀀 Catherine Tong University of Oxford A thesis submitted for the degree of Doctor of Philosophy Trinity Term 2021-2022 Acknowledgements My gratitude for help in writing this thesis runs wide and deep. As ever, my thanks go ﬁrst to my supervisor Nic Lane. He brought his wisdom and experience to my support through all important moments during my studies. I thank him especially for taking a leap of faith in a Physics student who knew very little about the world of Computer Science. At the very beginning ofthe DPhil programme, I took up an internship at Nokia Bell Labs in Cambridge. It turned out to an invaluable experience that sparked my interest in Ubicomp. I owe thanks to Fahim Kawsar, Akhil Mathur, Sourav Bhattacharya, Alberto Gil Ramos, and Vincent Tseng, who ﬁrst introduced me to the ﬁeld. I also would like to thank Danielle Belgrave who was my manager during an internship at Microsoft Research Cambridge, for whose advice I am particular grateful. At Oxford, I spent a considerable amount of time at the Big Data Institute working with Aiden Doherty and his group, namely Shing Chan, Hang Yuan, Rosemary Walmsley and Andrew Creagh, who always made me feel welcomed. I am also grateful to my friends and family who have been a constant source of support and advice during the past few years, particularly: Milad Alizadeh, Javier Fernandez-Marques, Shyam Tailor, Edgar Liberis, Shuyu Lin, Ronnie Clark, Bo Yang, Yuge Shi, Hyeokhyen Kwon, Harish Haresamudram, Chongyang Wang, Yan Gao, Xinchi Qiu, Priscilla Wong, Emma Rocheteau, Elden Tse, Dongge Han and Lily Liu. I also thank my mother, without whom my journey in the UK would not have even started. Finally, I would like to thank William, who has to endure the ups and downs of a PhD without earning one himself. He has powered me through different challenges and this thesis would not have been possible without him. Abstract The lack of large-scale, labeled datasets impedes progress in developing robust and generalized predictive models for human activity recognition (HAR) from wearable inertial sensor data. Labeled data is scarce as sensor data collection is expensive, and their annotation is time-consuming and error-prone. As a result, public inertial HAR datasets are small in terms of number of subjects, activity classes, hours of recorded data, and variation in recorded environments. Machine learning models, developed using these small datasets, are effectively blind to the diverse expressions of activities performed by wide-ranging populations in the real world, and progress in wearable inertial sensing is held back by this bottleneck for activity understanding. But just as Internet-scale text, image and audio data have pushed their respective pattern recognition ﬁelds to systems reliable enough for everyday use, easy access to large quantities of data can push forward the ﬁeld of inertial HAR, and by extension wearable sensing. To this end, this thesis pioneers the idea of exploiting the visual modality as a source domain for cross-modal learning, such that data and knowledge can be transferred across to beneﬁt the target domain of inertial HAR. This thesis makes three contributions to inertial HAR through cross-modal approaches. First, to overcome the barrier of expensive inertial data collection and annotation, we contribute a novel pipeline that automatically extracts virtual accelerometer data from videos of human activities, which are readily annotated and accessible in large quantities. Second, we propose acquiring transferable representations about activities, from HAR models trained using large quantities of visual data to enrich the development of inertial HAR models. Finally, the third contribution exposes HAR models to the challenging setting of zero-shot learning; we propose mechanisms that leverage cross-modal correspondence to enable inference on pr

FAQs

Why are large-scale labeled inertial datasets important for activity recognition?

Large-scale labeled data enables robust and generalized predictive models. Without it, inertial HAR models trained on small public datasets struggle to handle diverse real-world activity expressions.

How does the thesis propose to obtain inertial information without expensive inertial collection?

It introduces a pipeline that automatically extracts virtual accelerometer data from videos of human activities, leveraging readily annotated and abundant visual data.

What does the thesis mean by zero-shot learning in inertial HAR?

It studies mechanisms that use cross-modal correspondence so HAR models can infer on previously unseen activity classes.

Aug 5, 2026

Upload by Ivy