Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations

Jul 16, 2026

Knowledge distillation (KD) often uses a single, fixed-teacher perspective, which limits diversity in supervisory signals and may reduce generalization. Multi-teacher KD improves diversity but is expensive in computation and storage. Virtual-view generation from one teacher trades off efficiency versus controlled diversity. Shift-Augmented Knowledge Distillation (SAKD) uses evolving student features as dynamic conditions to generate parameter-free cyclic shifts, enabling single-stage training. Experiments on CIFAR-100 and ImageNet show SAKD surpasses random perturbations and matches two-stage accuracy with fewer parameters and no pre-training.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages8

Reading Time20 min

Introduction

Knowledge distillation and its limitations

Existing single-teacher virtual-view methods

Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations Xuyi Yu, Yaohua Liu, Ziming Song, Yinghai Zhao, Huipeng Zhang, and Kuizhi Mei, Member, IEEE arXiv :2607 . 11557v1 [ cs .CV] 13 Jul 2026 Abstract—Knowledge distillation (KD) typically relies on the fixed perspective of a single teacher, limiting the diversity of supervisory signals. While multi-teacher distillation addresses this by aggregating knowledge from multiple models, it incurs prohibitive computational and storage costs. To balance efficiency and diversity, recent research has focused on generating virtual views from a single teacher. However, existing methods face a trade-off: random perturbation approaches offer efficiency but lack controlled diversity, while structured augmentation methods require multi-stage training and incur linear parameter growth. We observe that this trade-off stems from a common design choice: using the teacher’s strong but static features to generate views. Instead, we propose Shift-Augmented Knowledge Distillation (SAKD), a simple yet effective framework that leverages the student’s evolving features as a dynamic condition for perturbation generation. This shift in perspective enables singlestage training while producing adaptive, diverse views through a parameter-free cyclic shift. Extensive experiments on CIFAR-100 and ImageNet demonstrate that SAKD consistently outperforms random perturbation methods and achieves accuracy on par with two-stage approaches, while using significantly fewer parameters and eliminating pre-training requirements. Index Terms—Knowledge distillation, Model compression, Teacher augmentation, Structured perturbation I. INTRODUCTION The deployment of deep neural networks in edge devices, mobile platforms, and other resource-constrained environments is often hindered by their substantial computational and memory requirements. Knowledge distillation (KD) [1] has emerged as a leading solution, transferring knowledge from a cumbersome but accurate teacher model to a lightweight student. While original KD aligned softened output probabilities, subsequent research has expanded to richer supervisory forms including feature-based [2], [3], [4], [5], relation-based [6], [7], [8], and logit-based approaches [9], [10], [11], [12],[13] . Despite these advances, a critical limitation remains: the student receives supervision from only one fixed perspective of the teacher, limiting the diversity of guidance and potentially constraining generalization. Xuyi Yu, Huipeng Zhang, and Kuizhi Mei are with the State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an 710049, China (e-mail: [yuxuyi@stu.xjtu.edu.cn](yuxuyi@stu.xjtu.edu.cn); [zhp429858387@stu.xjtu.edu.cn](zhp429858387@stu.xjtu.edu.cn); [meikuizhi@mail.xjtu.edu.cn](meikuizhi@mail.xjtu.edu.cn)). Yaohua Liu is with the Guangdong Institute of Intelligence Science and Technology, Zhuhai 519031, China (e-mail: [liuyaohua@gdiist.cn](liuyaohua@gdiist.cn)). Ziming Song is with the Institute of Collaborative Innovation, University of Macau, Taipa, Macau SAR, China (e-mail: zimingsong [um@163.com](um@163.com)). Yinghai Zhao is with the Beijing Huahang Institute of Radio Measurement, Beijing 102445, China (e-mail: [513043512@qq.com](513043512@qq.com)). Multi-teacher distillation aggregates knowledge from multiple independently trained models to mitigate this issue [14],[15], [16], [17] . However, the cost of training and storing multiple large teachers is often prohibitive. A more practical direction generates multiple virtual teacher perspectives from a single pre-trained model. Existing approaches fall into two categories. Methods like TeKAP [18](Fig. 1(a)) inject random noise into teacher features or logits during training, creating diversity through controlled stochasticity in a single-stage, parameter-efficient man

FAQs

What problem does single-teacher knowledge distillation face?

The student is supervised using only one fixed teacher perspective, which limits diversity of guidance and can constrain generalization.

How does SAKD differ from random perturbation and structured augmentation methods?

SAKD generates diversity using dynamic, evolving student features and a parameter-free cyclic shift, avoiding the randomness of random perturbations and the multi-stage parameter growth of structured reconstruction approaches.

What do the experiments on CIFAR-100 and ImageNet show for SAKD?

SAKD consistently outperforms random perturbation methods and reaches accuracy comparable to two-stage approaches while using significantly fewer parameters and eliminating pre-training requirements.

Jul 16, 2026

Upload by Theodora