Limited Data, Unlimited Potential - A Study on ViTs Augmented by Masked Autoencoders - arXiv 2310.20704v2

Sep 15, 2026

Uploaded by Cart

Vision Transformers (ViTs) are widely used in computer vision but often struggle under limited training data due to their weaker inductive biases than CNNs. This study examines why jointly optimizing ViTs for a primary task together with a Self-Supervised Auxiliary Task (SSAT) can outperform the common SSL-then-fine-tuning pipeline. Experiments on 10 image classification datasets and two video datasets show SSAT improves accuracy while lowering carbon footprint, including generalization to deepfake detection, supported by additional model and feature analyses.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Abstract

Introduction

Related Work

Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders arXiv :2310 .20704v2 [ cs .CV] 27 Dec 2023 Srijan Das 1 , Tanmay Jain2 , Dominick Reilly 1 , Pranav Balaji3 , Soumyajit Karmakar4 , Shyam Marjit4 , Xiang Li5 , Abhijit Das3 , and Michael S. Ryoo5 1UNC Charlotte, 2 Delhi Technological University, 3 BITS Pilani Hyderabad 4 Indian Institute of Information Technology Guwahati, 5 Stony Brook University[sdas24@charlotte.edu](sdas24@charlotte.edu) Abstract Vision Transformers (ViTs) have become ubiquitous in computer vision. Despite their success, ViTs lack inductive biases, which can make it difficult to train them with limited data. To address this challenge, prior studies suggest training ViTs with self-supervised learning (SSL) and fine-tuning sequentially. However, we observe that jointly optimizing ViTsfor the primary task and a Self-Supervised Auxiliary Task (SSAT) is surprisingly beneficial when the amount of training data is limited. We explore the appropriate SSL tasks that can be optimized alongside the primary task, the training schemes for these tasks, and the data scale at which they can be most effective. Our findings reveal that SSAT is a powerful technique that enables ViTs to leverage the unique characteristics of both the self-supervised and primary tasks, achieving better performance than typical ViTs pre-training with SSL and fine-tuning sequentially. Our experiments, conducted on 10 datasets, demonstrate that SSAT significantly improves ViT performance while reducing carbon footprint. We also confirm the effectiveness of SSAT in the video domain for deepfake detection, showcasing its generalizability. Our code is available at [https://github.com/dominickrei/Limited](https://github.com/dominickrei/Limited)data-vits. 1. Introduction Vision Transformers (ViTs) have become a common sight in computer vision owing to their success across various visual tasks, and are now considered a viable alternative to Convolutional Neural Networks (CNNs) . Despite this, ViTs are structurally deficient in inductive bias compared to CNNs, which necessitates training them with largescale datasets to achieve acceptable visual representation, as noted by Dosovitskiy et al. [13] . As a result, when dealing with small-scale datasets, it is essential to utilize a ViT pre-trained on a large-scale dataset such as ImageNet [12] relative accuracy 30 20 10 0 -10 -20 ViT-T (5 .4M) ViT-S (21 .4M) size of data (log scale) size of data (log scale) Figure 1 . Relative classification accuracy on three datasets with different sizes: (i) Oxford Flower [37] (2K samples), (ii) CIFAR [25](50K samples), and (iii) ImageNet-1K [12] (IN-1K, 1.2M samples) . SSAT consistently outperforms others on all three datasets with two backbones. On the other hand, given the same SSL method, SSL+FT achieves a compromised performance than SSAT, especially on the tiny Oxford Flower dataset (even worse than training from scratch) . or JFT-300M [50] . However, in domains such as medical datasets, pre-training ViTs on ImageNet or JFT-300M may not result in an optimal model for fine-tuning on those datasets due to a significant domain gap. Thus, the aim of this research is to address the following question: how can ViTs be trained effectively in domains with limited data? Following the introduction of ViTs, second-generation vision transformers have emerged with two different approaches. The first approach is to use a hierarchical structure to introduce inductive bias in ViTs [18, 34, 54] . The second approach involves using hybrid architectures, such as introducing convolutional blocks within ViTs [39, 57] . However, both approaches primarily benefit medium-sized datasets and not small-scale datasets. Several efforts have been made to enhance their locality inductive bias, as reported in literature [15, 28, 29, 33] . Among these methods, SSL has demonstrated exceptional efficacy in training transformers

FAQs

Why do Vision Transformers (ViTs) require special handling when training data is limited?

ViTs lack strong inductive biases, making it difficult to learn effective visual representations from small datasets.

What is SSAT and how does it differ from SSL+FT?

SSAT jointly optimizes a self-supervised auxiliary task with the primary task in a single training process, while SSL+FT trains sequentially with self-supervision followed by fine-tuning.

Which auxiliary task setting works best in this study?

The results show SSAT is most effective when the auxiliary task is image reconstruction from missing pixels using masked autoencoder-style objectives.

Sep 15, 2026

Uploaded by Cart