Update Your Transformer to the Latest Release - Re-Basin of Task Vectors

Aug 4, 2026

Upload by Rowan

Foundation models underpin many specialized systems obtained via fine-tuning, yet updated pretrained checkpoints can quickly make existing fine-tuned models obsolete. The work studies training-free transfer of fine-tuning to a new model release by re-basing task vectors onto the new checkpoint. It adapts model re-basin for Transformer architectures using a two-level spectral approach that permutes attention heads and tunes parameters in selected head pairs. Experiments on visual and textual tasks show seamless transfer without any training steps or datapoints, with code released publicly.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages16

Reading Time40 min

AccessFree to download

Introduction

Motivation: fine-tuning becomes obsolete after checkpoint updates

Goal: data-free training-free transfer of fine-tuning

Task vector transport via model re-basin for Transformers

Proposed two-level spectral method

Experimental evaluation and results

This is the peer reviewd version of the followng article: Update Your Transformer to the Latest Release: Re-Basin of Task Vectors / Rinaldi, Filippo; Capitani, Giacomo; Bonicelli, Lorenzo; Crisostomi, Donato; Bolelli, Federico; Rodolà, Emanuele; Ficarra, Elisa; Calderara, Simone; Porrello, Angelo. - (2025) . (Intervento presentato al convegno International Conference on Machine Learning tenutosi a Vancouver, Canada nel Jul 13-19) . Terms of use: The terms and conditions for the reuse of this version of the manuscript are specified in the publishing policy. For all terms of use and more information see the publisher's website. 12/06/2025 11:04 (Article begins on next page) Update Your Transformer to the Latest Release: Re-Basin of Task Vectors Filippo Rinaldi * 1 Giacomo Capitani * 1 Lorenzo Bonicelli 1 Donato Crisostomi 2 Federico Bolelli 1 Elisa Ficarra 1 Emanuele Rodol2 Simone Calderara 1 Angelo Porrello 1 Abstract Foundation models serve as the backbone for numerous specialized models developed through fine-tuning. However, when the underlying pretrained model is updated or retrained (e.g., on larger and more curated datasets), the fine-tuned model becomes obsolete, losing its utility and requiring retraining. This raises the question: is it possible to transfer fine-tuning to a new release of the model? In this work, we investigate how to transfer fine-tuning to a new checkpoint without having to re-train, in a data-free manner. Todo so, we draw principles from model re-basin and provide a recipe based on weight permutations to re-base the modifications made to the original base model, often called task vector. In particular, our approach tailors model re-basin for Transformer models, taking into account the challenges of residual connections and multi-head attention layers. Specifically, we propose a twolevel method rooted in spectral theory, initially permuting the attention heads and subsequently adjusting parameters within select pairs of heads. Through extensive experiments on visual and textual tasks, we achieve the seamless transfer of fine-tuned knowledge to new pre-trained backbones without relying on a single training step or datapoint. Code is available at [https://](https://)[ ](https://)[github.com/aimagelab/TransFusion](github.com/aimagelab/TransFusion). 1. Introduction Recently, there has been a notable shift among researchers and practitioners towards fine-tuning pre-trained models, rather than building them from scratch. This method leverages backbones developed on large-scale datasets, consider- *Equal contribution 1AImageLab, University of Modena and Reggio Emilia, Italy. 2 Sapienza, University of Rome, Italy. Correspondence to: Filippo Rinaldi <[filippo.rinaldi@unimore.it](filippo.rinaldi@unimore.it) >. Proceedings of the 42 st International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025 . Copyright 2025 by the author(s) . Figure 1: Transporting task vector τ from a fine-tuned base model θftA = θA + τ to a new release θB . ably decreasing the amount of data and training time needed to tailor models for specific downstream tasks. For this reason, pre-trained backbones such as OpenAI’s CLIP (Radford et al., 2021) are being extensively utilized as base foundation models. As a result, the corresponding fine-tuned versions play a crucial role in numerous real-world applications like medical imaging (Lu et al., 2024) and satellite image analysis (Mall et al., 2024) . However, while these pre-trained backbones are widely adopted, their evolution poses new challenges, with tech companies and academic institutions frequently releasing updated checkpoints. Often, these updates do not modify the underlying architecture but simply consist of new weights, trained on increasingly large datasets compared to their predecessors (Ilharco et al., 2021) . Moreover, the additional training data may be more curated or specifically tailored to specialized domains, boosting

FAQs

What problem does the paper address when new pretrained checkpoints are released?

When the underlying pretrained model is updated or retrained, previously fine-tuned models can become obsolete, requiring costly retraining.

How does the proposed method transfer fine-tuning to a new model release without training?

It transports the task vector from the original base model to a suitable basin of the new release using weight permutations, re-basing the modifications without training steps or datapoints.

What are the main components of the Transformer-specific re-basin approach?

The method uses a two-level strategy based on spectral theory: first permuting attention heads, then adjusting parameters within selected pairs of heads to improve compatibility and low-loss transfer.

Aug 4, 2026

Upload by Rowan