Engineering flexible machine learning systems by traversing functionally-invariant paths - Differential geometry framework

Aug 5, 2026

Upload by Anda

Transformers are the foundation of modern neural networks for language and vision, yet the mathematics that govern adaptation without losing prior knowledge remains unclear. This work proposes a differential geometry framework, functionally-invariant paths (FIP), treating neural weights as a curved Riemannian manifold. Adaptation is defined as geodesic movement in weight space aligned with secondary objectives, using the metric spectrum to identify low-rank subspaces. With modest computation, FIP achieves competitive results on continual learning and sparsification across BERT, vision transformers, and CNNs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

AccessFree to download

Abstract

Introduction

Transformers and foundation models

Weight adaptation and fine-tuning

Geometric view of weight space

Problem setting: preventing knowledge loss

arXiv :2205 .00334v4 [ cs .LG] 3 Sep 2023 Engineering flexible machine learning systems by traversing functionally-invariant paths Guruprasad Raghavan 1 , Bahey Tharwat2 ,Surya Narayanan Hari 1 , Dhruvil Satani 1 , Matt Thomson 1∗ 1 Department of Biology and Biological Engineering, Caltech 1200 E California Blvd, Pasadena, CA 91125 2 Alexandria University Alexandria, Egypt ∗ To whom correspondence should be addressed; [mthomson@caltech.edu](mthomson@caltech.edu) [graghava@caltech.edu](graghava@caltech.edu) Abstract Transformers have emerged as the state ofthe art neural network architecture for natural language processing and computer vision. In the foundation model paradigm, large transformer models (BERT, GPT3/4, Bloom, ViT) are pre-trained on self-supervised tasks such as word or image masking, and then, adapted through fine-tuning for downstream user applications including instruction following and Question Answering. While many approaches have been developed for model fine-tuning including low-rank weight update strategies (eg. LoRA), underlying mathematical principles that enable network adaptation without knowledge loss remain poorly understood. Here, we introduce a differential geometry framework, functionally invariant paths (FIP), that provides flexible and continuous adaptation of neural networks for a range of machine learning goals and network sparsification objectives. We conceptualize the weight space of a neural network as a curved Riemannian manifold equipped with a metric tensor whose spectrum defines low rank subspaces in weight space that accommodate network adaptation without loss of prior knowledge. We formalize adaptation as movement along a geodesic path in weight space while searching for networks that accommodate secondary objectives. With modest computational resources, the FIP algorithm achieves comparable to state of the art performance on continual learning and sparsification tasks for language models (BERT), vision transformers (ViT, DeIT), and the CNNs. Broadly, we conceptualize a neural network as a mathematical object that can be iteratively transformed into distinct configurations by the path-sampling algorithm to define a sub-manifold of weight space that can be harnessed to achieve user goals. Introduction Transformers are now the state of the art machine learning paradigm for natural language understanding, computer vision, biological sequence analysis, and context sensitive reasoning tasks [19, 54, 5, 7] . As models have scaled in parameter number, models trained on generic masking tasks have exhibited emergent behaviors including zero-shot task performance, generalized reasoning, and instruction following [7, 11, 43] . In the ‘foundation model’ paradigm [5], transformers with 108 to 1012 parameters are trained over large data sets on self-supervised tasks such as masked language modeling, causal language modeling, or image masking [22, 43, 12, 11] . Following self supervised training, models can be adapted to increase performance on specific applications including ques- tion/answer, instruction following, distillation of financial or medical documents, and sparsified or quantized to reduce memory requirements and inference speeds in deployment environments. Due to the central role of model adaptation for transformer optimization and deployment, many algorithms have emerged for updating model weights to increase performance without experiencing a catastrophic loss of the knowledge gained during self-supervised pre-training. For example, LoRA (Low Rank Adaptation) exhibits impressive performance on fine-tuning of billion parameter models through low rank weight updates enforced by matrix factorization [24] . Empirical experiments with LoRA find that low-rank updates enable fine tuning of models with 100B parameters in benchmark tasks. However, the performance of fine-tuning and sparsification frameworks remains predominantly grounded in empirical results. The machin

FAQs

What is functionally-invariant paths (FIP) in this paper?

FIP is a differential geometry framework that enables flexible, continuous neural network adaptation by traversing geodesic paths in weight space while aiming to maintain functional invariance.

How does the paper model neural network adaptation mathematically?

It treats the weight space as a curved Riemannian manifold with a metric tensor whose spectrum defines low-rank subspaces, and formalizes adaptation as movement along geodesic paths aligned with secondary objectives.

What tasks does the FIP algorithm target and how does it perform?

FIP targets continual learning and network sparsification objectives and reports performance comparable to state of the art on language models (e.g., BERT), vision transformers, and CNNs with modest computation.

Aug 5, 2026

Upload by Anda