iDARTS - Differentiable Architecture Search with Stochastic Implicit Gradients

Aug 2, 2026

Upload by Liam

Differentiable Architecture Search (DARTS) drives neural architecture search through an efficient gradient-based bi-level optimization that alternates between inner model weights and outer architecture parameters in a weight-sharing supernet. A central scalability and quality issue is how to differentiate through the inner-loop optimization. This work derives DARTS hypergradient computation using the implicit function theorem, making it depend only on the inner-loop solution and not on the optimization path. It further introduces a stochastic hypergradient approximation, with theory and experiments showing convergence to stationary points and architectures that outperform baseline methods.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages18

Reading Time45 min

Introduction

Background: Neural Architecture Search and DARTS

Key Challenge: Hypergradient Computation

Proposed Method: iDARTS

Contributions and Organization

iDARTS: Differentiable Architecture Search with Stochastic Implicit Gradients Miao Zhang 1 2 Steven Su 2 Shirui Pan 1 Xiaojun Chang 1 Ehsan Abbasnejad 3 Reza Haffari 1 Abstract Differentiable ARchiTecture Search (DARTS) has recently become the mainstream of neural architecture search (NAS) due to its efﬁciency and simplicity. With a gradient-based bi-level optimization, DARTS alternately optimizes the inner model weights and the outer architecture parameter in a weight-sharing supernet. A key challenge to the scalability and quality of the learned architectures is the need for differentiating through the inner-loop optimisation. While much has been discussed about several potentially fatal factors in DARTS, the architecture gradient, a.k.a. hypergradient, has received less attention. In this paper, we tackle the hypergradient computation in DARTS based on the implicit function theorem, making it only depends on the obtained solution to the innerloop optimization and agnostic to the optimization path. To further reduce the computational requirements, we formulate a stochastic hypergradient approximation for differentiable NAS, and theoretically show that the architecture optimization with the proposed method, named iDARTS, is expected to converge to a stationary point. Comprehensive experiments on two NAS benchmark search spaces and the common NAS search space verify the effectiveness of our proposed method. It leads to architectures outperforming, with large margins, those learned by the baseline methods. 1. Introduction Neural Architecture Search (NAS) is an efﬁcient and effective method on automating the process of neural network design, with achieving remarkable success on image recognition (Tan & Le, 2019 ; Li et al., 2021b ; 2020), language modeling (Jiang et al., 2019), and other deep learning ap- 1Faculty of Information Technology, Monash University, Australia 2Faculty of Engineering and Information Technology, University of Technology Sydney, Australia 3Australian Institute for Machine Learning, University of Adelaide, Australia. Correspondence to: Shirui Pan <[Shirui.Pan@monash.edu](Shirui.Pan@monash.edu) > . Proceedings of the 38 th International Conference on Machine Learning, PMLR 139, 2021 . Copyright 2021 by the author(s) . plications (Ren et al., 2020 ; Cheng et al., 2020 ; Chen et al., 2019b ; Hu et al., 2021 ; Zhu et al., 2021 ; Ren et al., 2021) . The early NAS frameworks are devised via reinforcement learning (RL) (Pham et al., 2018) or evolutionary algorithm (EA) (Real et al., 2019) to directly search on the discrete space. To further improve the efﬁciency, a recently proposed Differentiable ARchiTecture Search (DARTS) (Liu et al., 2019) adopts the continuous relaxation to convert the operation selection problem into the continuous magnitude optimization for a set of candidate operations. By enabling the gradient descent for the architecture optimization, DARTS signiﬁcantly reduces the search cost to several GPU hours (Liu et al., 2019 ; Xu et al., 2020 ; Dong & Yang, 2019a) . Despite its efﬁciency, more current works observe that DARTS is somewhat unreliable (Zela et al., 2020a ; Chen & Hsieh, 2020 ; Li & Talwalkar, 2019 ; Sciuto et al., 2019 ; Zhang et al., 2020c ; Li et al., 2021a ; Zhang et al., 2020b) since it does not consistently yield excellent solutions, performing even worse than random search in some cases. Zela et al. (2020a) attribute the failure of DARTS to its supernet training, with empirically observing that the instability of DARTS is highly correlated to the dominant eigenvalue of the Hessian matrix of the validation loss with respect to architecture parameters. On the other hand, Wang et al.(2021a) turn to the magnitude-based architecture selection process, who empirically and theoretically show the magnitude of architecture parameters does not necessarily indicate how much the operation contributes to the supernet's performance. Chen & Hsieh (2020) observe a precipit

FAQs

What problem does iDARTS address in DARTS?

iDARTS targets the difficulty of computing the architecture hypergradient, i.e., differentiating through the inner-loop optimization in DARTS to improve scalability and architecture quality.

How does iDARTS compute the hypergradient?

iDARTS derives the hypergradient using the implicit function theorem so the computation depends on the inner-loop solution and is independent of the optimization path.

How does iDARTS reduce computational cost?

It approximates the needed inverse via a Neumann series and further introduces a stochastic hypergradient approximation to reduce the computational burden for differentiable NAS.

Aug 2, 2026

Upload by Liam