Forget Narrowly Retain Broadly Unlearning as an Asymmetric Generalization Problem

Jul 16, 2026

Machine unlearning in large language models targets the removal of specific knowledge while preserving all other capabilities, which is essential for privacy and safety. Existing evaluation benchmarks fail to reliably verify two complementary errors: under-forgetting, where forgotten facts reappear under paraphrased or indirect queries, and over-forgetting, where unrelated knowledge is degraded. Both issues stem from an asymmetric generalization problem between a finite forget set and an implicitly defined retain set. The work introduces SUITE, a protocol and training corpus that captures forget–retain structure for real-world factual domains, improving methods and enabling JensUn++ for better trade-offs.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages58

Reading Time2 h 26 min

Introduction

Asymmetric generalization: under-forgetting and over-forgetting

SUITE: selective unlearning evaluation and training corpus

JensUn++: unlearning algorithm and improvements

arXiv :2607 .09236v 1 [ cs .LG] 10 Jul 2026 Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem Amit Peleg Naman Deep Singh Naama Pearl Bibhabasu Mohapatra Matthias Hein Tübingen AI Center, University of Tübingen Abstract Machine unlearning in LLMs is the targeted removal of specific knowledge while preserving all other capabilities, critical for privacy and safety. Yet existing benchmarks measure it unreliably. They miss knowledge that resurfaces under paraphrased or indirect queries, a failure we call under-forgetting, and lack the semantic, syntactic, and lexical probes needed to verify that unrelated knowledge is preserved, a failure we call over-forgetting. Both failures reflect an asymmetric generalization problem. Forget evaluation must cover diverse query formulations of the same target facts, testing whether forgetting holds beyond exact training prompts. Retain evaluation must probe a far larger and implicitly defined set, namely every fact disjoint from the forget target. The retain set thus defines the effective forget set, yet current datasets provide no fine-grained annotation of this forget-retain boundary. We address this with SUITE, an evaluation protocol and training corpus that captures forget-retain structure for real-world factual domains. Methods trained on SUITE improve substantially, showing that training data is as important as algorithmic design. Building on the obtained insights, we introduce JensUn++, an unlearning algorithm that achieves the best forget-retain utility trade-off across three LLMs, in both sequential and joint unlearning settings. Code and datasets are available at [https://amitpeleg.github.io/forget-narrowly-retain-broadly](https://amitpeleg.github.io/forget-narrowly-retain-broadly). 1 Introduction Although several benchmarks evaluate unlearning in LLMs [19, 42, 7, 14, 30, 21, 36], one fundamental question remains open: what does it mean to forget a fact while retaining knowledge of everything else? We argue that forgetting and retaining are governed by an asymmetric generalization problem. The forget set is finite, but successful forgetting must generalize intensively across all query formulations of the target facts: paraphrases [28, 43], indirect queries, and latent inferential dependencies on correlated facts the model still retains [48, 3, 20, 2, 13, 25] . When this fails, target information resurfaces under alternative phrasings [43, 44] or indirect queries-a failure mode we call under-forgetting. The retain scope, by contrast, cannot be enumerated: it spans “everything else” the model knows, and must generalize extensively across a much larger and only implicitly defined set-every fact disjoint from the forget set. Unlearning that ripples outward into semantically related concepts of the forget facts produces over-forgetting [37, 3, 44] . The retain set thus defines the effective forget set, and any benchmark that does not annotate this forget-retain boundary at a fine-grained level cannot distinguish genuine forgetting from suppression, nor genuine retention from collateral damage. Existing benchmarks [14, 19, 30, 45] largely lack this structure, which is a major obstacle for further progress in unlearning for LLMs. To address this, we propose SUITE: Selective Unlearning of Isolated Topics and Events, a finegrained evaluation protocol and training corpus for unlearning real-world factual knowledge that captures both sides of the asymmetry. On the forget side, we probe under-forgetting using indirect Preprint. questions requiring multi-hop reasoning, which are unseen during training (where only direct and reverse questions are used) . Combined with paraphrases of every question type, this tests whether knowledge has been genuinely forgotten or merely suppressed. On the retain side, we probe overforgetting along several axes. We check over-fitting to query form (syntactic structure) and to query entities (lexical structure)

FAQs

What is the core meaning of “forgetting” and “retaining” in LLM unlearning?

Forgetting removes targeted knowledge while retaining preserves all other capabilities. The challenge is ensuring deletion generalizes across all query formulations of the target facts, not just the exact prompts seen in training.

What are under-forgetting and over-forgetting?

Under-forgetting is when supposedly forgotten information resurfaces under paraphrased or indirect queries. Over-forgetting is when the unlearning process harms unrelated knowledge, revealed by semantic, syntactic, and lexical probes.

How does SUITE address the evaluation gap in current unlearning benchmarks?

SUITE evaluates both sides of the asymmetry by probing under-forgetting with indirect multi-hop questions and paraphrases, and probing over-forgetting across graded semantic proximity and syntactic/lexical sensitivity. It also constructs training and evaluation splits independently to test whether performance comes from data rather than only algorithm design.

Jul 16, 2026

Upload by Seraphina