PPE-Bench：评估在私人-公共纠缠下的多模态大语言模型遗忘基准

Jul 16, 2026

Upload by Aurora

Multimodal Large Language Models (MLLMs) can memorize sensitive private information from web data, creating privacy and copyright risks. Machine unlearning removes private knowledge without full retraining, yet current MLLM unlearning benchmarks fall short: they use overly simplified images and assume a fully separated forget/retain split. PPE-Bench introduces entangled private-public scenes where the target to forget coexists with public figures and landmarks, plus methods to preserve public context. Experiments show prior approaches often leak less privacy but significantly degrade adjacent public information.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

Introduction

Existing Benchmarks

Limitations of Current Benchmarks

Proposed PPE-Bench

Methods for Preserving Public Information

Experimental Findings

PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement Xianren Zhang1 , Delvin Ce Zhang2 , Dongwon Lee1 , Suhang Wang1 1The Pennsylvania State University 2 University of Sheffield {xzz5508,dongwon,[szw494}@psu.edu](szw494}@psu.edu), [delvin.ce.zhang@sheffield.ac.uk](delvin.ce.zhang@sheffield.ac.uk) arXiv :2607 .02897v 1 [ cs .CR] 3 Jul 2026 Abstract Multimodal Large Language Models (MLLMs) have shown strong capabilities, but they may memorize private information from web data, raising privacy concerns. Machine unlearning offers a way to remove such private knowledge without retraining from scratch. However, existing MLLM unlearning benchmarks have two major limitations. First, they rely on simplified images that contain only the single target individual, failing to reflect the visual complexity of real-world photos. Second, they typically assume that the forget set and retain set are fully separated, ignoring the fact that private information is often visually entangled with benign public information. For example, a private individual may appear with a public figure orin front of a well-known landmark, where unlearning the private target should not damage the public context. To address these limitations, we propose PPE-Bench, a new benchmark for evaluating MLLM unlearning under privatepublic entanglement. Each image contains a target individual to be forgotten and public information to be preserved, including public figure and landmark. We further introduce two simple but effective methods to better preserve public information during unlearning. Through experiments, we find that existing unlearning methods can reduce private information leakage, but often substantially harm adjacent public information. 1 1 Introduction Multimodal Large Language Models (MLLMs) have demonstrated strong performance on a wide range of multimodal tasks (Li et al., 2024a), such as visual question answering (Kuang et al., 2025) and image captioning (Sarto et al., 2025) . However, as MLLMs are trained on large-scale data from internet that may contain sensitive and private information, they can memorize and reproduce such 1Data: [https://github.com/Zood123/PPE_Bench](https://github.com/Zood123/PPE_Bench) content (Huang et al., 2024), which raises significant privacy and copyright concerns. Privacy regulations like GDPR (Hoofnagle et al., 2019) and CCPA (Pardau, 2018) enforce the right to be forgotten (Dang, 2021) . For instance, personal images and online profile information shared on social media and websites could unintentionally be included in the training data (Caldarella et al., 2024 ; Yanet al., 2024), causing privacy issues. In such cases, image owners may request that MLLMs forget the influence of this data. However, retraining MLLMs from scratch to remove sensitive knowledge is often impractical due to the high computational cost. As a result, unlearning methods (Liu et al., 2025 ; Huo et al., 2025 ; Zhaopan Xu et al., 2025 ; Li et al., 2024b ; Wu et al., 2025) are applied to MLLM models to “forget” such sensitive information without retraining the model from scratch. For example, some methods (Liu et al., 2025 ; Huo et al., 2025) try to remove visual patterns associated with specific entities, such as personal information including home address, occupation, and age. These methods typically finetune MLLMs using different objectives, such as maximizing the loss on private information or minimizing preference scores for sensitive content. Gradient Ascent (GA) (Yao and Xu, 2024), Gradient Difference (GD) (Liu et al., 2022) or Negative Preference Optimization (NPO)(Zhang et al., 2024) are common approaches used for MLLM unlearning. Recently, several benchmarks (Dontsov et al., 2025 ; Liu et al., 2025) have been designed to evaluate the effectiveness of unlearning methods under the multimodal setting. CLEAR (Dontsov et al., 2025) is the first open-sourced benchmark specifically for multi

FAQs

PPE-Bench解决了哪些现有MLLM遗忘基准的主要不足？

现有基准使用过于简化的图像（仅含目标个体），且将forget set与retain set完全分离，忽略私人信息与公共信息在真实场景中的视觉纠缠。

PPE-Bench的图像构建包含哪些元素？

每张图像都包含需要遗忘的目标个体，同时包含需要保留的公共信息，例如公共人物与知名地标。

实验结果显示了什么权衡问题？

实验表明，现有遗忘方法确实能降低私人信息泄露，但往往会显著损害与之相邻的公共信息。

Jul 16, 2026

Upload by Aurora