MindAU：双流流形对齐实现脑电条件面部动作单元编辑

Jul 16, 2026

Upload by Himbo

Recent brain decoding enables reconstruction of perceived visual content from neural signals, yet EEG-guided facial expression editing remains largely unexplored. Rather than recovering what a subject sees, it requires deriving facial-action patterns from noisy EEG and grounding them in localized, identity-preserving expression edits. MindAU presents a unified EEG-conditioned framework for fine-grained facial action unit (AU) control: AU-discriminative EEG learning, Dual-Stream Manifold Alignment with multimodal Qwen2.5-VL, and an EEG-aware multimodal diffusion editor. E-CAFE benchmark supports standardized evaluation and high-fidelity editing for assistive expression use.

LanguageEnglish

CategoryHealthcare

FormatPDF

Pages20

Reading Time50 min

Abstract

Introduction

Contributions

MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment Zhenhang Li† Binghamton University [zli74@binghamton.edu](zli74@binghamton.edu) Xin Zhou† Binghamton University Massachusetts General Hospital [xzhou11@binghamton.edu](xzhou11@binghamton.edu) arXiv :2607 .004 10v 1 [ cs .CV] 1 Jul 2026 Hao Deng∗ Massachusetts General Hospital Harvard Medical School [hdeng1@mgh.harvard.edu](hdeng1@mgh.harvard.edu) Lijun Yin∗ Binghamton University [lijun@cs.binghamton.edu](lijun@cs.binghamton.edu) Abstract Recent brain decoding studies have made substantial progress in reconstructing externally perceived visual content from neural signals. However, using electroencephalography (EEG) recordings to guide facial expression editing remains largely unexplored and poses a distinct challenge: rather than recovering what a subject sees, it requires identifying facial-action related patterns from noisy EEG signals and grounding them in localized, identity-preserving expression edits. In this paper, we investigate EEG-conditioned facial image editing for fine-grained facial action unit (AU) control and propose MindAU, a unified framework for controlling facial AU edits from EEG signals. MindAU first learns noise-robust and AU-discriminative EEG representations through temporal masked reconstruction and AU classification supervision. It then bridges the modality gap via DualStream Manifold Alignment, aligning EEG features with AU-level text semantics and identity-reduced visual displacement trajectories in the multimodal space of Qwen2.5-VL. Finally, MindAU incorporates EEG-aware Multimodal Rotary Positional Embeddings, landmark-guided reference masking, and AU-aware region supervision into a multimodal diffusion-based editor for high-fidelity identitypreserving editing. We also introduce E-CAFE, a curated benchmark for EEGConditioned Action-Unit Facial Editing with paired EEG-face editing samples and standardized evaluation protocols. Extensive experiments demonstrate the effectiveness of MindAU and suggest its potential as a step towards future assistive expression technologies for individuals with facial neuromuscular disorders. 1 Introduction Recent brain decoding studies have shown remarkable progress in reconstructing externally perceived visual content from neural signals, ranging from static images [1, 2, 3] to dynamic videos [4, 5, 6] . Most existing methods focus on recovering what a subject sees, typically by aligning neural representations with visual or semantic spaces through contrastive learning [7, 8] or signal reconstruction [9] . However, an underexplored and potentially impactful direction is to move beyond passive perceptual reconstruction towards controllable facial behavior editing conditioned on brain activity, establishing a pathway from neural signals to interpretable facial actions. This is especially relevant when paired † Equal contribution. ∗ Corresponding authors. Preprint. neural recordings and facial-expression observations are scarce: electroencephalography (EEG) -conditioned facial editing may enrich EEG–face supervision and facilitate cross-modal modeling between brain activity and facial dynamics. It may also support future internal-state modeling, such as affect-or pain-related analysis, and assistive communication interfaces where expressive facial behavior is driven by neural signals. Facial expressions can be decomposed into facial action units (AUs), providing a structured and interpretable control space for linking neural signals to localized facial movements. Unlike reconstructing perceived stimuli, EEG-conditioned facial editing requires extracting subtle and noisy facial-action cues from EEG signals and grounding them in fine-grained, identity-preserving expression changes on a reference face, posing a distinct and challenging problem. To the best of our knowledge, no prior work has addressed EEG-conditioned facial image editing with AU-level con

FAQs

MindAU解决了什么核心问题？

MindAU研究的是基于脑电（EEG）的面部表情编辑：从嘈杂的EEG中提取细粒度面部动作单元（AU）线索，并将其落地到参考人脸上的局部、身份保持的表情编辑。

MindAU的框架由哪些关键模块构成？

论文提出三个主要部分：AU感知的EEG编码器（用时间掩码重建与AU分类监督学习表示）、Dual-Stream Manifold Alignment（将EEG特征对齐到Qwen2.5-VL的多模态空间），以及带EEG感知M-RoPE、参考引导掩码和AU区域监督的多模态扩散编辑器。

E-CAFE基准是什么，用于做什么评估？

E-CAFE是一个整理的基准，包含5,000对EEG-人脸编辑样本，并提供用于自参考与跨身份评估的两种标准化协议，用来衡量EEG条件面部AU编辑效果。

Jul 16, 2026

Upload by Himbo