DS-MTNet Structured Multi-Task EEG Decoding for Human-Machine Collaboration

Jul 16, 2026

Upload by Riley

Human-machine collaboration systems often miss operators’ internal perceptual, intention-related, and state-related processes, relying mainly on environment-facing sensors. Electroencephalography (EEG) offers a noninvasive, time-resolved signal that can act as an additional sensing channel for HMC. The proposed DS-MTNet integrates EEG waveforms, task-routed source embeddings, and temporal-spectral power features into reusable slots with dual gating to route task-specific components. Evaluated on sustained-attention driving EEG data, it improves mean performance across multiple readouts and strengthens steering-response stage decoding, producing unified source-slot evidence.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages11

Reading Time28 min

Introduction

Motivation and limitation of current HMC sensing

EEG as an additional information source

Challenge of decoding mixed EEG evidence

Overview of the proposed structured multi-task framework

DS-MTNet: Structured Multi-Task EEG Decoding for Human-Machine Collaboration Xinjia Yua, Yang Zhoua, Jing Yanga, Tielin Shia, and Tao Chengb* aSchool of Mechanical Science and Engineering, Huazhong University of Science and Technology, 1037 Luoyu Road, Wuhan, 430074, China bSchool of Urban Transport and Logistics, Shenzhen Technology University, 3002 Lantian Road, Shenzhen, 518118, China Abstract—Current human-machine collaboration (HMC) systems rely on environment-facing sensors to observe visible actions and scene states, but the internal perceptual, intention-related, and staterelated processes of operators remain insufficiently integrated into machine perception. Electroencephalography (EEG) provides a noninvasive, time-resolved modality to capture neural activity associated with these processes and can serve as an additional sensing channel in HMC. However, HMC-relevant EEG evidence is often mixed in continuous recordings. Existing EEG decoding methods usually target task-specific classification or aggregate prediction, so multiple HMCrelevant readouts are rarely organized in a unified EEG representation. To address this gap, this paper proposed the Decomposed-Source Multi-Task Network (DS-MTNet), a structured multi-task EEG decoding framework. DS-MTNet integrated three streams, namely EEG waveforms, task-routed source embeddings, and temporalspectral power features, into reusable slots and used dual gating mechanisms to route task-specific components. The model was tested on a sustained-attention driving EEG dataset with three representative readouts: lane-departure-related epochs for environmental-event processing, steering-response stage for response preparation, and reaction-time-defined alertness state for internal state. DS-MTNet achieved the best mean performance among traditional, single-task deep, and multi-task EEG baselines, with the most robust gains observed for steering-response stage decoding. Ablation and interpretability analyses suggested that DS-MTNet jointly decoded multiple readouts and organized event-related, response-related, and state-related EEG evidence in a unified source-slot representation. These findings provide a computational step toward incorporating operator-related neural evidence into machine perception in HMC. Index Terms — Electroencephalography (EEG), Human-machine Collaboration, Structured Representation Learning, Multi-task Learning, Cognitive Sensing I. INTRODUCTION Human-machine collaboration (HMC) has expanded beyond industrial manufacturing tasks (e.g., human-robot collaborative assembly and disassembly) [1], [2] to open and ubiquitous settings such as autonomous driving [3], surgical assistance [4], and daily service robot applications [5], [6] . In these environments, safe and efficient collaboration requires robust bi-directional understanding between human operators and collaborative systems [7] . Through advanced user interfacesand physical feedback, humans can usually interpret the motion state and environmental context of machines. However, machine-side understanding of the human partner remains limited [8] . The current HMC systems do not sufficiently capture and use human-related information, particularly the perceptual and cognitive contributions of the operator to the collaborative loop. This limitation mainly arises from the machine-side perception architecture used in the current HMC systems. These systems heavily rely on cameras, LiDAR, and other environment-facing sensors [8],[9] . Such sensors work well in structured environments but remain insufficient for complex human-machine interaction. They can capture only observable physical manifestations and cannot directly measure intrinsic human states or the physiological processes underlying decision-making [7] . Visible human motion is merely the final endpoint of complex neural processes such as perception, evaluation, and action preparation. Thus, the use of behavioral readouts misses the rich

FAQs

Why do current HMC systems have limited understanding of human partners?

They primarily depend on environment-facing sensors like cameras and LiDAR, which capture only observable motion and cannot directly measure intrinsic operator states or the physiological processes behind decision-making.

What gap does DS-MTNet address in EEG decoding for HMC?

Existing EEG decoding approaches often focus on task-specific classification or aggregate prediction, leaving multiple HMC-relevant readouts insufficiently organized into a unified EEG representation.

How does DS-MTNet model multi-readout EEG decoding?

It builds a structured multi-task network with three integrated streams—EEG waveforms, task-routed source embeddings, and temporal-spectral power features—then uses dual gating to route task-specific components into reusable slots for joint decoding.

Jul 16, 2026

Upload by Riley