EyeXNet - Enhancing Abnormality Detection and Diagnosis via Eye-Tracking and X-ray Fusion - Abstract

Aug 5, 2026

Upload by Aurora

Integrating eye gaze data with chest X-ray images in deep learning has produced conflicting results in prior studies. EyeXNet addresses this by arguing that researchers often ignore the human element of eye tracking and apply raw signals without proper preprocessing. The work introduces EyeXNet, a multimodal model combining CXR images with radiologists’ fixation masks to predict abnormality locations. Fixation maps are analyzed around reporting moments, showing more targeted focus during reporting. Across eight experiments, fixation-mask integration improves recall and precision versus baseline, supporting human-centered multimodal learning for CXR analysis.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages17

Reading Time43 min

AccessFree to download

Introduction

Chest X-ray diagnosis and deep learning

Eye-tracking datasets and multimodal architectures

Conflicting evidence in the literature

Study motivation and objective

machine learning & knowledge extraction Article EyeXNet: Enhancing Abnormality Detection and Diagnosis via Eye-Tracking and X-ray Fusion Chihcheng Hsieh 1,†, André Luís 2,3,†, José Neves 2,3, Isabel Blanco Nobre 4, Sandra Costa Sousa 4, Chun Ouyang 1, Joaquim Jorge 2,3 and Catarina Moreira 1,2,5, * Citation: Hsieh, C.; Luís, A.; Neves, J.; Nobre, I.B.; Sousa, S.C.; Ouyang, C.; Jorge, J.; Moreira, C. EyeXNet: Enhancing Abnormality Detection and Diagnosis via Eye-Tracking and X-ray Fusion. Mach. Learn. Knowl. Extr. 2024, 6, 1055–1071. [https://](https://)[ ](https://)[doi.org/10.3390/make6020048](doi.org/10.3390/make6020048) Academic Editor: Andreas Holzinger Received: 24 March 2024 Revised: 19 April 2024 Accepted: 6 May 2024 Published: 9 May 2024 Copyright: © 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license ([https://](https://)[ ](https://)[creativecommons.org/licenses/by/](creativecommons.org/licenses/by/)[ ](creativecommons.org/licenses/by/)[4.0/](4.0/)) . 1 School of Information Systems, Queensland University of Technology, Brisbane, QLD 4000, Australia; [chihcheng.hsieh@hdr.qut.edu.au](chihcheng.hsieh@hdr.qut.edu.au) (C.H.); [c.ouyang@qut.edu.au](c.ouyang@qut.edu.au) (C.O.) 2 Instituto Superior Técnico, Universidade de Lisboa, 1049-001 Lisboa, Portugal; [andre.t.luis@tecnico.ulisboa.pt](andre.t.luis@tecnico.ulisboa.pt) (A.L.); [jose.s.neves@tecnico.ulisboa.pt](jose.s.neves@tecnico.ulisboa.pt) (J.N.); [jorgej@tecnico.ulisboa.pt](jorgej@tecnico.ulisboa.pt) (J.J.) 3 INESC-ID, 1000-029 Lisbon, Portugal 4 Grupo Lusíadas, Imagiology Department, 1500-458 Lisbon, Portugal; [isabel.blanco.nobre@lusiadas.pt](isabel.blanco.nobre@lusiadas.pt) (I.B.N.); [sandra.costa.sousa@lusiadas.pt](sandra.costa.sousa@lusiadas.pt) (S.C.S.) 5 Human Technology Institute, University of Technology Sydney, Sydney, NSW 2007, Australia * Correspondence: [catarina.pintomoreira@uts.edu.au](catarina.pintomoreira@uts.edu.au)[ ](catarina.pintomoreira@uts.edu.au)† These authors contributed equally to this work. Abstract: Integrating eye gaze data with chest X-ray images in deep learning (DL) has led to contradictory conclusions in the literature. Some authors assert that eye gaze data can enhance prediction accuracy, while others consider eye tracking irrelevant for predictive tasks. We argue that this disagreement lies in how researchers process eye-tracking data as most remain agnostic to the human component and apply the data directly to DL models without proper preprocessing. We present EyeXNet, a multimodal DL architecture that combines images and radiologists’ fixation masks to predict abnormality locations in chest X-rays. We focus on fixation maps during reporting moments as radiologists are more likely to focus on regions with abnormalities and provide more targeted regions to the predictive models. Our analysis compares radiologist fixations in both silent and reporting moments, revealing that more targeted and focused fixations occur during reporting. Our results show that integrating the fixation masks in a multimodal DL architecture outperformed the baseline model in five out of eight experiments regarding average Recall and six out of eight regarding average Precision. Incorporating fixation masks representing radiologists’ classification patterns ina multimodal DL architecture benefits lesion detection in chest X-ray (CXR) images, particularly when there is a strong correlation between fixation masks and generated proposal regions. This highlights the potential of leveraging fixation masks to enhance multimodal DL architectures for CXR image analysis. This work represents a first step towards human-centered DL, moving away from traditional data-driven and human-agnostic approaches. Keywords: multimodal deep learning; eye tracking; object detection; X-rays; fixation maps 1

FAQs

Why do studies integrating eye gaze data with chest X-rays reach conflicting conclusions?

The disagreement stems from how eye-tracking data are processed: many works apply the data directly to models while remaining agnostic to the human component and using insufficient preprocessing.

What is EyeXNet and how does it use eye-tracking information?

EyeXNet is a multimodal deep learning architecture that combines chest X-ray images with radiologists’ fixation masks to predict abnormality locations, emphasizing fixation maps during reporting moments.

How do fixation masks affect model performance compared with a baseline?

In eight experiments, incorporating fixation masks outperformed the baseline in five experiments for average Recall and in six for average Precision, particularly when fixation masks correlate with proposal regions.

Aug 5, 2026

Upload by Aurora