A comparative study of statistical methods for identifying differentially expressed genes in spatial transcriptomics - Independent GEE vs Wilcoxon rank-sum test

Sep 22, 2026

Spatial transcriptomics (ST) enables genome-wide gene expression profiling while preserving spatial context, which is crucial for interpreting complex tissue architectures such as those seen in cancers. Seurat’s default differential expression analysis uses the Wilcoxon rank-sum test, which ignores spatial correlations and can inflate false positive rates. A Generalized Estimating Equations (GEE) framework is proposed to incorporate spatial dependence. Simulations comparing GEE-based tests with Wilcoxon and z-test show that the robust standard error “Independent GEE” achieves superior Type I error control with comparable power, and real data applications highlight poor p-value calibration and potential false positives under Wilcoxon. The method is implemented in R package SpatialGEE.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages19

Reading Time48 min

AccessFree to download

Abstract

Methods: GEE-based framework and comparisons

Simulation results: Type I error control and power

Real-data applications and p-value calibration

Implementation: R package SpatialGEE

OPEN ACCESS Citation: Wang Y, Zang C, Li Z, Guo CC, Lai D, Wei P (2026) A comparative study of statistical methods for identifying differentially expressed genes in spatial transcriptomics. PLoS Comput Biol 22(2): e1013956. [https://doi.org/10.1371/](https://doi.org/10.1371/)[ ](https://doi.org/10.1371/)[journal.pcbi.1013956](journal.pcbi.1013956) Editor: Shaun Mahony, Penn State University: The Pennsylvania State University, UNITED STATES OF AMERICA Received: February 17, 2025 Accepted: January 27, 2026 Published: February 11, 2026 Copyright: © 2026 Wang et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data availability statement: We have implemented our proposed methods in R package ‘SpatialGEE’ available at [https://](https://)[ ](https://)[github.com/yishan03/SpatialGEE](github.com/yishan03/SpatialGEE. The)[. The](github.com/yishan03/SpatialGEE. The) spatial transcriptomics datasets of breast cancer and prostate cancer analyzed here are available on METHODS A comparative study of statistical methods for identifying differentially expressed genes in spatial transcriptomics Yishan Wang1,2, Chenxuan Zang1, Ziyi Li1, Charles C. Guo3, Dejian Lai2, Peng Wei1* 1 Department of Biostatistics, The University of Texas MD Anderson Cancer Center, Houston, Texas, United States of America, 2 Department of Biostatistics and Data Science, The University of Texas Health Science Center at Houston (UTHealth), Houston, Texas, United States of America, 3 Department of Pathology, The University of Texas MD Anderson Cancer Center, Houston, Texas, United States of America * [pwei2@mdanderson.org](pwei2@mdanderson.org) Abstract Spatial transcriptomics (ST) provides unprecedented insights into gene expression patterns while retaining spatial context, making it a valuable tool for understanding complex tissue architectures, such as those found in cancers. Seurat, by far the most popular tool for analyzing ST data, uses the Wilcoxon rank-sum test by default for differential expression analysis. However, as a nonparametric method that disregards spatial correlations, the Wilcoxon test can lead to inflated false positive rates and misleading findings. This limitation highlights the need for a more robust statistical approach that effectively incorporates spatial correlations. To this end, we propose a Generalized Estimating Equations (GEE) framework as a robust solution for differential gene expression analysis in ST. We conducted a comprehensive comparison of the GEE-based tests with existing methods, including the Wilcoxon rank-sum test and z-test. By appropriately accounting for spatial correlations, extensive simulations showed that the GEE test with robust standard error, referred to as the Independent GEE, demonstrated superior Type I error control and comparable power relative to other methods. Applications to ST datasets from breast and prostate cancer showed poor calibration of the p-values and potential false positive findings from the Wilcoxon rank-sum test. Our comparative study based on simulations and real data applications suggests that the Independent GEE test is well-suited for ST data, offering more accurate identification of biologically relevant gene expression changes and complementing the Wilcoxon rank-sum test. We have implemented the proposed method in R package “SpatialGEE”, available on GitHub. PLOS Computational Biology | [https://doi.org/10.1371/journal.pcbi.1013956](https://doi.org/10.1371/journal.pcbi.1013956) February 11, 2026 1 / 19 10×Genomics websites: [https://www.10xge](https://www.10xge)[nomics.com/datasets/human-breast-cancer](nomics.com/datasets/human-breast-cancer)block-a-section-1-1-standard-1-1-0 and [https://](https://)[ ](https://)[www.10xgenomics.com/datasets/human-pros](www.10xgenomics.

FAQs

Why can the Wilcoxon rank-sum test be problematic for spatial transcriptomics differential expression analysis?

It ignores spatial correlations in ST data, which can inflate false positive rates and produce misleading findings.

What statistical approach is proposed to address spatial correlations in ST?

The study proposes a Generalized Estimating Equations (GEE) framework for differential gene expression analysis, including a robust standard error version called Independent GEE.

How does the Independent GEE test perform compared with existing methods?

Extensive simulations show better Type I error control and comparable power relative to Wilcoxon rank-sum and z-test, and real-data applications show improved p-value calibration versus Wilcoxon.

Sep 22, 2026

Upload by Theodore