Large Language Models for Multi-Lingual Equivalent Mutant Detection: An Extended Empirical Study

Jul 16, 2026

Upload by Himbo

Mutation testing is a widely used software quality technique, but equivalent mutants create unnecessary cost and bias, reducing practical effectiveness. This study investigates whether large language models (LLMs) can improve equivalent mutant detection (EMD), addressing limitations of pure code-analysis approaches and data-scarce machine learning methods. Using 3,302 Java and 1,088 C mutant pairs, experiments compare strategies, evaluate cross-lingual generalization, and benchmark against state-of-the-art baselines. LLM-based models yield higher F1-scores, with fine-tuned code embeddings providing the best accuracy and inference efficiency comparable to prior ML.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages37

Reading Time1 h 33 min

Introduction

Mutation testing overview

Challenges of equivalent mutants

Motivation for LLM-based EMD

arXiv :2607 .005 1 1v 1 [ cs . SE] 1 Jul 2026 Large Language Models for Multi-Lingual Equivalent Mutant Detection: An Extended Empirical Study HONGLIN SHU∗ , Tianjin University, China and Kyushu University, Japan ZHAO TIAN∗ , College of Intelligence and Computing, Tianjin University, China DONG WANG†, College of Intelligence and Computing, Tianjin University, China JUNJI YU, College of Intelligence and Computing, Tianjin University, China JIAZHE ZHANG, College of Intelligence and Computing, Tianjin University, China XUEJIE CAO, College of Intelligence and Computing, Tianjin University, China JUNJIE CHEN, College of Intelligence and Computing, Tianjin University, China YASUTAKA KAMEI, Kyushu University, Japan Mutation testing is a powerful technique for ensuring software quality. However, the presence of equivalent mutants introduces unnecessary costs and biases, limiting its practical effectiveness. Although numerous equivalent mutant detection (EMD) methods have been proposed, they often face distinct challenges: pure-code analysis methods can be limited by their reliance on specific compiler infrastructures, while existing machinelearning approaches remain constrained by scarce training data and limited generalization to unseen mutants. Large language models (LLMs) have recently demonstrated remarkable performance across diverse coderelated tasks by better capturing program semantics. Yet their potential for EMD remains largely unexplored, particularly in the multi-lingual context. This paper presents the first comprehensive empirical study on LLMs for EMD, using 3,302 Java and 1,088 C mutant pairs to benchmark against state-of-the-art methods, explore strategy variations, assess efficiency, and evaluate cross-lingual generalization. Experimental results show that LLM-based approaches achieve higher F1-scores than the evaluated traditional methods, with fine-tuned code embedding yielding the highest detection accuracy among the tested strategies. Moreover, LLM-based approaches strike a practical balance between effectiveness and efficiency with inference times comparable to existing machine-learning models. Importantly, fine-tuned LLMs demonstrate measurable generalization across programming languages. These findings establish LLMs as a viable and efficient approach for tackling the longstanding challenge of equivalent mutant detection, offering new directions for advancing mutation testing in practice. CCS Concepts: • Software and its engineering → Software testing and debugging. ∗ These authors contributed equally to this work.†Corresponding Author Authors’ Contact Information: Honglin Shu, Tianjin University, Tianjin, China and and Kyushu University, Fukuoka, Japan, [shu.honglin.167@s.kyushu-u.ac.jp](shu.honglin.167@s.kyushu-u.ac.jp); Zhao Tian, College of Intelligence and Computing, Tianjin University, Tianjin, China, [tianzhao@tju.edu.cn](tianzhao@tju.edu.cn); Dong Wang, College of Intelligence and Computing, Tianjin University, Tianjin, China, [dong_w@tju.edu.cn](dong_w@tju.edu.cn); Junji Yu, College of Intelligence and Computing, Tianjin University, Tianjin, China, [junjiyu@tju.edu.cn](junjiyu@tju.edu.cn); Jiazhe Zhang, College of Intelligence and Computing, Tianjin University, Tianjin, China, [2839197907z@gmail.com](2839197907z@gmail.com); Xuejie Cao, College of Intelligence and Computing, Tianjin University, Tianjin, China, [caoxuejie@tju.edu.cn](caoxuejie@tju.edu.cn); Junjie Chen, College of Intelligence and Computing, Tianjin University, Tianjin, China, [junjiechen@tju.edu.cn](junjiechen@tju.edu.cn); Yasutaka Kamei, Kyushu University, Fukuoka, Japan, [kamei@ait.kyushu-u.ac.jp](kamei@ait.kyushu-u.ac.jp). Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for compon

FAQs

Why does equivalent mutant detection matter in mutation testing?

Equivalent mutants introduce unnecessary computational cost and bias, which limits mutation testing’s practical value. EMD aims to identify and handle these redundant mutants more effectively.

What data and evaluation setup does the study use for EMD?

The paper benchmarks LLM approaches using 3,302 Java mutant pairs and 1,088 C mutant pairs, comparing against state-of-the-art traditional methods.

How do LLM-based methods perform compared with traditional EMD approaches?

Experimental results show LLM-based approaches achieve higher F1-scores than evaluated traditional methods. Fine-tuned code embedding provides the highest detection accuracy among the tested strategies while maintaining inference times comparable to existing ML models.

Jul 16, 2026

Upload by Himbo