AutoMatBench An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking

Jul 16, 2026

Material property prediction (MPP) infers key physical, chemical, or functional properties from chemical composition and structure, speeding discovery and optimization of novel materials. MatBench is a widely used benchmark defining more than ten representative MPP problems and evaluating in-distribution (ID) performance, but it fails to capture out-of-distribution (OOD) behavior. AutoMatBench combines MatBench benchmarking pipelines with OOD evaluation to cover extensive configuration spaces, revealing large performance discrepancies and reducing cost via Bayesian optimization within twelve optimization steps.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

Introduction

Material property prediction and MatBench

Limitations of in-distribution benchmarking

Goal: unified ID/OOD evaluation methodology

Tasks, models, and configuration factors

AutoMatBench: An Automatic Optimization Toolkit for the Acceleration of Material Properties Prediction Benchmarking Hongxiao Lia,b , Wanling Gaoa,b a Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China b University of Chinese Academy of Sciences, Beijing, China arXiv :2607 . 1 1526v 1 [ cs .LG] 13 Jul 2026 ARTICLE INFO Keywords: Artificial intelligence for science Artificial intelligence for materials Material properties prediction Benchmark Evaluation Automatic optimization AB STRACT Material property prediction (MPP) infers key properties from chemical composition and structure, accelerating the discovery and optimization of novel materials. In the realm of MPP, MatBench is a widely accepted benchmarking tool that defines over ten significant problems and provides the paradigm of performance evaluation for AI prediction models. Even though MatBench works wellin benchmarking the performances of prediction models on in-distribution (ID) tasks and datasets, it lacks the ability to reflect their performances on out-of-distribution (OOD) material data, resulting failure in new material discovery. By combining the pipelines of MatBench and the existing researches on OOD performance evaluation, this study enables a huge space of benchmarking configurations, comprehensively reflecting the performances, abilities, and disadvantages of various AI prediction models. This work reports that the discrepancy of performances at different configuration values is huge and can be illustrated with prior knowledge and novel insights, therefore consideration of causal effect of configurations on performance results is necessary. In case of the impossibility of enumerative benchmarking at every configuration, this work further proposes AutoMatBench, an automatic toolkit with Bayesian optimization. Experiments with AutoMatBench reports that, within twelve steps of optimization, the similar results with MatBench and former OOD research can be accessed while more than half of the cost are saved. Besides, this tool also yields more essential findings on MPP benchmarking, positively contributing to the cost and efficiency of new material discovery. 1. Introduction Material property prediction (MPP), as a signficant subfield of Artificial Intelligence for Science (AI4S), aims at predicting or inferring the physical, chemical, or functional properties of materials directly from their composition and structure data. By learning relationships of compositionalor structural information from experimental and computational data, MPP models enable high-efficiency screening of demanded materials, guide target-oriented synthesis, and reduces reliance on high-cost experiments or ab initio simulations. MatBench [1] emerges as a widely accepted benchmarking suite that defines over ten diverse representative prediction tasks for both classification and regression problems, curated from solid datasets. MatBench provides a consistent training-test splitting and cross-fold validation, establishing a generalized and comparable paradigm for evaluating the prediction accuracies of AI models across tasks and datasets. It has become the most adopted de facto reference for developing and validating new material informatics methods. Unfortunately, benchmarking results with MatBench only represent the performances on in-distribution (ID) tasks and datasets, while the performances on out-of-distribution (OOD) data are totally different. Research from Omee et al. [2] and Fung et al. [3] also support such conclusion. Experiments of this work also report similar outcomes. This study aims at establishing an evaluation methodology the abilities of AI prediction models that works both on ID and OOD data, referring the pipelines of MatBench [1] and the existing OOD performance evaluation. There are two main challenges. On the one hand, the existing OOD performance evaluation [4, 5, 6, 2] are neither completely undergone under t

FAQs

What limitation of MatBench motivates this work?

MatBench benchmarks mainly in-distribution (ID) tasks and datasets, while out-of-distribution (OOD) material data yields different performance, so it cannot reflect real generalization for new material discovery.

How does AutoMatBench improve benchmarking beyond MatBench?

AutoMatBench integrates MatBench’s pipeline with existing OOD performance evaluation ideas, enabling a much larger space of benchmarking configurations and more comprehensive reflection of model strengths and weaknesses.

Why is configuration sensitivity important in MPP benchmarking?

Performance for the same model and task can vary greatly across configuration values, so relying on a single or a few configurations can introduce bias and misrepresent the model’s true ability.

Jul 16, 2026

Upload by Theodora