PHITSBench: an execution-scored benchmark for AI-assisted PHITS radiation-transport input generation using natural language

Jul 16, 2026

Upload by Violet

PHITSBench introduces an execution-scored benchmark for AI-assisted input generation for the Monte Carlo Particle and Heavy Ion Transport code System (PHITS). The benchmark contains 282 transport-scorable tasks across three workflow categories: parameter editing, syntax repair, and full simulation generation from natural-language descriptions. Evaluation uses a Composite Metric Score combining execution success and agreement with reference transport observables. Experiments with GPT-5.4 settings show strong editing and repair performance, while scratch simulation generation initially fails. A machine-readable PHITS knowledge catalog and agentic execution improve reproduce-track success to 57% and up to 66–73%, respectively.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

Introduction

PHITS overview and workflow challenges

Motivation for AI-assisted radiation-transport input generation

Related work and benchmarks

arXiv :2607 .09789v 1 [ cs .AI] 8 Jul 2026 ARTICLE PHITSBench: an execution-scored benchmark for AI-assisted PHITS radiation-transport input generation using natural language Xianglin Ji and Svetlana V. Boriskina Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA ARTICLE HISTORY Compiled July 14, 2026 ABSTRACT We introduce PHITSBench, an execution-scored benchmark for the Monte Carlo Particle and Heavy Ion Transport code System (PHITS) . PHITSBench comprises 282 transport-scorable tasks spanning three common workflow categories: parameter editing (Edit ), syntax repair (Repair ), and complete simulation generation from natural-language descriptions (Reproduce) . Each task is evaluated using a Composite Metric Score that combines execution success with agreement between generated and reference transport observables. Using PHITSBench, we evaluate five GPT-5.4-based configurations ranging from zero-shot prompting to knowledge-augmented and agentic workflows. Without domain-specific knowledge, the model performs well on editing and repair tasks (95% and 70% success, respectively) but fails to generate correct simulations from scratch (0% success on the Reproduce track) . A structured, machine-readable PHITS knowledge catalog, supplied alongside the user manual, raises single-shot Reproduce-task success to 57% . Agentic execution provides a further improvement to 66-73%, but at increased computational cost. Failure analysis shows that the remaining errors are dominated by incorrect selection and configuration of physical observables rather than syntax generation. These results suggest that future progress in AI-assisted radiation-transport modeling will depend as much on machinereadable knowledge bases, curated domain-training datasets, and execution-grounded evaluation environments as on advances in foundation models themselves. KEYWORDS PHITS; Monte Carlo radiation transport; large language models; benchmark; input file generation; agentic systems 1. Introduction Accurate modeling of radiation transport is essential for nuclear energy and fusion engineering, particle accelerator operation, medical physics, outer space exploration, and advanced manufacturing, which often require analysis and design of systems operating in radiation-rich environments. Monte Carlo (MC) radiation transport codes such as PHITS [1], MCNP [2], GEANT4 [3], and FLUKA [4] are state-of-the-art tools for predicting how neutrons, photons, electrons, and ions interact with materials in arbitrary three-dimensional (3D) geometries. These platforms encode decades of validated physics, but their simulation setup, execution, and validation of results remain challenging even for experienced designers. Constructing a valid simulation requires c Application Medical physics | | | --- | | | 150 MeV proton beam, water Shielding & facility design 662 keV beam, lead Detection & spectroscopy Cs-137 source + NaI(Tl) Figure 1 . PHITS at a glance. (a) A PHITS input file consists of a sequence of named keyword sections defining simulation parameters, particle sources, materials, geometry, and tally requests. (b) During execution, particles are emitted from the specified source, transported through the geometry, and scored according to user-defined [T-*] tally definitions. In the example shown, a neutron source emits neutrons that propagate through the geometry and interact with the material region M1 (orange), producing secondary particles at the interaction points (orange dots); the simulation returns the neutron track-length flux scored by a [T-Track] tally. (c) The same radiation-transport framework underpins diverse application areas; the panels show three representative PHITS simulations from this study—medical physics (a proton beam in water), shielding and facility design (a photon beam in lead), and radiation detection and spectroscopy (a Cs-137 source and a NaI(Tl) detec

FAQs

What is PHITSBench designed to evaluate?

PHITSBench evaluates AI systems that generate PHITS radiation-transport inputs from natural language, using execution-scored tasks and observable-level agreement with references.

How are tasks organized in PHITSBench?

Tasks are grouped into three workflow categories: parameter editing, syntax repair, and complete simulation generation from natural-language descriptions.

Why do models perform worse on generating simulations from scratch?

Failure analysis indicates remaining errors are dominated more by incorrect selection and configuration of physical observables than by syntax generation, especially on the Reproduce track.

Jul 16, 2026

Upload by Violet