Rethinking Complexity Metrics for LLM-Integrated Applications - Beyond Source Code

Sep 21, 2026

Uploaded by Aurora

LLM-integrated applications combine natural-language prompts with executable code, yet established software complexity metrics evaluate code only and leave prompt-driven logic unmeasured. The work introduces HECATE, quantifying complexity across both prompt and code layers using Prompt-as-Specification grounded in Hoare logic. From 25 taxonomy dimensions, it generates 52 candidate metrics and tests them on 118 components from 18 open-source repositories with commit-history maintenance signals. Ten metrics remain significant after controlling for code size, including seven newly proposed measures; prompt-layer metrics stay predictive even after controlling for the strongest code metric, and two top metrics generalize to held-out repositories.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

AccessFree to download

Introduction

Complexity metrics for code-only systems

Why LLM-integrated applications break code-centric assumptions

Challenges in measuring prompt complexity

Overview of the proposed approach

Rethinking Complexity Metrics for LLM-Integrated Applications: Beyond Source Code Zihao Xu∗ , Yuekang Li∗ , Gelei Deng†, Yi Liu‡, Zhenchang Xing§¶ ∗ University of New South Wales, Australia [x123201298@gmail.com](x123201298@gmail.com), [yuekang.li@unsw.edu.au](yuekang.li@unsw.edu.au) †Nanyang Technological University, Singapore [gelei.deng@ntu.edu.sg](gelei.deng@ntu.edu.sg) ‡Griffith University, Australia [yi009@e.ntu.edu.sg](yi009@e.ntu.edu.sg) § CSIRO’s Data61, Australia ¶ Australian National University, Australia [zhenchang.xing@anu.edu.au](zhenchang.xing@anu.edu.au) arXiv :2607 .0 1903v 1 [ cs .AI] 2 Jul 2026 Abstract—LLM-integrated applications interleave natural language prompts with executable code. A substantial share of their behavior is governed by the prompt layer, yet all established complexity metrics target code exclusively, leaving this logic unmeasured. We introduce HECATE, the first tool that quantifies the complexity of such applications across both the prompt layer and the code layer. HECATE rests on Prompt-as-Specification, a formalism rooted in Hoare logic that treats each prompt as a behavioral specification. Drawing on 25 complexity dimensions from published taxonomies, it produces 52 candidate metrics. We evaluate all 52 on 118 components drawn from 18 open-source repositories, using maintenance signals extracted from commit history as ground truth, and retain only those metrics that remain significant after controlling for code size. Ten of the 52 pass this filter. Seven are newly proposed metrics, each capturing distinct structural elements—such as LLM call sites, memory attributes, and prompt templates—a quality we term structural breadth as opposed to raw volume. Among the three surviving traditional baselines, RFC captures a comparable breadth, whereas Halstead N and V persist only as a weak size residual; our strongest metrics surpass all three. The prompt-layer metrics remain significant even after additionally controlling for the strongest code metric, confirming that prompt complexity constitutes an independent dimension. In a held-out evaluation on 20 components from six previously unseen repositories, the two strongest metrics retain their predictive power, providing evidence of generalizability. Index Terms—LLM-integrated applications, software complexity, NL+code metrics, prompt-as-specification, static analysis I. INTRODUCTION Large language models (LLMs) are now capable enough to follow natural-language instructions, write code, and reason through multi-step tasks. Because of this, developers increasingly embed them inside real software, where an LLM acts as the engine that plans steps, calls tools, and decides what todo next. The result is a fast-growing class of programs called LLM-integrated applications, software that interleaves naturallanguage (NL) prompts with executable code to perform tasks such as code generation, web browsing, and multi-step planning [1]–[3] . Systems such as AutoGPT, MetaGPT [4], and browser-use have together drawn hundreds of thousands of GitHub stars, and frameworks such as LangChain [5] are already used in production. As these systems move from prototypes to production, their complexity becomes a real engineering concern. Complexity is what makes software costly, since complex programs take more effort to change, contain more defects, and are harder to debug [6], [7] . To keep this cost visible, engineers rely on complexity metrics. Measures such as McCabe’s cyclomatic complexity [8], Halstead’s software science [9], and the Chidamber–Kemerer suite [10] have long acted as earlywarning signals, pointing teams to the modules that most need review or refactoring [11] . A good metric turns a vague sense of risk into a number a team can track. These metrics share one assumption. They measure code, and only code. The tools that implement them, such as Radon [12] and Lizard [13], read source and ignore everything else. That assumpti

FAQs

What problem does the paper address about complexity metrics for LLM-integrated applications?

Existing complexity metrics target code only, but LLM-integrated applications also rely heavily on prompt-layer logic that shapes behavior. As a result, prompt complexity remains unmeasured and can cause traditional metrics to miss key risk drivers.

What is HECATE and how is it constructed?

HECATE is a tool that quantifies complexity across both the prompt layer and the code layer. It is based on Prompt-as-Specification, treating each prompt as a behavioral specification rooted in Hoare logic, and it derives candidate metrics from established complexity taxonomies.

How do the evaluation results show that prompt complexity is an independent factor?

After controlling for code size, ten of 52 metrics remain significant, and seven are newly proposed. Additionally, prompt-layer metrics remain significant even when controlling for the strongest code metric, indicating prompt complexity contributes independently. Two strongest metrics also retain predictive power on held-out repositories.

Sep 21, 2026

Uploaded by Aurora