Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows

Jul 16, 2026

Product data scientists often use LLM-based agents for repeatable execution tasks like cleaning data, writing SQL, choosing statistical tests, and formatting results. Reusable skill files reduce repeated prompting, yet expert curation can become a bottleneck. This work tests whether low-curation LLM-generated skills improve performance beyond task-only prompting across data preparation, extraction, statistical analysis, and reporting. Full skills and ablated variants show no reliable gains, with results stable across token-matched controls.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

Abstract

Introduction

Related Work

Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows Wei-Jung Huang Independent Researcher United States arXiv :2607 .07504v 1 [ cs .AI] 8 Jul 2026 Abstract Product data scientists often ask LLM-based agents to help with recurring execution tasks such as cleaning data, writing SQL, choosing statistical tests, and formatting results. Reusable skill files are meant to avoid prompting from scratch by packaging guidance fora task family. Expert-written skills can encode high-quality guidance, but writing and maintaining them across many data-science task families creates a manual bottleneck. We ask whether LLMgenerated skills offer a useful low-curation alternative: do they improve performance over the task prompt alone? We test this question across four lifecycle stages: data preparation, data extraction, statistical analysis, and reporting, using one generated skill per stage. We find no reliable improvement from full generated skills over No-Skill prompting. We then ask whether any part of the skill is useful by ablating different skill components. The main ablation covers 56 tasks, nine model configurations, and three providers, yielding 7,560 runs. Compared with prompting using the task alone, neither the full generated skill nor any ablated skill variant significantly improves performance; all 􀀿-values are at least 0.396, and the total spread across variants is only 1.2 pp. A supplemental token-matched control adds 1,512 runs and finds that Full skills perform similarly to task-irrelevant skill-formatted content. The results caution against using one LLM-generated skill per data-science workflow as a default single-shot prompting strategy. CCS Concepts • Computing methodologies → Artificial intelligence. Keywords large language models, data-science agents, agent skills, prompt engineering, ablation study, data-science automation 1 Introduction Product data scientists often ask LLM-based agents to help with recurring execution tasks: cleaning data, writing SQL, choosing statistical tests, computing effect sizes, and formatting reports. We refer to these systems as data-science agents, following recent benchmarks for data-science automation [5, 15] . For these systems, model choice is only part of the design problem. The other part is how to supply domain knowledge. Recent agent platforms and benchmarks use reusable skill files, such as [SKILL.md](SKILL.md), to package task instructions, examples, and reference notes [1, 8]. For data-science agents, this makes a tempting workflow: write one skill for recurring task families such as data preparation, SQL generation, statistical analysis, and reporting, then prepend it to future tasks. Expert-written skills can encode high-quality guidance, but they require practitioners to decide what to include, write examples and reference notes, and keep the content current as tools and task conventions change. This manual process scales poorly when teams need guidance for many data-science task families. LLMgenerated skills offer a lower-curation alternative: generate taskfamily guidance once and reuse it across related tasks. The question is whether this low-curation version works in practice. Existing evidence suggests that this is not guaranteed. SkillsBench [8](86 tasks, 7,308 trajectories) found that human-curated skills improve performance substantially (+16.2 pp), while LLMgenerated skills provide no aggregate benefit. However, SkillsBench does not specifically evaluate reusable skills for data-science workflows. That null result also raises a component-level question: when an LLM-generated skill fails to improve performance, is every component unhelpful, or are useful sections canceled out by harmful ones? For builders of data-science agents, it is useful to ask both whether LLM-generated skills help on data-science workflows and which failure patterns appear when they do not. We study these

FAQs

What problem does the study address about LLM-generated skills?

It examines whether reusable, LLM-generated skill files improve AI data-science agent performance compared with prompting using the task alone, while also identifying which components may help or harm.

How are the data-science workflows evaluated?

The evaluation covers four lifecycle stages—data preparation, data extraction, statistical analysis, and reporting—using one generated skill per stage in a simplified single-shot setup with deterministic verifiers.

What are the main findings from the full-skill and ablation experiments?

Neither full generated skills nor any ablated skill variant significantly improves performance over task-only prompting, and token-matched controls show similar behavior for full skills and irrelevant skill-formatted content.

Jul 16, 2026

Upload by Evangeline