PIAST - Rapid Prompting with In-context Augmentation for Scarce Training data

Sep 4, 2026

Uploaded by Maeve

LLMs are highly sensitive to prompt design, yet effective prompting often demands careful few-shot construction. PIAST introduces a fast automatic prompt construction method that augments human instructions by generating a compact set of few-shot examples. The algorithm iteratively replaces, drops, or keeps examples using Monte Carlo Shapley estimation of each example’s utility, while aggressive subsampling and a replay buffer accelerate evaluations. Under tight compute budgets it improves simplification and mathematical reasoning while still achieving strong classification and summarization results, and with larger budgets reaches new state-of-the-art performance.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

AccessFree to download

1 Introduction

2 Related Work

PIAST: Rapid Prompting with In-context Augmentation for Scarce Training data Paweł Batorski Paul Swoboda Heinrich Heine Universität Düsseldorf {pawel.batorski,[paul.swoboda}@hhu.de](paul.swoboda}@hhu.de) arXiv :2512 . 1 10 13v2 [ cs .CL] 7 Apr 2026 Abstract LLMs are highly sensitive to prompt design, but handcrafting effective prompts is difficult and often requires intricate crafting of few-shot examples. We propose a fast automatic prompt construction algorithm that augments human instructions by generating a small set of few shot examples. Our method iteratively replaces/ -drops/keeps few-shot examples using Monte Carlo Shapley estimation of example utility. For faster execution, we use aggressive subsampling and a replay buffer for faster evaluations. Our method can be run using different compute time budgets. Under a limited budget, it outperforms prior automatic prompting methods on text simplification and mathematical reasoning (GSM8K, DeepMath, Math500), while achieving second-best results on classification and summarization and third-best on MedQA. With an extended, yet still modest budget, PIAST sets a new state of the art among automatic prompting methods on classification, simplification, GSM8K, DeepMath, and Math500 . Overall, our results suggest that optimizing in-context examples, rather than exhaustively searching over instruction rewrites is the dominant lever for fast and data-efficient prompt engineering. We will release code and data upon acceptance. 1 Introduction Automatic prompt engineering has emerged as a practical way to adapt LLMs without gradient updates. However, many existing methods are impractical in time and data constrained settings: (i) some require hours of compute to explore a large prompt search space, and (ii) they rely on sizeable training sets to reliably score candidates. Recent work on prompt generation for previously unseen tasks (Batorski and Swoboda, 2025) alleviates the per-task tuning burden, but still lags behind methods that are optimized separately for each task. Moreover, most prior work optimizes only the instruction string, for example via rephrasing (using e.g. evolutionary algorithms or extensive search), ignoring the most impactful component of in-context learning (ICL): few-shot examples. The main exceptions are (Batorski et al., 2025), which synthesizes examples, but its computational cost is dozens of hours, and (Pryzant et al., 2023), which only selects examples from the training set. Our method is, to our knowledge, the first method that is fast, synthesizes new few-shot examples not found in the training set and requires relatively less access to training examples. When run long enough, our method additionally obtains new state of the art results among automatic prompting methods fora number of tasks. Our method works as follows: We first synthesize a proposal set of in-context examples that we append to our initial prompt. Then, in our optimization loop, we evaluate its efficacy on a small randomized evaluation set and identify the least helpful examples using a Monte Carlo Shapley estimator and replace, drop or keep it. If replaced, we draw from a pool of newly proposed few-shot examples. For efficiency and stability, we use a replay buffer for the evaluation set. Our algorithm has a favorable anytime performance: When run with a small computational budget, we attain second best results among our baselines on classification and summarization and already exceeds previous SoTA on simplification and mathematical reasoning tasks. When run with an extended budget that is still comparable to some other baselines, we exceed previous methods additionally on classification. Interestingly, even when running without the iterative update loop and only using the first generated few-shot examples, we often still get competitive results. To summarize, our contributions are as follows: Conceptual: We propose PIAST, an automatic prompt construction m

FAQs

What problem does PIAST address in prompt engineering?

PIAST targets the difficulty of handcrafting effective prompts, especially the challenge of creating useful few-shot examples when compute and data are limited.

How does PIAST optimize the few-shot examples?

It iteratively replaces, drops, or keeps examples using Monte Carlo Shapley estimation to measure each example’s utility in context.

What efficiency techniques does PIAST use for faster evaluation?

It uses aggressive subsampling and a replay buffer to speed up evaluations while keeping the optimization loop stable.

Sep 4, 2026

Uploaded by Maeve

PIAST - Rapid Prompting with In-context Augmentation for Scarce Training data_Page 1

Open in WPS to view all 20 pages