Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation

Jul 16, 2026

Upload by Himbo

Large language models are used to generate synthetic data, especially when replicating private text. Producing such replicas requires careful balancing of privacy and utility. This paper proposes Realistic and Privacy-Preserving Synthetic Data Generation (RPSG), which uses private seeds and adds privacy-preserving strategies, including a formal differential privacy mechanism during candidate selection. Experiments against state-of-the-art private synthetic data methods show high fidelity to private data with strong privacy protection.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages22

Reading Time55 min

Introduction

Synthetic replicas of private text

Limitations of existing privacy-preserving methods

The RPSG method pipeline

Experimental evaluation and contributions

Private Seeds, Public LLMs: Realistic and Privacy-Preserving Synthetic Data Generation Qian Ma, Sarah Rajtmajer Information Sciences and Technology, The Pennsylvania State University {qfm5033, [smr48}@psu.edu](smr48}@psu.edu) arXiv :2604 .07486v 3 [ cs .CR] 13 Jul 2026 Abstract Large language models (LLMs) have emerged as a powerful tool for synthetic data generation. A particularly important use case is producing synthetic replicas of private text, which requires carefully balancing privacy and utility. We propose Realistic and Privacy-Preserving Synthetic Data Generation (RPSG), which uses private seeds and integrates privacy-preserving strategies, including a formal differential privacy (DP) mechanism in the candidate selection, to generate realistic synthetic data. Comprehensive experiments against state-of-theart private synthetic data generation methods demonstrate that RPSG achieves high fidelity to private data while providing strong privacy protection. 1 1 Introduction Synthetic data generation is an active area of work in natural language processing (NLP) (Bommasaniet al., 2019 ; Yu et al., 2022) with applications ranging from clinical text analysis (Walonoski et al., 2018 ; Tang et al., 2023) to social media synthesis (Cao et al., 2023 ; Lu et al., 2023) . One type of synthetic data of significant practical interest is synthetic replicas of private text (Hou et al., 2024 ; Yu et al., 2023) . Oftentimes, text data that would be of benefit, e.g., to researchers, policymakers, or technologists, cannot be shared due to privacy considerations. Text collected from social media platforms, for example, often contains users’ voluntarily disclosed personal information–so-called self-disclosures (see (Ashuri and Halperin, 2024) for a recent interdisciplinary review) . Such text can be used for user targeting and manipulation; if the personal information shared is more sensitive, e.g., personally identifiable information (PII), the risks can be graver (Gruzd and Hernández-García, 2018) . 1All code is available at [https://github.com/masonmq/](https://github.com/masonmq/)[ ](https://github.com/masonmq/)rpsg Mainstream methods for generating privacypreserving synthetic data include fine-tuning LLMs (e.g., DistilGPT2 (Wolf et al., 2020)) with differential privacy (DP) mechanisms applied through extensive modifications of gradient descent during training (Hou et al., 2024), as well as using prompt engineering (e.g., GPT-4 (OpenAI, 2023)) to guide models toward producing semantically similar synthetic data (Yukhymenko et al., 2024) . However, fine-tuning approaches without rigorous privacy safeguards have been shown to suffer from memorization and leakage vulnerabilities (Mireshghallah et al., 2022 ; Li et al., 2024a), and are increasingly impractical as many modern LLMs are accessible only via APIs. In contrast, prompt-based methods are model-access agnostic and can be applied to both API-based and open-source models. However, they struggle to generate high-quality synthetic data that preserves the richness and utility of the original private data (Xie et al., 2024) . In response to these limitations, we propose a novel method for realistic and privacy-preserving synthetic data generation (RPSG) (Figure 1) . RPSG is designed using private data as seeds to generate high-quality synthetic data that closely resembles the original while reducing privacy risk. In Phase 1, an abstraction model produces multiple sentiment-aligned abstracted candidates for each private seed, reducing identifiable patterns and semantic structures; a formal DP mechanism is then applied to select candidates. In Phase 2, an LLM generates variations of these DP-selected candidates. In Phase 3, synthetic variants are refined to reduce memorization risks and redact any PII. The resulting samples constitute a final set of realistic and privacy-preserving synthetic data, suitable for downstream applications. We conduct comprehensive expe

FAQs

What problem does RPSG address in synthetic data generation?

RPSG targets generating realistic synthetic replicas of private text while balancing privacy risk and data utility for downstream use.

How does RPSG incorporate privacy protection?

RPSG applies a formal differential privacy mechanism during candidate selection in Phase 1, and further refines outputs to reduce memorization risks and redact any PII.

How is RPSG evaluated and what does it achieve compared with baselines?

Experiments use a PubMed benchmark dataset and an original Reddit dataset, comparing against gradient-based and prompt-based DP baselines plus a one-to-one rewriting baseline. Results report improved fidelity, diversity, lexical quality, and stronger resistance to membership inference attacks.

Jul 16, 2026

Upload by Himbo