HarDBench: Draft-Based Co-Authoring Jailbreak Attacks for Safe Human–LLM Collaborative Writing

Jul 16, 2026

Large language models increasingly act as co-authors in collaborative writing, where users start from rough drafts and ask the model to complete, revise, and refine text. This process can be abused: malicious users provide incomplete harmful drafts and jailbreak the model into producing dangerous completions. HarDBench is introduced as a systematic benchmark to evaluate robustness across high-risk domains, with realistic prompt structures and domain cues. A safety-utility balanced preference-optimization alignment reduces harmful outputs while preserving helpful co-authoring performance, establishing a new evaluation paradigm.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages47

Reading Time1 h 58 min

Introduction

HarDBench Overview

Threat Model: Draft-Based Co-Authoring Jailbreaks

Dataset Construction and Benchmark Design

Safety–Utility Balanced Alignment

Experimental Findings

HarDBench: A Benchmark for Draft-Based Co-Authoring Jailbreak Attacks for Safe Human–LLM Collaborative Writing Euntae Kim1 , Soomin Han2 , Buru Chang1 * 1 Korea University, 2 Sogang University {untae0122,[buru_chang}@korea.ac.kr](buru_chang}@korea.ac.kr), [soominsion@u.sogang.ac.kr](soominsion@u.sogang.ac.kr) Warning. This paper includes references to hazardous procedures, such as cyberattacks and explosives, solely to analyze and mitigate LLM vulnerabilities for research purposes. arXiv :2604 . 19274v 3 [ cs .CL] 13 Jul 2026 Abstract Large language models (LLMs) are increasingly used as co-authors in collaborative writing, where users begin with rough drafts and rely on LLMs to complete, revise, and refine their content. However, this capability poses a serious safety risk: malicious users could jailbreak the models—filling incomplete drafts with dangerous content—to force them into generating harmful outputs. In this paper, we identify the vulnerability of current LLMs to such draft-based co-authoring jailbreak attacks and introduce HarDBench, a systematic benchmark designed to evaluate the robustness of LLMs against this emerging threat. HarDBench spans a range of high-risk domains—including Explosives, Drugs, Weapons, and Cyberattacks—and features prompts with realistic structure and domain-specific cues to assess the model susceptibility to harmful completions. To mitigate this risk, we introduce a safety-utility balanced alignment approach based on preference optimization, training models to refuse harmful completions while remaining helpful on benign drafts. Experimental results show that existing LLMs are highly vulnerable in co-authoring contexts and our alignment method significantly reduces harmful outputs without degrading performance on co-authoring capabilities. This presents a new paradigm for evaluating and aligning LLMs in human-LLM collaborative writing settings. Our new benchmark and dataset are available on our project page at [https://github.com/](https://github.com/)[ ](https://github.com/)untae0122/HarDBench 1 Introduction Large language models (LLMs) have demonstrated the ability to generate responses grounded in knowledge acquired from large-scale text corpora. As a result, many users now incorporate LLMs as coauthors in their writing processes, drawing upon * Corresponding author. the models’ knowledge to complete and refine their writing (Lee et al., 2022 ; Noy and Zhang, 2023) . In particular, users often begin with a rough draft and employ LLMs to fill in missing knowledge, address argumentative gaps, and polish the text, thereby maximizing the model’s utility. Recent research on human preference optimization has improved LLMs’ co-authoring capabilities by utilizing preference data that capture aspects such as helpfulness, clarity, and writing quality (Ouyang et al., 2022 ; Ethayarajh et al., 2024) . However, such draft-based co-authoring processes entail potential misuse. As shown in Figure 1, a malicious user can input an incomplete yet harmful draft (e.g., a partially written drug synthesis procedure) and prompt the model to polish it. Even with safety mechanisms in place, the LLM generates more harmful outputs from its internal knowledge, including detailed and executable instructions that could cause real-world harm. This exposes a significant risk: the model’s co-authoring capabilities can be exploited to surface harmful knowledge that would otherwise be restricted by system-level safeguards. Despite such risks, this issue remains largely unexplored in current research. In this paper, we propose HarDBench (Harmful Draft Benchmark), a benchmark grounded in the co-authoring process to systematically evaluate the vulnerabilities of LLMs. We begin by manually collecting representative domain-specific keywords (e.g., PETN, fentanyl, M16, Whonix) spanning four high-risk domains: Explosives, Drugs, Weapons, and Cyberattacks. Using these keywords, we prompt the drafter m

FAQs

What vulnerability does HarDBench focus on in human–LLM collaborative writing?

HarDBench targets draft-based co-authoring jailbreaks, where a malicious user supplies an incomplete harmful draft and prompts the model to polish it, leading to harmful and detailed completions from internal knowledge.

Which high-risk domains are covered by HarDBench?

HarDBench spans four high-risk domains: Explosives, Drugs, Weapons, and Cyberattacks, using representative domain-specific keywords and prompts that simulate realistic user behavior.

How does the proposed alignment method reduce harmful outputs without harming co-authoring ability?

The approach uses preference optimization to balance safety and utility: harmful completions are treated as rejected while benign cooperative completions are preferred, training the model to refuse harmful drafts yet remain helpful on benign ones.

Jul 16, 2026

Upload by Ava Thompson