BioSecBench-Refusal: Calibrated Refusal and Safe Usefulness in Dual-Use Biology Settings

Jul 16, 2026

Upload by Noah

BioSecBench-Refusal introduces a benchmark for measuring risk identification and refusal behavior in agentic biosecurity evaluation. The benchmark pairs 61 Routine tasks, derived from published life-science research, with 46 Red-Team tasks that use realistic fictional scenarios to conceal a biosecurity hazard. Across 16 model-harness configurations, refusal rates span 7%–74% for Routine tasks and 1%–62% for Red-Team tasks. Refusals frequently stem from provider API filters, while reasoning-enabled models can better surface real threats, supporting safer calibration for agentic biotech R&D.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages9

Reading Time23 min

Introduction

Refusal in LLM and provider safeguards

Need for calibrated refusal in agentic workflows

Biosecurity evaluation benchmarks and trade-offs

BioSecBench-Refusal benchmark design

Red-Team tasks with concealed hazards

Routine tasks adapted from published research

Results summary

Refusal rates across configurations

Provider filters vs task-level risk analysis

BioSecBench-Refusal A paired metric for performance and alignment in agentic biosecurity risk assessment Edwin H. Wintermute1, Harmon Bhasin2, Christina M. Agapakis1, Dianzhuo Wang2, Evan Seeyave2, Arjun Banerjee2, Daniel Fulop, Matthew C. Watson, Adam J. Meyer, Sandrine Boissel, Jens H. Kuhn, Rishi Jain, Noah D. Taylor, Helena Shomar, Patrick M. Boyle1,*, Kenny Workman2,* 1American Wetware, Boston, MA 2LatchBio, San Francisco, CA [Correspondence: patrick@americanwetware.com](Correspondence: patrick@americanwetware.com), [kenny@latch.bio](kenny@latch.bio) 05462v1 [ cs .CR] 6 Jul 2026 ABSTRACT As AI agents are incorporated into life science workflows, the capabilities that speed discovery might also enable misuse. We present BioSecBench-Refusal, a benchmark for risk identification and refusal behavior for biological research tasks. The benchmark pairs 61 Routine tasks, legitimate analyses adapted from the published literature, with 46 Red-Team tasks, fictional scenarios that resemble real research but conceal a biosecurity hazard. Across 16 model-harness configurations, refusal rates ranged from 7% to 74% on Routine tasks and 1% to 62% on Red-Team tasks, with many configurations refusing legitimate Routine work at comparable or higher rates than concealed hazards. Refusals were most often triggered by provider API filters applied prior to agentic reasoning. However, models given room to reason showed the potential to identify more real threats. We release BioSecBench-Refusal as a tool for model developers to calibrate capability and caution for agentic biotech R&D. arXiv :2607 . Introduction AI agents are increasingly able to design proteins, plan experiments, and interpret complex data. These capabilities present an acute dual-use problem: they can accelerate beneficial discoveries or, if misused, enable harm [1, 2, 3, 4] . Assessing biosecurity risks requires both technical and normative analysis. To be deployed safely, agentic tools must identify complex biological risksand act upon them in alignment with the values of human communities and institutions[5, 6] . Refusal training is the primary mechanism by which model developers mitigate biosecurity risks in large language models (LLMs) . Refusal operates at two layers: the model itself, trained to decline harmful requests [7, 8], and a provider-side safeguard that screens prompts and outputs before or after the model runs [9, 10] . Recent work has sought to characterize refusal behavior against a range of toxic signals [11] and specifically against biologically relevant signals [12] . This work is motivated, in part, by concerns that certain refusal behaviors ("over-refusals") maybe too stringent, disrupting legitimate work and reducing trust in AI tools for productive biological research. Agentic workflows deepen the challenge of calibrating refusal behavior: a refusal at any step of a multi-step reasoning chain can cause the entire task to fail. A model well-calibrated on simple tasks may over-refuse on complex ones. Refusal therefore needs to be measured over the extended reasoning chains where agents actually operate. Benchmarks provide a standardized way to measure LLM performance. Recent benchmarks have been developed to evaluate biosecurity capabilities including in virology [13], synthesis screening [14], or biological design tools [15], among other areas of concern[16, 17] . Unlike strictly technical tasks, a biosecurity refusal evaluation has no single correct answer. Preferred model behavior might range from open access, with minimal safeguards, to complete refusal of biology-related tasks. The choice will depend on a cost-benefit analysis taken in the context of prevailing laws, institutional values and other security measures in place to control access to a particular LLM. BioSecBench-Refusal is a biosecurity-specific benchmark in- tended to help model builders and policymakers navigate this trade-off in refusal behavior. It is comp

FAQs

What does BioSecBench-Refusal benchmark measure?

It measures risk identification ability and refusal behavior for biological research tasks, with emphasis on how models decline requests in dual-use settings.

How are Routine and Red-Team tasks defined in the benchmark?

Routine tasks are adapted from published life-science analyses that a biologist might perform, while Red-Team tasks are fictional scenarios that resemble real research but conceal a biosecurity hazard.

What were the observed refusal-rate ranges across model configurations?

Across 16 model-harness configurations, refusal rates ranged from 7% to 74% on Routine tasks and 1% to 62% on Red-Team tasks.

Jul 16, 2026

Upload by Noah