Choose Your Own Adventure - Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models

Aug 28, 2026

Story generation is an open-ended and subjective task, making evaluation of story generation models difficult. CHOOSE YOUR OWN ADVENTURE introduces a collaborative writing protocol for pairwise model evaluation: two models propose next-line suggestions, writers select between them, and the system records preferences. The protocol also supports analysis of how writers revise suggestions, and combines these signals with automatic metrics to characterize model performance under both small and large generation differences.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages10

Reading Time25 min

AccessFree to download

Abstract

1 Introduction

2 Choose Your Own Adventure

2.1 Writing Setup

Choose Your Own Adventure: Paired Suggestions in Collaborative Writing for Evaluating Story Generation Models Elizabeth Clark 1 and Noah A. Smith 1 ;2 1Paul G. Allen School of Computer Science & Engineering, University of Washington 2Allen Institute for Artiﬁcial Intelligence {eaclark7,[nasmith}@cs.washington.edu](nasmith}@cs.washington.edu) Abstract Story generation is an open-ended and subjective task, which poses a challenge for evaluating story generation models. We present CHOOSE YOUR OWN ADVENTURE, a collaborative writing setup for pairwise model evaluation. Two models generate suggestions to people as they write a short story; we ask writers to choose one of the two suggestions, and we observe which model's suggestions they prefer. The setup also allows further analysis based on the revisions people make to the suggestions. We show that these measures, combined with automatic metrics, provide an informative picture of the models' performance, both in cases where the differences in generation methods are small ([nucleus vs. top-](nucleus vs. top-k sam)[k](nucleus vs. top-k sam)[ sam](nucleus vs. top-k sam)pling) and large (GPT2 vs. Fusion models) . 1 Introduction Systems that automatically generate text suggestions to human authors have emerged as a new application of natural language generation models. Evaluating such models, however, is challenging. Typically, writers rate a single system's quality after some period of use, for example while authoring an entire story or poem (e.g., Clark et al., 2018 ; Ghazvininejad et al., 2017) . A model's quality is measured using Likert scale scores, sometimes combined with additional analysis, like the type or quantity of writer edits (e.g., Roemmele and Gordon, 2015 ; Akoury et al., 2020) . In contrast, a pairwise system evaluation—where evaluators are given two suggestions at the same time and asked to choose between them—would allow researchers to compare generation models directly. Comparative evaluations have been shown to produce more reliable and consistent results than Likert-scale ratings (Callison-Burchet al., 2007 ; Kiritchenko and Mohammad, 2017), and they have been used to evaluate natural language generation systems for translation and dialogue (Otani et al., 2016 ; Sedoc et al., 2019) . Figure 1: CYOA has a writer write a line of the story alone, and then two models generate suggestions for the next line. The writer chooses one (in this case, MODEL 1), edits it, and then adds it to the story. They repeat this process 5 times. CYOA collects writers' preferences between the two models, along with the humanauthored, machine-generated, and human-edited text, to evaluate the models. We propose CHOOSE YOUR OWN ADVENTURE (CYOA), a protocol for pairwise evaluations of collaborative writing models, focusing on story generation. Instead of scoring a single model, we compare two models. At ﬁxed points during the writing process, each generates a suggestion, and writers choose one to continue their story (see Fig. 1) . The result is utterance-level feedback on which model's generated text writers prefer at that point in the story. Along with the writer's revisions to the generated suggestions and comparisons between the generated and human-authored portions of the story, this evidence can help a researcher answer the following questions about their model: 1. Is my model better at generating story suggestions than a baseline model? 2. How useful are my model's suggestions? 3566 Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3566–3575 June 6–11, 2021 . ©2021 Association for Computational Linguistics 3. How does my model's generated text compare to human-authored text? In this paper, we show how CYOA can answer these questions and provide insights into story model behavior, both in cases when the expected differences in text quality are large (e.

FAQs

What problem does CHOOSE YOUR OWN ADVENTURE address in story generation evaluation?

It addresses the challenge of evaluating open-ended, subjective story generation quality by replacing single-model scoring with structured pairwise comparisons during writing.

How does the paired evaluation work in CHOOSE YOUR OWN ADVENTURE?

At fixed points in a writer’s story, two models each generate a suggestion, the writer selects one to continue, and the system logs which model’s suggestion is preferred.

What kinds of signals does the protocol collect to assess model performance?

It collects writers’ preferences, the text revisions writers make to the suggestions, and comparisons between generated and human-authored story portions, alongside automatic quality metrics.

Aug 28, 2026

Uploaded by Clementine