An animated picture says at least a thousand words - Selecting Gif-based Replies in Multimodal Dialog - EMNLP 2021

Aug 28, 2026

Uploaded by Jiven

Online conversations extend beyond plain text, and image-based replies such as memes and animated GIFs have become culturally recognizable and often humorous. Yet multimodal dialog systems have mainly focused on textual generation. This paper introduces a large dataset of 1.56M text–GIF conversation turns and a multimodal model, PEPE THE KING PRAWN, for selecting GIF-based replies, showing strong relevance and quality, and better community reception in a large randomized control trial with real users.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages30

Reading Time1 h 16 min

AccessFree to download

Introduction

GIF Communications

Data

Model and Experiments

An animated picture says at least a thousand words: Selecting Gif-based Replies in Multimodal Dialog Xingyao Wang University of Michigan [xingyaow@umich.edu](xingyaow@umich.edu) David Jurgens University of Michigan [jurgens@umich.edu](jurgens@umich.edu) Abstract Online conversations include more than just text. Increasingly, image-based responses such as memes and animated gifs serve as culturally recognized and often humorous responses in conversation. However, while NLP has broadened to multimodal models, conversational dialog systems have largely focused only on generating text replies. Here, we introduce a new dataset of 1.56M text-gif conversation turns and introduce a new multimodal conversational model PEPE THE KING PRAWN for selecting gif-based replies. We demonstrate that our model produces relevant and high-quality gif responses and, in a large randomized control trial of multiple models replying to real users, we show that our model replies with gifs that are signiﬁcantly better received by the community. 1 Introduction Conversations are central to many online social platforms. While most conversations are textbased, computer mediated dialog also affords alternative forms of communication, such as emoji or stickers like bitmoji, that allow users to express themselves (Tang and Hew, 2019 ; Konrad et al., 2020) . Increasingly, these visual forms of communication have become common in social media (Bourlai and Herring, 2014 ; Highﬁeld and Leaver, 2016), with a notable use of the reaction gif (Bakhshi et al., 2016 ; Miltner and Highﬁeld, 2017) . These gifs are short video sequences that depict a particular scene and sometimes contain text that acts as a meta-commentary (Eppink, 2014) . As a result, conversations become multimodal where individuals reply to one another using combinations of text and gifs (Figure 1) . While conversational AI systems have been developed in a purely textbased setting, such systems do not capture the full multimodal behavior seen online. Here, we study multimodal conversation by introducing new dialog models for selecting gif replies in conversation. PizzaMagic: Ahhhhh!!! The EMNLP deadline is in 24 hours!! x CasualModel: Figure 1: Gif responses in conversation like the one shown above are embodied dialog that use visual imagery to convey reactions and emotions. This paper develops a system to select the appropriate gif response to messages. (PDF best viewed with Adobe Acrobat) Conversation analysis is central to NLP and multiple approaches have analyzed this dialog structure (Jurafsky et al., 1998 ; Pareti and Lando, 2018 ; Cohn et al., 2019) and developed conversational agents to engage with people (e.g., Fang et al., 2018 ; Xu et al., 2020 ; Hong et al., 2020) . Recent work has focused on generating open domain social chatbots that engage in sustained conversations ina natural way (Ram et al., 2018) . Because many of these systems are designed to support voicebased dialog, they overlook non-textual forms of interaction used in social media conversations. In parallel, multimodal NLP systems have been developed for image data, often focusing on image-totext tasks such as image captioning (Melas-Kyriaziet al., 2018 ; Sharma et al., 2018) and visual question answering (Antol et al., 2015 ; Huang et al., 2019 ; Khademi, 2020) . More recent work has focused on the reverse text-to-image dimension, such as generating an image from a description (Niu et al., 2020 ; Ramesh et al., 2021) . Our work unites these two strands of research by integrating imagebased communication into conversational agents. Our paper offers three main contributions. First, we propose the new task of selecting gif responsesin multimodal conversation analysis and introduce a new dataset of 1,562,701 real-world conversation turns with gif replies. Second, we introduce a new model PEPE THE KING PRAWN that fuses image and text-based features to select a relevant gif response. In in-house experimen

FAQs

What problem does the paper address in multimodal conversation systems?

It addresses the gap where conversational AI systems largely generate only text replies, despite the widespread use of animated GIFs in real dialogs.

What new dataset and task does the paper introduce?

It introduces a new task of selecting GIF responses and a dataset containing 1,562,701 real-world conversation turns with GIF replies.

How is the PEPE THE KING PRAWN model used and evaluated?

The model fuses image and text features to select relevant GIF replies; evaluations include in-house experiments for ranking quality and a randomized controlled trial where community votes favor its GIFs.

Aug 28, 2026

Uploaded by Jiven