AI Overviews in Academic Search: Evaluating AI-generated Summaries of Search Results

Jul 16, 2026

Upload by Eliana

Evaluating search engine results pages (SERPs) to assess academic relevance is a demanding step in scholarly discovery. Using a formative mixed-methods design, the study tests AI-generated, SERP-level multi-document summaries as a feature in an academic search engine for social science. Summaries are manually assessed across top-five results for multiple queries with two general-purpose LLMs and yield an exploratory error taxonomy plus five safeguards. A user study (n=30) compares interfaces with vs. without AI summaries and analyzes workload, usefulness, satisfaction, confidence, and behavior.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages14

Reading Time35 min

Abstract

Introduction

Problem: Relevance triage under uncertainty in academic search

Intervention: SERP-level AI-generated overview of top results

Prior work and research gap

AI Overviews in Academic Search: Evaluating AIgenerated Summaries of Search Results in a Domain specific Search Engine Schott, Kevin GESIS – Leibniz Institute for the Social Sciences, [Germany | kevin.schott@gesis.org](Germany | kevin.schott@gesis.org) Silva, Kanishka GESIS – Leibniz Institute for the Social Sciences, [Germany | kanishka.silva@gesis.org](Germany | kanishka.silva@gesis.org) Frommholz, Ingo Modul University Vienna, [Austria | ifrommholz@acm.org](Austria | ifrommholz@acm.org) Mayr, Philipp GESIS – Leibniz Institute for the Social Sciences, [Germany | philipp.mayr@gesis.org](Germany | philipp.mayr@gesis.org) Kern, Dagmar GESIS – Leibniz Institute for the Social Sciences, [Germany | dagmar.kern@gesis.org](Germany | dagmar.kern@gesis.org) Hienert, Daniel GESIS – Leibniz Institute for the Social Sciences, [Germany | daniel.hienert@gesis.org](Germany | daniel.hienert@gesis.org) ABSTRACT Evaluating search engine results pages (SERPs) to assess result relevance is a demanding step in academic search. Ina formative mixed-methods design study, we examine AI-generated SERP-level summaries as a support feature inan academic search engine for social science information. First, we manually evaluated summaries of the top five results for 10 queries using two general-purpose models, one commercial and one open, deriving an exploratory sixcategory error taxonomy and five safeguards for scholarly deployment. We then conducted a within-subjects user study (􀝊 = 30) comparing interfaces with and without AI summaries. Confirmatory analyses showed consistent but non-significant trends favoring AI summaries for subjective workload, perceived usefulness, satisfaction, and decision-making confidence. Exploratory analyses suggested lower mental demand, with frustration also tending tobe lower. Behaviorally, participants rarely expanded the summaries and descriptively made slightly fewer result clicks and query reformulations when summaries were available. Drawing on Information Foraging Theory and participant feedback, we suggest that AI summaries may concentrate SERP-level information scent to support early triage. Overall, the findings indicate that SERP-level AI summaries are a context-and user-dependent aid rather thana universal improvement, while contributing an error taxonomy, safeguard-aware deployment guidance, and concrete design implications for scholarly search. KEYWORDS Interactive information retrieval (IIR); academic search engines; multi-document summarization; information triage INTRODUCTION Researchers today face an ever-expanding volume of academic publications and research data (Bawden & Robinson, 2009; Extance, 2018; Hołyst et al., 2024; Tenopir et al., 2003) . In academic search, this abundance creates the challenge of rapidly triaging and assessing relevance under uncertainty using limited cues on the search engine results page (SERP), such as titles, abstracts, and other metadata (Marshall & Shipman, 1997) . Prior work on academic information seeking highlights that early-stage searching is often characterized by uncertainty and evolving problem understanding, and that users can benefit from support that helps them orient and refine their focus (Kuhlthau, 1991) . In high-volume contexts, this triage burden can also contribute to feelings of overwhelm and information overload (Bawden & Robinson, 2009; Belabbes et al., 2022; Roetzel, 2019) . AI-powered tools offer the possibility to synthesize cues that can help users assess and organize result sets more efficiently (Extance, 2018; Grainger et al., 2025) . In this work, we examine a specific interface intervention: inserting a single, AI-generated abstractive summary of the five top-ranked results at the top of the SERP in an academic search engine for social science information. Our research question is whether an off-the-shelf, generalpurpose LLM (without domain-specific fine-tuning) can generate useful multi-document summaries that su

FAQs

What does the study evaluate about AI in academic search?

The study evaluates AI-generated summaries shown at the SERP level that synthesize the top-ranked results, assessing whether they help researchers triage and discover relevant sources in an academic search engine.

How were the AI summaries assessed for quality?

Summaries of the top five results for a set of queries were manually evaluated using two general-purpose models—one commercial and one open—leading to an exploratory error taxonomy and a set of safeguards for scholarly deployment.

What were the user study findings when AI summaries were added to the interface?

Across subjective measures (workload, perceived usefulness, satisfaction, and decision-making confidence), analyses showed consistent but non-significant trends in favor of AI summaries. Exploratory results suggested lower mental demand and frustration. Behaviorally, users rarely expanded summaries, with slightly fewer clicks and query reformulations when summaries were available.

Jul 16, 2026

Upload by Eliana