Think Big, Search Small: Where Capacity Matters in Hierarchical Search Agents

Jul 16, 2026

Large language model based search agents increasingly use multi-agent architectures where a main agent decomposes a question into sub-queries and delegates them to parallel sub-agents. Existing systems typically instantiate all roles from identical-scale models, leaving uncertainty about how capacity should be allocated. This work factorizes hierarchical search into delegation, execution, and a fixed answer-generation control role, then runs controlled capacity sweeps on multi-hop QA benchmarks.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages21

Reading Time53 min

Abstract

Introduction

arXiv :2607 .07548v 1 [ cs .CL] 8 Jul 2026 THINK BIG, SEARCH SMALL: WHERE CAPACITY MATTERS IN HIERARCHICAL SEARCH AGENTS? Qinnan CaiYibo Zhao∗& Xiang Li † School of Data Science and Engineering East China Normal University ABSTRACT Large language model based search agents increasingly adopt multi-agent architectures in which a main agent decomposes a complex question into sub-queries and dispatches them to parallel sub-agents. However, existing systems instantiate all roles from a single model of identical scale, leaving open how model capacity should be distributed across roles. We factorize hierarchical search into three roles: a delegation role responsible for task decomposition, an execution role responsible for retrieval and evidence extraction, and an answer generation role held fixed as a confound control. We then conduct controlled capacity sweeps along the delegation and execution axes on five multi-hop QA benchmarks. The experiments yield three findings. First, role factorization consistently outperforms a single-agent baseline, improving exact match from 4.5 to 8.6 points across six model scales. Second, capacity sensitivity is asymmetric: scaling the delegation backbone improves EM by ∼ 11 points, whereas scaling the execution sub-agent moves EM by only ∼2.6 points, identifying decomposition as the capability bottleneck. Third, a 1.7B-parameter executor trained via quality-filtered trajectory distillation matches a frontier sub-agent in accuracy while consuming 37% fewer sub-agent tokens, advancing the Pareto frontier. These results suggest a concrete recipe for building hierarchical search agents: concentrate capacity at delegation and downsize execution without sacrificing accuracy. Our code is available at [https://github.com/QinnanCai0115/](https://github.com/QinnanCai0115/)[ ](https://github.com/QinnanCai0115/)[role-factorized-search](role-factorized-search.)[.](role-factorized-search.) 1 INTRODUCTION Large language model (LLM) based search agents have rapidly emerged as a central topic in the LLM community (Singh et al., 2025; Li et al.; Shi et al., 2025; Team et al., 2026) . By interleaving reasoning with retrieval, modern search agents can autonomously decompose a question, issue a sequence of queries, and synthesize evidence scattered across multiple retrieved documents (Jin et al., 2025; Trivedi et al., 2023; Yao et al., 2022; Li et al., 2025b; Jiang et al., 2023) . As both model capability and task ambition scale up, the scope of search keeps expanding from single-hop factoid questions (Mallen et al., 2023) to complex multi-hop questions whose answers must be composed from many pieces of evidence (Yang et al., 2018; Ho et al., 2020; Trivedi et al., 2022; Press et al., 2023), making agentic search a demanding and informative testbed for LLM-based agents. To date, the majority of search agents follow a single-agent route: one model is responsible for planning, issuing queries, reading retrieved documents, and producing the final answer, all within a single shared context (Jin et al., 2025; Song et al., 2025; Gao et al., 2025) . This design is conceptually simple and has proven highly effective (Singh et al., 2025; Li et al.) . By construction, every retrieved passage and every intermediate observation accumulates in the same context window. However, as the number of hops and retrieval rounds grows, this context grows with it: passages that are each only locally relevant to one sub-question pile up alongside the original question and the agent’s own plan (Du et al., 2025; Liu et al., 2024) . The single model must therefore act as ∗Equal contribution. †Corresponding author: [xiangli@dase.ecnu.edu.cn](xiangli@dase.ecnu.edu.cn). planner, reader, and synthesizer over a context that keeps expanding, a tension that grows more pronounced as questions require more reasoning hops and retrieval rounds. A complementary route addresses this tension through multi-agent collaboration. Instead

FAQs

What core problem does the paper address in hierarchical search agents?

The paper studies how to allocate model capacity across different roles in hierarchical search agents rather than using identical-scale models for all roles.

How are roles factorized in the proposed hierarchical search framework?

Hierarchical search is split into a delegation role for task decomposition, an execution role for retrieval and evidence extraction, and an answer generation role kept fixed as a control.

What do the controlled capacity experiments show about which role is the bottleneck?

Capacity sensitivity is asymmetric: scaling the delegation role improves exact match substantially more than scaling the execution role, indicating decomposition as the capability bottleneck.

Jul 16, 2026

Upload by Evangeline