Scientific Code Search at Scale: A Multi-Domain Dataset and Benchmark

Jul 16, 2026

Upload by Noah

Scientific research increasingly depends on open-source software, yet finding relevant scientific tools among over 600 million GitHub repositories remains difficult. Existing code search benchmarks emphasize general software engineering and miss scientific computing’s domain-specific terminology and data formats. This work releases a curated corpus of 5,264 high-quality, domain-classified NASA scientific repositories enriched with cleaned READMEs and extracted context. It adds two retrieval benchmarks: 219 expert-built repository queries and a large code snippet benchmark with 117,950 snippets and 119,720 queries across seven languages, with public HuggingFace release.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages32

Reading Time1 h 21 min

Abstract

Introduction

Related Work

Data Collection

arXiv :2607 .05443v 1 [ cs .IR] 3 Jul 2026 Scientific Code Search at Scale: A Multi-Domain Dataset and Benchmark Nishan Pantha 1 , Pranath Reddy Kumbam 1 , Sajil Awale 1 , Pushwitha Krishnappa 1 , Muthukumaran Ramasubramanian 1 , Nidhi Jha 1 , Emily Foshee 1 , Ankur Kumar 1 , Rachel Slank2 , Ashkbiz Danehkar2 , and Rahul Ramachandran3 1 The University of Alabama in Huntsville (UAH), Huntsville, AL, USA 2 Universities Space Research Association (USRA), USA 3 NASA Marshall Space Flight Center, Huntsville, AL, USA Abstract Scientists increasingly rely on open-source tools to support their research workflows, yet discovering relevant software among over 600 million GitHub repositories remains challenging. Existing code search benchmarks focus on general software engineering tasks and fail to capture the domain-specific vocabulary and needs of scientific computing. We present a curated corpus of 5,264 high-quality, domain-classified scientific repositories spanning five NASA Science Mission Directorate divisions—Earth Science, Astrophysics, Planetary Science, Heliophysics, and Biological & Physical Sciences—enriched with cleaned READMEs, extracted topics, and additional context from crawled links. Building on this corpus, we introduce two novel information retrieval benchmarks: (1) a repository search benchmark with 219 expert-curated queries designed by domain scientists, and (2) a large-scale code snippet retrieval benchmark containing 117,950 code snippets and 119,720 queries across seven programming languages. Baseline evaluations on repository search reveal significant performance variation across scientific domains. Code snippet retrieval proves equally challenging, with substantial variation driven by differing documentation practices, coding standards, and programming language conventions across scientific communities. All datasets and benchmarks are publicly released on HuggingFace to support research on scientific tool discovery. Keywords: code search, information retrieval, scientific software, benchmark datasets, code retrieval 1 Introduction The open science movement has transformed how scientific software is developed and shared [18] . Researchers increasingly publish code on GitHub, enabling reproducibility and accelerating discovery through reuse [3 , 6] . However, with over 600 million repositories on GitHub 1 , finding relevant scientific tools remains challenging. Scientific code discovery differs from general-purpose code search: researchers seek tools for specific data formats (e.g. , FITS in astronomy, NetCDF in climate science), particular instruments or missions, and domain-specific algorithms—queries requiring specialized vocabulary that general search systems struggle to capture [6] . 1[https://github.blog/news-insights/octoverse/octoverse-a-new-developer-joins-github-every-secon](https://github.blog/news-insights/octoverse/octoverse-a-new-developer-joins-github-every-secon)d-as-ai-leads-typescript-to-1/ Current platforms like GitHub rely on lexical matching rather than semantic understanding [11]2. A researcher searching for “software to analyze time-series observations from the Kepler mission”may find nothing, while the same tool indexed under “exoplanet transit photometry pipeline” remains undiscovered. This discoverability gap leads to duplicated effort [10], hinders reproducibility [7], and confines effective tools to their originating communities. Existing code search benchmarks—CodeSearchNet [11], CoSQA [9], AdvTest [15]—draw from general-purpose software, reflect software engineering rather than research queries, and lack scientific vocabulary. Progress requires evaluation infrastructure grounded in real scientific queries and expert relevance judgments. We present datasets and benchmarks for scientific code discovery. Our contributions include: 1. Curated Scientific Repository Corpus: 5,264 high-quality, domain-classified GitHub repositories spanning five NASA SMD divis

FAQs

Why is scientific code search harder than general code search?

Scientific discovery requires domain-specific vocabulary, data formats, instruments, and mission-related algorithms. General lexical search benchmarks and systems struggle to match these specialized queries.

What dataset and benchmarks are introduced in this work?

It provides a curated corpus of 5,264 domain-classified scientific repositories and two retrieval benchmarks: a repository search benchmark with 219 expert-curated queries and a code snippet retrieval benchmark with 117,950 snippets and 119,720 queries across seven programming languages.

How do the authors evaluate retrieval across scientific domains?

Baseline evaluations for repository search show significant performance variation across scientific domains, with code snippet retrieval also varying due to documentation practices, coding standards, and language conventions.

Jul 16, 2026

Upload by Noah