Evaluating Semantic and Quality-Aware Retrieval for Source Code Repositories

Jul 16, 2026

Upload by Eden

Keyword-based retrieval in source code repositories underperforms when queries are written in natural language or target implementation intent and code quality instead of exact tokens. The work evaluates a prototype retrieval system combining function-level fragmentation, text-and-code embeddings stored in ChromaDB, LLM-derived quality metadata, and four retrieval modes: semantic, quality-filtered, hybrid, and automatic routing. Experiments use an educational C-code corpus with 15 manually judged queries, reporting semantic retrieval as the strongest overall approach.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages12

Reading Time30 min

Introduction

Background

Research Article Open Access Marek Horváth* and Emília Pietriková Evaluating Semantic and Quality-Aware Retrieval for Source Code Repositories arXiv :2607 .09 16 1v 1 [ cs . SE] 10 Jul 2026 Abstract: Keyword-based retrieval is limited for sourcecode repositories when queries are expressed in natural language or concern implementation intent and code quality rather than exact tokens. This study evaluates a prototype retrieval system that combines function-level fragmentation, text-and-code embeddings, ChromaDB vector storage, LLM-derived quality metadata, and four retrieval modes: semantic, quality-filtered, hybrid, and automatic routing. The concrete evaluation uses an educational Ccode corpus. The full corpus contains 563 anonymized programmer identifiers and 8,951 C files; a reproducible 10% indexed sample contains 56 programmer identifiers, 847 files, and 3,839 fragments. Across 15 manually judged queries, semantic retrieval achieved nDCG@5 of 0.820, Success@5 of 0.800, and MRR of 0.644 . The automatic router selected the expected mode for all 15 queries. In a small manual audit, LLM-derived quality scores were within one point of the manual assessment for 9 of 12 fragments. Within the reported query set, semantic retrieval was the strongest overall mode, while explicit quality metadata was most useful for explicitly quality-oriented queries. Keywords: semantic code search, retrieval-augmented generation, vector embeddings, source code analysis, code quality evaluation 1 Introduction Source-code repositories are difficult to inspect when the search intent concerns behavior, design choices, or code quality rather than exact tokens. Exact-token search remains useful when the user knows a function name, library call, or identifier, but it is less suitable for intent-level questions such as which fragments read from files, use dynamic memory allocation, or handle input errors. Such questions *Corresponding author: Marek Horváth, Technical University of Košice, Faculty of Electrical Engineering and Informatics, Department of Computers and Informatics, Košice, Slovakia, E-mail: [marek.horvath@tuke.sk](marek.horvath@tuke.sk) Emília Pietriková, Technical University of Košice, Faculty of Electrical Engineering and Informatics, Department of Computers and Informatics, Košice, Slovakia, E-mail: [emilia.pietrikova@tuke.sk](emilia.pietrikova@tuke.sk) require retrieval based on behavior and context rather than literal token overlap. Related source-code exploration work has also used static analysis and programmer profiles to summarize code artifacts beyond direct keyword matching [15] . Natural-language code search addresses part of this problem by mapping queries and code fragments into a shared vector space [13] . In source-code repositories, however, topical similarity is not the only useful signal. Users may also ask quality-oriented questions about readability, error handling, input validation, or memory management. Adding explicit quality metadata can support such questions, but it can also distort topical relevance by promoting fragments that score well on quality dimensions while only weakly matching the requested functionality. Educational C-code repositories provide a useful concrete evaluation setting because many implementations solve related tasks under comparable constraints. Such repositories occur in programming courses that combine repeated assignments, automated assessment, and gamecreative or problem-based learning activities [16, 17] . Atthe same time, findings from such a corpus should not be generalized to other repository types without further evaluation. This study therefore compares retrieval modes rather than assuming that quality scoring necessarily improves search. The evaluated prototype indexes sourcecode fragments, attaches LLM-derived quality metadata, stores embeddings and metadata in ChromaDB, and supports semantic, quality-filtered, hybrid, and automatically routed retrieval. R

FAQs

Why keyword-based retrieval is limited for source code repositories?

It struggles when queries describe natural-language intent or code-quality aspects rather than exact tokens. This makes literal token overlap a weak signal for behavior and implementation intent.

What retrieval system components and modes does the study evaluate?

The prototype fragments functions, uses text-and-code embeddings in ChromaDB, attaches LLM-derived quality metadata, and supports semantic, quality-filtered, hybrid, and automatic routing retrieval modes.

What do the reported results suggest about semantic vs. quality-aware retrieval?

Across the reported query set, semantic retrieval shows the strongest overall performance. Explicit quality metadata is most useful for queries that are explicitly oriented toward quality dimensions.

Jul 16, 2026

Upload by Eden