A RAG-Based Question-Answering Solution for Cyber-Attack Investigation and Attribution

Jul 18, 2026

Uploaded by Finn

RAG-based question-answering is proposed to support cybersecurity analysts in investigating cyber-attacks and attributing responsible actors. The approach combines a Retrieval Augmented Generation pipeline with a large language model to answer user queries using either a curated knowledge base or user-provided external resources. Evaluation includes multiple question categories and metric-driven assessment, including faithfulness, relevancy, context precision/recall, entity recall, similarity, and correctness. The method is compared with GPT-3.5 and GPT-4o, showing improved reliability through source-grounded answers and reduced hallucinations, and better results when few-shot examples are used.

LanguageEnglish

CategoryResearch & Report

FormatPDF

Pages20

Reading Time50 min

AccessFree to download

Introduction

Attribution levels and challenges

RAG for mitigating hallucinations

Evaluation methodology

arXiv :2408 .06272v1 [ cs .CR] 12 Aug 2024 A RAG-Based Question-Answering Solution for Cyber-Attack Investigation and Attribution Sampath Rajapaksha⋆[0000−0001−7772−3774], Ruby Rani⋆[0000−0003−1257−8478], and Erisa Karafili [0000−0002−8250−4389] University of Southampton University Road, Southampton SO17 1BJ, UK {srwg1m24, r.rani, [e.karafili}@soton.ac.uk](e.karafili}@soton.ac.uk) Abstract. In the constantly evolving field of cybersecurity, it is imperative for analysts to stay abreast of the latest attack trends and pertinent information that aids in the investigation and attribution of cyberattacks. In this work, we introduce the first question-answering (QA) model and its application that provides information to the cybersecurity experts about cyber-attacks investigations and attribution. Our QA model is based on Retrieval Augmented Generation (RAG) techniques together with a Large Language Model (LLM) and provides answers to the users’ queries based on either our knowledge base (KB) that contains curated information about cyber-attacks investigations and attribution or on outside resources provided by the users. We have tested and evaluated our QA model with various types of questions, including KB-based, metadata-based, specific documents from the KB, and external sourcesbased questions. We compared the answers for KB-based questions with those from OpenAI’s GPT-3 .5 and the latest GPT-4o LLMs. Our proposed QA model outperforms OpenAI’s GPT models by providing the source of the answers and overcoming the hallucination limitations of the GPT models, which is critical for cyber-attack investigation and attribution. Additionally, our analysis showed that when the RAG QA model is given few-shot examples rather than zero-shot instructions, it generates better answers compared to cases where no examples are supplied in addition to the query. Keywords: cyber-attack attribution · LLMs · RAG · QA 1 Introduction Investigating and attributing cyber-attacks are crucial processes, as they allow the implementation of efficient countermeasures. Cyber-attack attribution, which involves identifying the attacker responsible for a cyber-attack, is an essential process that enables experts to implement attacker-oriented countermeasures and pursue legal actions. There are three levels of attribution. The first level ⋆ These authors contributed equally to this work. Thus, the alphabetical order is applied. 2 Rajapaksha, Rani, and Karafili involves understanding the tools, tactics, techniques, and procedures (TTPs) used by the attacker. The second level identifies the geographical location of the attack. The third and most critical level determines the individual or organization behind the attack [10] . Details about cyber-attacks are published in various formats, such as text and PDF. Cybersecurity professionals often need to sift through vast amounts of these textual data to stay ahead of evolving threats [11,7] . The lack of a standard for

FAQs

How does few-shot prompting affect performance?

When few-shot examples are provided instead of only zero-shot instructions, the model generates better answers, as shown by the documented comparison in the work.

Jul 18, 2026

Uploaded by Finn