\chapter{AI Integration for Threat Detection}
\section{Introduction}
Modern Security Operations Centers (SOCs) are usually filled with telemetry: a single investigation (using Security Onion) potentially traverses Suricata alerts, Zeek connection and application logs, Elastic Agent events, and historical context stored across multiple indices. The volume and heterogeneity of these data make the initial steps of triage the most complex and highly demanding phase in a typical threat detection flow: formulating the right searches, paging through raw hits, and stitching together a coherent story... This is just a small portion of the work that analysts have to do when processing and depicting information from the triage procedures. Traditional dashboards and saved searches help, but they still require translation from the intent of an analyst (\emph{“show me everything related to this host around this time”}) to a detailed implementation detail (\emph{“which indices, which fields, which filters, which sort?”}). In terms of enterprise scale, this translation requires considerable effort, effectively delaying the time to first insight.
\medskip \newline
Large Language Models (LLMs) are increasingly embedded into corporate workflows precisely to reduce that translation time: they can map natural-language intent to structured actions, summarize noisy evidence, and surface pivots worth a second look.
Nonetheless, “adding AI” throughout security operations is not equivalent to a fully autonomous AI-managed classification cycle. It is a design choice for \emph{where} to place assistive intelligence so that analysts remain in control and evidence remains reproducible. This thesis adopts this philosophy. Within the Network Detection \& Response platform built on Security Onion (Zeek, Suricata, and Elasticsearch), the AI component is set as a prompt-driven assistant that \emph{reads} from the same data the SOC already trusts and produces end-to-end auditable outputs (prompt $\rightarrow$ query $\rightarrow$ result), aligning with the project’s stated objective to explore an AI-assisted workflow on top of the deployed stack.
\medskip \newline
The motivation is pragmatic and it starts from a simple observation: analysts think in questions, while Security Onion stores answers in Elasticsearch. An effective assistant must therefore translate intent into the concrete language that Elasticsearch understands. In practice, the connection is established through OpenWebUI, which hosts a small “tool” that communicates with the Security Onion Elasticsearch endpoint using a scoped, read-only API key; the assistant never writes data, and all traffic can be protected with TLS. When the analyst types a prompt (\emph{“investigate this source IP over the last 48 hours and correlate HTTP, DNS, and alerts”}), the assistant turns that request into an Elasticsearch \emph{Domain-Specific Language} Query — i.e., a structured JSON search body with filters, time bounds, selected fields, and (when needed) aggregations. This DSL is the bridge between human intent and machine retrieval: it describes \emph{what} to fetch (indices, datasets, pivots like \texttt{network.community\_id} or Zeek \texttt{uid}) and \emph{how} to shape the results (sorting, size limits), while remaining inspectable and reproducible. In this way prompts become DSL, DSL becomes results, and results become short, verifiable narratives. In this sense, LLMs do not “automate triage” so much as they lower the time required to get from a question to first evidence, while fitting into enterprise expectations of security, provenance, and accountability.
\clearpage

\section{Environment Setup}
The integration relied on a layered containerized environment designed to preserve separation of concerns while still allowing the assistant to query live Security Onion data. OpenWebUI, chosen as the platform for the LLM interaction orchestration, was deployed in two stages. A first container was installed locally on the analyst’s machine and served as a working environment for development and debugging. This container mirrored the production OpenWebUI deployment but ran in an isolated user-controlled environment; at this point, the developer was able to iterate on the Python tool without interfering with the corporate infrastructure. Once tested, the local container relayed prompts to the original OpenWebUI instance hosted in the company’s environment, in which resided an NVIDIA H100 GPU server. On that host, the production-grade Large Language Model, \emph{Qwen2.5 32B}, was running and made accessible only inside the corporate network. This arrangement reflected a realistic enterprise configuration in which personal development workstations interact with the centralized, resource-intensive AI infrastructure while maintaining clear trust boundaries.
\medskip \newline
The decision to expose the company-hosted LLM only inside the protected and monitored LAN further underscored the experimental but security-conscious nature of the setup. Unlike a production deployment, where TLS + certificate would normally be mandatory to secure communications, here traffic did not traverse untrusted networks. The assistant could only reach the LLM endpoint from within the internal LAN, which was monitored and access-controlled, ensuring that no external party could exploit the integration. This choice simplified the configuration during experimentation while still respecting the principles of containment and least privilege.
\medskip \newline
From the Security Onion perspective, the connection was established directly against Elasticsearch through its REST APIs. This required the creation of a dedicated AI user profile, configured via Kibana’s role management interface. A role was defined with the sole privilege of reading and viewing index metadata from the indices relevant to network threat detection (\texttt{.alerts-security.alerts}, \texttt{.ds-logs-zeek-*}, \texttt{.ds-logs-suricata-*}, among others). Then, this role was associated with a new service user, from which an API key was generated. The key was injected into the OpenWebUI tool at run-time and allowed the assistant to perform queries without ever writing to or altering Security Onion data. Because the API key was tied to a dedicated role and user, its lifecycle could be monitored and revoked independently of other system accounts, aligning with enterprise requirements for accountability.
\medskip \newline
Before proceeding, a short comparison is due to clarify why the integration favored the use of Elasticsearch APIs instead of Security Onion’s own native APIs. The community edition of Security Onion does not expose a public, stable API for programmatic queries beyond the SOC graphical interface. Functionalities as such are reserved for the professional edition of the platform, which provides additional integration hooks for automation. Conversely, Elasticsearch is already the storage and query back-end for Security Onion logs, and its REST API is fully documented, widely adopted, and suitable for direct integration. Using the Elastic API thus ensured that the assistant could operate with the freely available community edition of Security Onion, while also following practices already familiar to enterprises that integrate Elastic into SIEM pipelines. In effect, the assistant speaks the “native language” of the data store, making its queries portable, inspectable, and auditable without dependence on proprietary extensions.
\medskip \newline
This set-up of the environment reflects a balance between practicality and alignment with the realities of enterprise deployment. A container loaded on a local machine enabled agile development, whereas the company-hosted H100 GPU served the resource-intensive Qwen2.5 32B model under strict LAN-only access, Elasticsearch APIs provided a standard and open integration point, and the use of a scoped API key maintained clear limits of responsibility. These design decisions not only made the experiment feasible in a constrained setting but also highlighted how such an assistant could be deployed responsibly in production environments where stronger guarantees such as TLS, external audit logging, and API usage monitoring would be required.
\begin{figure}[H]
    \centering
    \includegraphics[width=1\linewidth]{figures/content/Environment.png}
    \caption{Environment Setup}
    \label{fig:env-setup}
\end{figure}
\clearpage

\subsection*{Operational Trust Boundaries}
In practice, establishing the integration required a series of configuration steps that defined the operational boundaries of the experiment, which follow here:
\begin{itemize}
  \item \textbf{Deployment of a local OpenWebUI container.} A fresh instance of OpenWebUI was downloaded and configured on the analyst’s machine. Through its administrative panel, this container was then connected to the original LLM (Qwen2.5~32B hosted on the H100 GPU server) using a URL and an API key. The API key was issued to the analyst’s experimental account, which was a valid LDAP account subject to regular company governance and monitoring.
  \item \textbf{Network adjustments on the NG-FW.} Network configurations on the NG-FW. Firewall policies on the next-generation firewall were increased to allow the LLM access to Security Onion, but through the internal company network only. There were not any outside access routes.
  \item \textbf{Firewall adjustments on Security Onion.} The internal firewall configuration of Security Onion was edited to allow incoming requests from the LLM host to the Elasticsearch REST API endpoints, while other services remained closed.
  \item \textbf{Python environment preparation.} Preparing Python environment. In the OpenWebUI container, appropriate Python modules were set up and installed to communicate with Elasticsearch using a custom-made tool that can be utilized to authenticate using the API key and submit questions.
\end{itemize}
These steps, though experimental, guaranteed that the assistant would run end-to-end without circumventing existing governance: using a scrutinized LDAP account kept accountability, firewall edits were localized to the LAN perimeter, and the assistant ran with read-only access against Elasticsearch.
\clearpage

\section{OpenWebUI}
OpenWebUI is a modern, extensible, and user-friendly interface for self-hosting Large Language Models (LLMs). Designed to operate entirely offline if required, it supports a variety of LLM runners—including Ollama and OpenAI-compatible APIs, making it a powerful yet modular platform for enterprise deployments. 
\cite{openwebui_docs}
\medskip \newline
OpenWebUI provides a ChatGPT-style web interface with robust features like role-based access control, support for Retrieval-Augmented Generation (RAG), and full Markdown plus LaTeX support. Administrators benefit from multi-user management, SCIM\footnote{SCIM (System for Cross-domain Identity Management) is an open standard for identity lifecycle management (provision, update, deprovision) across systems using REST/JSON.}-compatible provisioning, and a Progressive Web App (PWA) experience for mobile devices. These capabilities enable companies to deliver AI-powered assistance across teams with controlled access, auditability, and rich interaction options. \cite{openwebui_docs}
\medskip \newline
Installation and deployment come with many flavors, supporting Docker, Kubernetes (via Helm or Kustomize), and even standalone Python setups through pip. GPU-enabled environments are supported via tags like `:cuda`, while Ollama integration for LLM model management is available via `:ollama` Docker tags\cite{openwebui_docs}. This flexibility allows organizations to adopt OpenWebUI incrementally, starting with local experimentation and scaling to full production usage with enterprise SLAs, theme customization, and support.
\medskip \newline
Following the earlier description, OpenWebUI results as an efficient open-source interface to a self-hosted LLM, offering a functional combination of usability, modularity, and governance. Its modular “tool” system allows developers to extend capabilities, in this case integrating with external APIs and Elasticsearch back-ends, without touching the core UI. This made it an ideal choice for integrating an assistant into Security Onion, as the existing framework could be extended with a custom “tool” for query composition and retrieval, while letting enterprise users interact via a polished and familiar web UI.

\subsection{Valves and Tools in OpenWebUI}
OpenWebUI’s extensibility is based on two complementary constructs: \texttt{Tools} and \texttt{Valves}, each serving distinct purposes yet working hand in hand to build a dynamic and secure interface.

\paragraph{Tools}  
In OpenWebUI, \textbf{Tools} are modular extensions that add capabilities beyond the pre-trained knowledge of an LLM. They can access external data sources, invoke APIs, or execute custom Python logic, enabling use cases like live data retrieval, domain-specific analysis, or security query execution. Tools are registered via the interface, often require minimal configuration, and are activated transparently during an LLM conversation when relevant. In this integration, our custom tool functions as a bridge between user prompts and Elasticsearch DSL queries, enabling reactive log searches within Security Onion. This approach leverages the built-in tool framework—requiring no modification of OpenWebUI’s core and benefiting from the plug-and-play model of extensions. \cite{openwebui_docs}

\paragraph{Valves}  
\textbf{Valves} are configuration hooks that allow administrators to inject dynamic parameters into pipelines, tools, or filters. Defined as Pydantic models (`Valves` or `UserValves` classes), they translate into GUI elements—dropdowns, input fields, toggles—that provide operators control over behavior without coding. For instance, an API key or index name can be exposed via a Valve, enabling flexible run-time configuration. Valves are admin-configurable via the WebUI and can optionally extract values from environment variables to support different deployment scenarios. \cite{openwebui_docs}

\paragraph{Tools \& Valves in Action}  
In practice, integrating with Security Onion meant creating a custom Tool class that implements search logic using Elasticsearch’s REST API. A Valve was used to define the read-only API key parameter, enabling secure configuration without embedding secrets in the code. In this way, operators retained control using the Valve UI, while analysts could use the custom tool throughout the chat with the LLM.
In other words, Tools and Valves provide OpenWebUI with the advantage of extensibility and manageability: Tools unlock new functionality, while Valves make it safely configurable.
\clearpage

\section{The Tool for Elasticsearch Integration}
To effectively establish the connection between OpenWebUI and Security Onion, a dedicated \texttt{Tool} class was developed. This class acts as the integration layer between analyst prompts and Elasticsearch queries, ensuring that requests are translated into valid, scoped, and auditable operations. The design emphasized clarity and modularity: all query logic is encapsulated within the Tool, while OpenWebUI provides the interface for invoking it during a chat session.
\medskip \newline
The Tool initializes two crucial elements. First, an \textbf{Elasticsearch connector} is set to instantiate the interface of communication with the Security Onion back-end using authentication via a read-only API key that has been specifically chosen to fulfill this purpose. Later, a \texttt{FIELDS\_BY\_DATASET} structured dictionary provides a standardized set of mappings of all the available fields across datasets, including \texttt{zeek.conn} and \texttt{suricata.alert}. The mappings themselves are then organized into the \emph{always}, \emph{conditional}, and \emph{rare} classifications and thus give the LLM a good understanding of the available fields and the correct querying processes, consequently reducing hallucinations and minimizing the chance of generating incorrect DSL queries.
\medskip \newline
The \texttt{FIELDS\_BY\_DATASET} dictionary serves as a schema-aware protection, outlining the fields that are of highest relevance to investigatory purposes. This allows the assistant to efficiently prioritize information retrieval and filter structuring within Elasticsearch DSL queries. In turn, prompts are converted from open-ended or groundless questions into specifically outlined requests that conform to the set system of Security Onion indices held within Elasticsearch.
\begin{lstlisting}[style=python, caption={Excerpt of \texttt{FIELDS\_BY\_DATASET}}, label={lst:fields-example}]
self.FIELDS_BY_DATASET = {
    "suricata.alert": {
        "always": {
            "@timestamp": "date",
            "rule.name": "keyword",
            "rule.severity": "long",
            "rule.action": "keyword",
            "source.ip": "ip",
            "destination.ip": "ip",
            "network.transport": "keyword",
            "network.community_id": "keyword",
            "log.id.uid": "keyword", 
            "event.dataset": "keyword",
        },
        "conditional": { ... },
        "rare": { ... },
    },
    "zeek.conn": { # additional schema here... },
    # more schemas ...
}
\end{lstlisting}
With this foundation, the Tool defines three core methods, each supporting a distinct aspect of the investigative workflow:
\begin{enumerate}
    \item A \emph{Matching Query Builder} for retrieving raw logs tied to specific indicators,  
    \item An \emph{Aggregation Query Builder} for summarizing activity patterns, and  
    \item A \emph{Correlator} for stitching together multi-dataset evidence into coherent timelines.
\end{enumerate}  
The following subsections examine these methods in detail, illustrating how they enable Security Onion to be queried through natural-language prompts while maintaining control and reproducibility.

\subsection{\texttt{run\_dsl\_query} (Matching)}
\noindent
\textbf{Objective.}
The method \texttt{run\_dsl\_query} dynamically creates and executes Elasticsearch matching (non-aggregation) queries. It provides a natural-language interface into Security Onion's Elasticsearch while, through enforced limitations and subsequently following processing, ensuring validity, efficiency, and safety.

\paragraph{Pipeline overview.}
The process drives a three-phase, LLM-assisted pipeline:
\begin{enumerate}
  \item \textbf{Index, dataset, and size selection.} The Large Language Model answers the analyst prompt by returning a strict triple \texttt{index,dataset,size}. The index is normalized, the dataset is left as selected, and the size defaults to 20 unless specified.
  \item \textbf{Field selection.} The LLM takes the dataset-dependent dictionary of fields \newline\texttt{FIELDS\_BY\_DATASET[dataset]} and returns a flat list of fields in the JSON format. The step always prioritizes \emph{always} fields, then adds \emph{conditional} fields if necessary, and \emph{rare} fields only absolutely necessary.
  \item \textbf{Body synthesis filtering.} The LLM generates solely the \texttt{"query": \{...\}} object, devoid of any top-level directives. The prompt encompasses canonical examples pertaining to Suricata and Zeek datasets to ground both syntax and semantics.
\end{enumerate}

\noindent
\textbf{Stage 1: index, dataset, size.}
A constrained system message forces a CSV response \texttt{index,dataset,size}. The output is sanitized and split:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
resp = await generate_chat_completion(__request__, form_index, user)
triple = sanitize_model_response(resp["choices"][0]["message"]["content"].strip())
size   = triple.split(",")[2].strip()
dataset = triple.split(",")[1].strip()
index  = normalize_index(triple.split(",")[0].strip())
\end{lstlisting}

\noindent
\textbf{Stage 2: schema-aware field selection.}
The model receives the dataset and the full field dictionary and must return only a flat JSON list:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
resp = await generate_chat_completion(__request__, form_fields, user)
fields = sanitize_model_response(resp["choices"][0]["message"]["content"].strip())
\end{lstlisting}

\noindent
\textbf{Stage 3: query body synthesis with exemplars.}
The system message instructs the model to emit only the filtering logic and bans risky constructs. Several minimal examples (Suricata alerts, Zeek DNS/HTTP/Conn) are injected as anchors. The raw content is then cleaned from accidental markdown fences:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
resp = await generate_chat_completion(__request__, form_data, user)
content = sanitize_model_response(resp["choices"][0]["message"]["content"])
cleaned = re.sub(r"^```(?:json)?|```$", "", content.strip(), flags=re.MULTILINE).strip()
\end{lstlisting}

\noindent
\textbf{Validation and assembly (guardrails).}
Model output is never executed as-is. It is unwrapped, parsed, type-checked, and then augmented with \texttt{\_source} fields and size via a helper:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
if isinstance(cleaned, str):
    if cleaned.startswith('"') and cleaned.endswith('"'):
        cleaned = json.loads(cleaned)
    if isinstance(cleaned, str):
        cleaned = json.loads(cleaned)
if not isinstance(cleaned, dict):
    raise ValueError("Decoded content is not a dictionary.")

dsl_query = assemble_final_dsl_query(cleaned, dataset, fields, int(size))
\end{lstlisting}

\noindent
\textbf{Execution and return modes.}
The finalized query is sent to Elasticsearch. The function supports raw and enriched outputs:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
results = es.search(index=index, body=dsl_query)
hits = results["hits"]["hits"]
if return_raw:
    return hits
return f"Show the user : {json.dumps(hits, indent=2)} and enrich with your considerations..."
\end{lstlisting}

\medskip
\noindent
\textbf{Correctness and safety guardrails.}
The system message constrains the expressive power of the LLM to avoid heavy or unsafe queries:
\begin{itemize}
  \item \emph{No scripts}: disallow \texttt{doc['field'].value} and script queries.
  \item \emph{Fuzziness}: only allowed on \texttt{text}/\texttt{keyword} fields; not on IPs, numerics, dates, or booleans.
  \item \emph{Keyword matching}: favor \texttt{regexp} with \texttt{case\_insensitive: true} on fuzzy searches over keyword fields with time filters and explicit size to minimize load.
  \item \emph{Lenient matching}: Privilege the use of \texttt{should} plus \texttt{minimum\_should\_match} rather than using \texttt{must}, unless strict filters are necessary.
  \item \emph{Structure}: all of the clauses within \texttt{must}/\texttt{should}/\texttt{must\_not}; never put filters as siblings of \texttt{bool}, and never as top-level directives within the model output.
\end{itemize}

\medskip
\noindent
\textbf{Failure handling.}
If the model returns malformed JSON or an unexpected structure, the function raises a localized error. In \texttt{return\_raw=True} mode a structured error object is returned; otherwise the user receives a controlled, readable message. Empty result sets are acknowledged with possible operational interpretations (for example, insufficient time window, mis-scoped dataset, missing logs, or benign absence).

\subsection{\texttt{run\_agg\_dsl\_query} (Aggregations)}

\noindent
\textbf{Goal.}
The method \texttt{run\_agg\_dsl\_query} extends the assistant with the ability to generate and execute Elasticsearch aggregation queries. It is invoked whenever the user request involves grouped insights or statistical summaries rather than raw log retrieval, e.g., “most frequent source IPs”, “average bytes transferred”, or “distribution of alerts over time”.

\paragraph{Pipeline overview.}
The approach executes an analogous multi-stage pipeline as the matching query approach but tailored for aggregation:
\begin{enumerate}
  \item \textbf{Index/dataset selection.} The system selects the index and dataset that are most relevant considering the intent of the user.
  \item \textbf{Query synthesis.} Utilizing field dictionaries, the model assembles an Elastic-search DSL query consisting of optional filters as well as a single or multiple blocks of aggregation.
  \item \textbf{Execution and normalization.} The query is performed on Elastic-search, and the unprocessed aggregation outcomes are standardized into a uniform JSON summary.
\end{enumerate}

\noindent
\textbf{Stage~1: index/dataset.}
The first step reuses the same prompting strategy as the matching query pipeline, but only expects an index and a dataset:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
resp = await generate_chat_completion(req, form_index, user)
triple = sanitize_model_response(resp["choices"][0]["message"]["content"])
index  = normalize_index(triple.split(",")[0].strip())
dataset = triple.split(",")[1].strip()
\end{lstlisting}

\noindent
\textbf{Stage~2: query synthesis with aggregations.}
The model is explicitly instructed to generate a top-level DSL object with \texttt{"size": 0} and one or more \texttt{aggs}.  
Examples anchor typical use cases such as \emph{top rule names}, \emph{average bytes}, or \emph{most common ports}:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
example1_query = {
  "query": {"bool": {"must": [{"term": {"event.dataset": "suricata.alert"}}]}},
  "size": 0,
  "aggs": {
    "top_rule_names": {
      "terms": {"field": "rule.name.keyword", "size": 10}
    }
  }
}
\end{lstlisting}

\noindent
\textbf{Stage~3: validation and execution.}
As enforced for matching queries, the model’s output is sanitized, parsed, and validated before submission to Elasticsearch:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
cleaned = re.sub(r"^```(?:json)?|```$", "", content.strip())
dsl_query = fix_json_agg_helper(json.loads(cleaned))
results = es.search(index=index, body=dsl_query, _source=False)
\end{lstlisting}

\noindent
\textbf{Stage~4: result normalization.}
To ensure consistent downstream handling, the function translates the Elasticsearch aggregation structures into a compact summary. Bucket-based groupings, single-value metrics, multi-value statistics, and top-hits aggregations are unified under a common schema:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
if "buckets" in agg_data:
    agg_result_summary[agg_name] = {
        "type": "buckets",
        "buckets": [{"key": b["key"], "count": b["doc_count"]}
                    for b in agg_data["buckets"]]
    }
elif "value" in agg_data:
    agg_result_summary[agg_name] = {"type": "single_value",
                                    "value": agg_data["value"]}
\end{lstlisting}

\medskip
\noindent
\textbf{Correctness and safety guardrails.}
\begin{itemize}
  \item \emph{Structure}: enforce top-level siblings \texttt{query} and \texttt{aggs}, never nested.
  \item \emph{Types}: only use \texttt{terms} on keyword/IP fields, statistical aggregations on numeric fields, and \texttt{date\_histogram} on \texttt{@timestamp}.
  \item \emph{Size}: always set \texttt{"size": 0} to avoid returning documents alongside buckets.
  \item \emph{Forbidden constructs}: no scripting, no \texttt{doc['field'].value}, no extraneous formatting.
\end{itemize}

\medskip
\noindent
\textbf{Return modes.}
If \texttt{return\_raw} is enabled, the method returns the normalized JSON summary for direct programmatic use. Otherwise, the assistant enriches the summary with human-readable commentary, interpreting trends such as dominant alert categories or traffic distributions.

\subsection{\texttt{run\_correlation} (Multi-dataset Pivoting)}
\noindent
\textbf{Goal.}
The method \texttt{run\_correlation} performs targeted, multi-dataset correlation starting from a concrete alert selected by the analyst. It is the third step in the pipeline after matching (\texttt{run\_dsl\_query}) and aggregation (\texttt{run\_agg\_dsl\_query}). Given a natural-language instruction referring to a specific alert, the method extracts robust pivots (timestamp, \texttt{network.community\_id}, optional \texttt{source.ip / destination.ip}, queries multiple datasets (e.g., \texttt{suricata.alert}, \texttt{zeek.conn / http/dns}), and returns a consolidated, chronologically sorted context.

\paragraph{Pipeline overview.}
\begin{enumerate}
  \item \textbf{Pivot extraction.} The LLM parses the user prompt to obtain \emph{timestamp} and \emph{correlation id} (\texttt{network.community\_id}); \texttt{source.ip}/\texttt{destination.ip} are optional. If timestamp or id are missing, the function is aborted upfront with a diagnostic.
  \item \textbf{Primary query.} Create a conservative time window (24 hours back from the alert’s timestamp) and a \texttt{bool.must} set from the available pivots, then search through \texttt{.ds-logs-*-so-*}.
  \item \textbf{Dataset discovery \& field planning.} From primary hits, derive the involved datasets and ask the LLM to select a minimal, correlation-oriented \texttt{\_source} set per dataset. Merge into a global \texttt{\_source}.
  \item \textbf{Secondary correlation (UIDs/FUIDs).} If present, derive \texttt{log.id.uid}, \texttt{log.id.uids}, \texttt{log.id.fuid} and run a second-tier query (same time window) to pull additional context.
  \item \textbf{Normalization.} Merge, de-duplicate \((\_index,\_id)\), and sort chronologically. Return either a machine-friendly list or an enriched narrative.
\end{enumerate}

\noindent
\textbf{Stage~1: pivot extraction (LLM-assisted).}
The model must return \emph{only} a strict JSON object; no markdown or commentary is accepted:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
resp = await generate_chat_completion(__request__, form_pivot_extractor, user)
raw  = sanitize_model_response(resp["choices"][0]["message"]["content"].strip())
cleaned = re.sub(r"^```(?:json)?|```$", "", raw.strip(), flags=re.MULTILINE).strip()

if isinstance(cleaned, str) and cleaned.startswith('"') and cleaned.endswith('"'):
    cleaned = json.loads(cleaned)
if isinstance(cleaned, str):
    cleaned = json.loads(cleaned)
if not isinstance(cleaned, dict):
    raise ValueError("Decoded content is not a dictionary.")

required = {"timestamp", "id"}
if "error" in cleaned or not required.issubset(cleaned):
    raise ValueError("Missing required fields.")
\end{lstlisting}

\noindent
\textbf{Stage~2: primary query assembly.}
Construct the 24h lookback window ending with the alert timestamp and build \texttt{must} clauses from available pivots:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
must = [{"term": {"network.community_id": cleaned["id"]}}]
if cleaned.get("source"):
    must.append({"term": {"source.ip": cleaned["source"]}})
if cleaned.get("destination"):
    must.append({"term": {"destination.ip": cleaned["destination"]}})

ts = parse_to_es_datetime(cleaned["timestamp"].strip())
timerange = {"range": {"@timestamp": {
    "gte": (ts - timedelta(hours=24)).isoformat(timespec="seconds").replace("+00:00","Z"),
    "lte":  ts.isoformat(timespec="seconds").replace("+00:00","Z")
}}}
must.append(timerange)

dsl = {"query": {"bool": {"must": must}}, "_source": ["event.dataset"], "size": 100}
results = es.search(index=".ds-logs-*-so-*", body=dsl)
\end{lstlisting}

\noindent
\textbf{Stage~3: dataset discovery and field selection.}
Collect involved datasets and ask the model for a compact, correlation-useful \texttt{\_source} per dataset:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
datasets = sorted({
    hit["_source"]["event"]["dataset"]
    for hit in results["hits"]["hits"]
    if "event" in hit["_source"] and "dataset" in hit["_source"]["event"]
})
schemas = {ds: self.FIELDS_BY_DATASET[ds] for ds in datasets}

resp = await generate_chat_completion(__request__, form_field_selector, user)
raw_fields = sanitize_model_response(resp["choices"][0]["message"]["content"].strip())
cleaned_fields = re.sub(r"^```(?:json)?|```$", "", raw_fields, flags=re.MULTILINE).strip()
if cleaned_fields.startswith('"') and cleaned_fields.endswith('"'):
    cleaned_fields = json.loads(cleaned_fields)
if isinstance(cleaned_fields, str):
    cleaned_fields = json.loads(cleaned_fields)
if not isinstance(cleaned_fields, dict):
    raise ValueError("Expected a dictionary of fields per dataset.")

source_fields = sorted(set(f for fs in cleaned_fields.values() for f in fs))
source_fields.extend(["log.id.uid", "log.id.fuid", "log.id.uids"])
\end{lstlisting}

\noindent
\textbf{Stage~4: secondary correlation on UIDs/FUIDs.}
If the primary pass reveals \texttt{uid}/\texttt{fuid}, expand the search to fetch adjacent context in the same time window:
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
uids, fuids = set(), set()
for h in results["hits"]["hits"]:
    src = h.get("_source", {})
    ids = src.get("log", {}).get("id", {})
    if ids.get("uid"):  uids.add(ids["uid"])
    if ids.get("fuid"): fuids.add(ids["fuid"])
    if isinstance(ids.get("uids"), list): uids.update(ids["uids"])

follow_hits = []
if uids or fuids:
    should = []
    if uids:
        should += [{"terms": {"log.id.uid": list(uids)}},
                   {"terms": {"log.id.uids": list(uids)}}]
    if fuids:
        should += [{"terms": {"log.id.fuid": list(fuids)}}]

    final_query = {
        "query": {"bool": {"must": [timerange], "should": should,
                           "minimum_should_match": 1}},
        "_source": sorted(set(source_fields)),
        "size": 100
    }
    follow = es.search(index=".ds-logs-*-so-*", body=final_query)
    follow_hits = follow["hits"]["hits"]
\end{lstlisting}

\noindent
\textbf{Stage~5: normalization (merge, dedupe, sort).}
\begin{lstlisting}[style=python,basicstyle=\ttfamily\footnotesize]
all_hits = results["hits"]["hits"] + follow_hits
seen, dedup = set(), []
for h in all_hits:
    key = (h["_index"], h["_id"])
    if key not in seen:
        seen.add(key)
        dedup.append(h)

timeline = sorted(dedup, key=lambda h: h.get("_source", {}).get("@timestamp", ""))
\end{lstlisting}

\medskip
\noindent
\textbf{Return modes.}
The function returns a structured, chronologically ordered set of hits that can be consumed directly or narrated by the assistant. If no hits are found, it surfaces a diagnostic (e.g., missing/incorrect pivots, time window too narrow, logs not ingested).

\medskip
\noindent
\textbf{Correctness and safety guardrails.}
\begin{itemize}
  \item \emph{Strict JSON pivot:} the Extractor should give back a pure JSON object;  absent \texttt{timestamp} or \texttt{id} returns early.
  \item \emph{Restricted range:} a predetermined 24-hour retrospective confines both expenses and variability; all correlation analyses occur within that temporal framework.
  \item \emph{Schema knowledge:} \texttt{\_source} fields will be chosen out of per-dataset schemas\newline(\texttt{FIELDS\_BY\_DATASET}); no field-guessing will be allowed.
  \item \emph{No scripting:} correlation operates on primitives solely by \texttt{term}/\texttt{terms}/\texttt{range}; no scripts or \texttt{doc['field'].value}.
  \item \emph{Stability:} the outputs are de-duplicated by \texttt{(\_index,\_id)} and sorted by timestamp for consistent timelines.
\end{itemize}
\clearpage

\section{Use Cases: A concrete example}
\subsection{Introduction}
This section documents three end--to--end investigations executed with the SOC assistant integrated into Security Onion via OpenWebUI. Each use case should be intended as a representation of one stage of the pipeline introduced in this thesis: \emph{matching} (raw retrieval), \emph{correlation} (cross--dataset pivoting), and \emph{aggregation} (post--incident summarization).

\subsection{Use Case 1: Suspicious Executable Download (Matching)}
\paragraph{Story.}
At morning triage, the analyst notices a high--severity Suricata alert:
\begin{quote}
\textbf{ET INFO exe download via HTTP --- Informational}
\end{quote}
The event suggests an \texttt{.exe} file transferred over plain HTTP from an external server. The analyst needs fast context:
\emph{Which internal host initiated the download? What exactly was fetched?}

\subsubsection*{Analyst prompt.}
\begin{lstlisting}[language=,basicstyle=\ttfamily\footnotesize]
Fetch Zeek http logs over the last 48 hours with source ip 172.22.2.222
\end{lstlisting}

\paragraph{System actions}
\begin{itemize}
  \item Selected dataset \texttt{zeek.http} and index \texttt{.ds-logs-zeek-so-*}; set the size at 20 if not specified otherwise.
  \item created a matching DSL query (filter--only body), incorporating a time filter (48h) as well as an additional \texttt{source.ip} clause.
  \item Retrieved HTTP Request/Response context from Elasticsearch and normalized the hits for display.
\end{itemize}

\paragraph{Outcome}
From the the acquired \texttt{zeek.http} records:
\begin{enumerate}
  \item \textbf{Source IP}: \texttt{172.22.2.222} on all requests (same internal host).
  \item \textbf{Destinations}: IPs matching the Microsoft update infrastructure.
  \item \textbf{User--Agent}: \texttt{Windows-Update-Agent/10.0.10011.16384\newline Client-Protocol/2.50}.
  \item \textbf{Methods/Status}: \texttt{GET} with primarily \texttt{200 OK} and \newline\texttt{304 Not Modified}.
  \item \textbf{URIs}: \texttt{.cab} and related update artifacts.
\end{enumerate}

\paragraph{Considerations.}
\begin{itemize}
  \item \emph{Normal behavior}: as anticipated by Windows Update; however it's worth validating destinations.
  \item \emph{Security posture}: implement allow-lists for executable downloads; alert on bare HTTP where possible.
  \item \emph{Operational hygiene}: consider WSUS or caching for bandwidth; monitor anomalies or unusual endpoints.
\end{itemize}

\paragraph{If results are empty.}
Probable reasons can be: narrow time frame, HTTP logging disabled/failure, offline host, or benign absence of activity.

\subsection{Use Case 2: Inter--Dataset Exploration (Correlation)}
\paragraph{Story.}
The analyst broadens the investigation to include lateral movement, coordinated activity, or C2 patterns related to the same host. The analyst needs to compare Suricata alerts with Zeek connection, HTTP, and DNS logs to check against correlated patterns and behaviors.

\subsubsection*{Analyst prompt.}
\begin{lstlisting}[language=,basicstyle=\ttfamily\footnotesize]
Investigate on source ip 172.22.2.222 and destination ip 199.232.210.172,
consider the network community id 1:gJVYgz5auC081RQ+r8NHFbLxk2o=
and timestamp 2025-04-28T23:00:00.300Z
\end{lstlisting}

\paragraph{System actions.}
\begin{itemize}
  \item Retrieved pivots through LLM: \texttt{timestamp}, \texttt{network.community\_id}, optional \newline\texttt{source.ip} / \texttt{destination.ip}. Early termination if \texttt{timestamp} or \texttt{id} missing.
  \item Create a main query on \texttt{.ds-logs-*-so-*} with a 24h lookback concluding at the pivot timestamp, included \texttt{term} filters as well as a \texttt{range} on \texttt{@timestamp}.
  \item Uncovered involved datasets for primary hits; asked per--dataset \texttt{\_source} fields (correlation--useful only) of the LLM by way of \texttt{FIELDS\_BY\_DATASET}.
  \item Correlation keys (\texttt{log.id.uid}, \texttt{log.id.uids}, \texttt{log.id.fuid}) were extracted and the second query was executed on the same time window.
  \item Merged, de--duplicated by \texttt{(\_index,\_id)}, and chronologically sorted all results.
\end{itemize}

\paragraph{Outcome.}
\begin{itemize}
  \item \textbf{Connections}: \texttt{172.22.2.222} $\rightarrow$ \texttt{199.232.210.172} on \texttt{tcp/80};\newline\texttt{network.community\_id} matches across entries.
  \item \textbf{HTTP}: \texttt{200 OK} for retrieving the resources for updates (such as patch payloads or JSON metadata).
  \item \textbf{Files}: metadata for files available (hashes), allowing for integrity verification.
  \item \textbf{Alerting}: Suricata rule \texttt{ET INFO exe download via HTTP --- Informational} firing near the same timestamps.
\end{itemize}

\paragraph{Recommendations.}
\begin{itemize}
  \item Check source endpoint and file hashes; keep allow-lists.
  \item Keep track of beaconing, abnormal periodicity, or policy breach across sets.
  \item Maintain a timeline for potential forensic escalation (pcap, host telemetry).
\end{itemize}

\subsection{Use Case 3: Post--Incident Summary (Aggregation)}
\paragraph{Story.}
After the incident, the analyst looks at what host\texttt{172.22.2.222} has been doing to see if there are any follow-up detections or behaviors that keep happening.

\subsubsection*{Analyst prompt.}
\begin{lstlisting}[language=,basicstyle=\ttfamily\footnotesize]
Group by count and rule name the alerts triggered by source ip 172.22.2.222
\end{lstlisting}

\paragraph{System actions.}
\begin{itemize}
  \item Selected \texttt{suricata.alert} and generated a pure aggregation DSL with \texttt{"size": 0} and a \texttt{terms} aggregation on \texttt{rule.name.keyword}, filtered by \texttt{source.ip}.
  \item Normalized Elasticsearch aggregation output into a compact summary of rule counts.
\end{itemize}

\paragraph{Outcome.}
\begin{itemize}
  \item \textbf{High volume SSH on unusual port}: investigate legitimacy; verify policies and access controls.
  \item \textbf{exe downloads over HTTP}: verify sources; restrict to trusted repositories.
  \item \textbf{Malware--tagged traffic (e.g., KEYPLUG/Conficker)}: initiate host audit and endpoint scanning.
  \item \textbf{Windows Update activity}: confirm legitimacy; monitor for anomalies.
  \item \textbf{ICMP Destination Unreachable}: check routing or firewall configurations.
\end{itemize}

\paragraph{Recommendations.}
\begin{itemize}
  \item Keep a closer eye on the asset; add specific hunting rules for patterns that happen over and over.
  \item Check the paths for updates and use HTTPS whenever you can.
  \item Do audits and patching on a regular basis; teach users about download rules.
\end{itemize}

\subsection{Final Summary: The Power of the SOC Assistant Pipeline}
\paragraph{Vision.}
The three use cases unfold as an iterative modular pipeline, elevating Security Onion to an AI-enabledSOC assistant. Each stage being both the launch pad as well as the feed-back-loop.

\paragraph{Pipeline (from detection to synthesis).}
\begin{enumerate}
  \item \textbf{Matching}: converts low–information notifications into activity context through querying applicable Zeek/Suricata logs.
  \item \textbf{Correlation}: relates context among datasets and reconstructs the timeline according to robust pivots (\texttt{community\_id}, \texttt{uid}, \texttt{destination.ip}).
  \item \textbf{Aggregation}: condenses activity into signal, surfaces dominant behaviors, and informs post--incident decisions.
\end{enumerate}

\paragraph{Iterative flow.}
Outputs from each stage become inputs for the next: aggregated top rules become new matching anchors; correlation pivots seed fresh timelines; matched hashes bootstrap threat hunting.

\paragraph{Why it matters.}
It is \emph{schema--aware}, \emph{validation--first}, and \emph{operator--centric}. It is modular, scalable, and an auditable platform supporting continuous as well as intelligent defense where the analyst remains in the picture.
