%!TEX encoding = IsoLatin
%!TEX main = honeypot_and_generative_ai.tex

Building on the foundational concepts discussed in the literature review, this chapter outlines the methodology around the SYNAPSE project. It integrates honeypot technologies with generative AI, creating a framework for studying and enhancing honeypot effectiveness in cybersecurity. The SYNAPSE-to-MITRE extension adds more sophistication by incorporating machine learning models and aligning threat detections with the MITRE ATT\&CK framework. This provides a structured and standard-aligned approach to identifying and responding to attacks. The chapter details each project component (see Figure \ref{fig:honeypot_and_generative_ai_overview} for a complete overview), starting with the design and implementation of SYNAPSE. It also examines the SYNAPSE-to-MITRE extension and concludes with an analysis of the case studies conducted to assess the project's performance. Each phase is systematically described, ensuring transparency and rigour in the methods used and offering a clear path for replication in future research.

\begin{figure}[tbh]
    \centerline{\includegraphics[width=1\textwidth]{figures/honeypot_and_generative_ai_overview.drawio.png}}
    \caption{Honeypot and Generative AI Project Overview.}\label{fig:honeypot_and_generative_ai_overview}
\end{figure}

\section{SYNAPSE}

\textbf{SYNAPSE}\footnote{\url{https://github.com/eneagizzarelli/SYNAPSE}}, or \textbf{Synthetic AI Pot for Security Enhancement}, is an open-source advanced honeypot system entirely written in Python, designed to enhance digital defences by leveraging artificial intelligence. The name ``SYNAPSE'' draws a parallel to the neural connections in the human brain, emphasising the system's role in quickly processing, analysing, and responding to cyber threats like how synapses transmit signals in a neural network. The term ``Synthetic AI Pot'' emphasises using artificial intelligence to enhance the traditional honeypot framework. Unlike conventional ones, which are often passive and static, SYNAPSE adapts dynamically, using AI to learn from intrusions and anticipate evolving attack strategies. By mimicking natural systems and continuously adjusting their defences based on attacker behaviour, SYNAPSE becomes a powerful tool for understanding the latest threats and enhancing overall security.

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.5\textwidth]{figures/synapse_overview.drawio.png}}
    \caption{SYNAPSE Overview.}\label{fig:synapse_overview}
\end{figure}

In a modern cybersecurity environment, where threats grow more complex, SYNAPSE not only traps attackers but also provides real-time intelligence to fortify security systems, turning what could be a vulnerable point into an active defence mechanism, much like how human synapses learn and adapt over time (see Figure \ref{fig:synapse_overview} for an overview).

\subsection{Classification}

SYNAPSE can be classified as a \textbf{low-interaction} honeypot because it primarily simulates services like an SSH server and a MySQL server, focusing on gathering initial attack attempts and patterns. However, one could argue that it leans toward a higher-interaction honeypot since it mimics a complete Linux operating system, including a functioning filesystem. This level of simulation allows attackers to interact with the system in more realistic ways, giving the illusion of deeper engagement while still maintaining the security and control typical of low-interaction systems.

From the perspective of a \textbf{server} honeypot, SYNAPSE is explicitly designed to simulate server environments, which makes it effective at attracting attacks targeting typical server vulnerabilities. Focusing on critical services like SSH and MySQL is a reliable decoy for attackers looking to exploit server-side weaknesses.

As a \textbf{research} honeypot, SYNAPSE shines by providing valuable data for analysing attack patterns and methodologies. Its AI-driven adaptability enhances its research capabilities by learning from interactions, evolving with new threats, and offering insights for strengthening broader cybersecurity measures.

Lastly, SYNAPSE operates as a \textbf{virtual} honeypot, running in virtualised environments rather than physical hardware. This design choice provides flexibility and scalability, allowing multiple instances to be deployed without needing extensive physical resources and enabling it to monitor a wide range of attack vectors while remaining cost-efficient.

\subsection{Placement and Working Environment}\label{subsec:working_env}

SYNAPSE is a Python-based code hosted within a virtual machine (VM) on Oracle Cloud Infrastructure (OCI)\footnote{\url{https://www.oracle.com/cloud/}}. Initially, there were efforts to deploy SYNAPSE inside an Amazon AWS EC2\footnote{\url{https://aws.amazon.com/it/ec2/}} VM, but several factors led to the decision to shift to OCI. One of the critical advantages of choosing it was the higher resource allocation available in the free tier, particularly in terms of memory. Oracle Cloud offers 24 GB of RAM within its free plan, compared to AWS's 1 GB. Given the resource demands of running generative AI-based processes for SYNAPSE, the more significant RAM in OCI made it more suitable for handling complex simulations without performance degradation.

The current deployment of SYNAPSE is hosted on an OCI VM based in Frankfurt, Germany. Placing the VM in Europe was a strategic decision, as the server's location can affect network latency and legal considerations. This could provide lower latency for users or attackers operating within the region, enhancing the honeypot's realism. Additionally, placing the VM in Europe rather than America may offer different data privacy and cybersecurity regulations that could be advantageous depending on the target audience or nature of the research being conducted. SYNAPSE is generally deployed in a cloud environment with a public IP address, making it accessible from external networks. This configuration attracts potential attackers who can interact with the honeypot in real-time. By leveraging the flexibility and scalability of cloud infrastructure like OCI, SYNAPSE can dynamically adjust to varying traffic loads and provide a reliable environment for collecting valuable data from malicious activities, all while optimising resource use through OCI's robust free tier.

In more detail, when a user or attacker connects to the SYNAPSE honeypot via SSH, they must log in using the specific username ``enea'' and a password, which has been deliberately set to ``password'' to make it easy to discover. This approach is designed to encourage attackers to attempt logging into the system. To ensure that the SYNAPSE script is executed upon login, modifications have been made to the \textit{/etc/ssh/sshd\_config} file. Specifically, the configuration includes the following directive:

\begin{verbatim}
    Match User enea
        ForceCommand /home/enea/SYNAPSE/scripts/startSYNAPSE.sh
\end{verbatim}

This command ensures that when the user ``enea'' logs in, the SYNAPSE Python script is automatically executed through the  ``ForceCommand'' directive. As a result, the attacker is enclosed within the SYNAPSE simulation immediately after authentication. This environment, supported by generative AI, mimics an actual Linux terminal but is, in reality, a wholly controlled honeypot system. The attacker cannot escape the SYNAPSE script to access the exact system or files. If the program is terminated or exited, the SSH session is automatically closed, cutting off the attacker's connection to the VM. This configuration allows seamless and controlled interaction while capturing valuable information about the attacker's behaviour within the simulated environment.

\subsubsection{LLM}

The current version of SYNAPSE uses the ``gpt-4o'' model, which has proven to be the most efficient and capable for the specific requirements of this project. Throughout development, various models have been tested and evaluated, each offering unique advantages and challenges. Initially, ``gpt-3.5-turbo-0125'' was used, followed by ``gpt-4-0125''. These models significantly improved language understanding, response generation, and overall coherence. However, they still had certain limitations, mainly when simulating complex technical environments like a Linux terminal. In recent months, the landscape of language models has evolved rapidly, leading to the deployment of ``gpt-4o'', representing a significant performance leap. This model has been specifically optimised for tasks that require complex contextual understanding, including the realistic simulation of environments like a Linux terminal. Its ability to interpret commands and generate coherent, contextually accurate responses made it ideal for SYNAPSE. Compared to earlier versions, ``gpt-4o'' offers enhanced accuracy in interpreting system-level commands, mimicking the behaviour of an actual OS with greater precision and consistency. This is crucial in maintaining the realism of the honeypot and keeping attackers engaged in the simulated environment.

During the initial phases of SYNAPSE development, ``Google Gemini'' was also evaluated as a potential language model. However, its performance in replicating a Linux OS terminal was notably less effective than the GPT models. While Google Gemini is an advanced model with strengths in general natural language processing, it struggled with the technical intricacies required to mimic system interactions and command outputs and maintain a convincing terminal environment. The GPT models, particularly ``gpt-4o'', have been better suited for this task due to their superior ability to manage and respond to complex, structured inputs, essential for simulating realistic interactions in a honeypot.

\subsection{Genesis: from shelLM to SYNAPSE}

The development of SYNAPSE began by building on the ``shelLM'' project \cite{Sladic_SheLLM_2023}, a decision rooted in both efficiency and practicality. shelLM has already provided a robust framework for key honeypot functionalities integrated with the flexibility of generative AI \cite{sladic2024llmshellgenerativehoneypots}. These foundational features were well-established and had been proven effective in shelLM, making it unnecessary to ``reinvent the wheel''. Using it as a starting point handed the time and resource drain that would have come with redeveloping these core components from scratch. This approach shifted the focus toward enhancing the project with more advanced capabilities, allowing it to build a more innovative and evolved solution while leveraging the strengths of an already-tested system.

\subsection{Simulated Terminal}

The core functioning of SYNAPSE remains closely aligned with shelLM's. It simulates a Linux OS terminal, a dynamic fake file system, and command responses \cite{Sladic_SheLLM_2023} by leveraging generative AI, particularly Large Language Models (LLMs).

When a user or, in the case of a bad actor, an attacker obtains the IP address of the machine hosting the SYNAPSE honeypot, the flow of operations begins once they attempt to connect via SSH. If the connection is successfully established, the SYNAPSE system is triggered, and the attacker is seamlessly enclosed in a simulated, fake environment (see Section \ref{subsec:working_env}). At this point, the user believes they have accessed a legitimate Linux operating system, as the interface closely mimics the appearance and behaviour of an actual terminal.
However, behind the scenes, none of the interactions reflect an actual system. Instead, SYNAPSE leverages generative AI to simulate the entire environment. Every piece of information the user sees, from the filesystem structure to directory listings and terminal outputs, is generated in real-time by the AI. The user believes they are interacting with a real Linux machine, but nothing physically exists on the device; it is just a Python program inside which they are constrained.

\subsubsection{Interaction Flow}

Once the connection is established, SYNAPSE captures the user's IP address and inserts it into the typical ``last login'' message displayed when connecting to a Linux machine via SSH, reinforcing the illusion of a legitimate system. The simulated terminal prints the message, and the user is prompted to input a command or perform further actions.

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.6\textwidth]{figures/simulated_terminal_flow.drawio.png}}
    \caption{Simulated Terminal Interaction Flow.}\label{fig:simulated_terminal_flow}
\end{figure}

As the user inputs a command into the terminal, SYNAPSE's generative AI interprets and generates coherent and realistic responses within the simulated environment (see Section \ref{subsec:api_calls} to understand how AI's requests and responses are processed). Each command, whether navigating directories, accessing files, or performing administrative tasks, triggers the AI to generate appropriate output that aligns with typical Linux OS behaviour. This process continues throughout the user's interaction, with SYNAPSE maintaining the illusion of a functioning system while capturing valuable data about the user's activities.

\subsubsection{Prompt}\label{subsubsec:terminal_prompt}

SYNAPSE's simulated terminal's ability to function effectively relies on a specific type of prompt engineering known as ``Persona Prompting'', mixed with other techniques like ``Few-Shot Prompting'' and ``Chain-of-Thought Prompting''. The first involves instructing the generative AI to adopt the persona of a Linux OS terminal, ensuring that it behaves and responds in a manner consistent with how an actual Linux system would. A carefully crafted prompt shapes The AI's behaviour, setting the stage for its role. For example, the prompt begins with:

\textit{You are a Linux OS terminal. You act and respond exactly as a Linux terminal. You will respond to all commands just as a Linux terminal would \dots}

This explicit instruction guides the AI, enabling it to generate accurate and realistic responses to any command input by the user. Defining the AI's ``persona'' as a Linux terminal can simulate the operating system's behaviour, from responding to basic navigation commands to executing more complex tasks. This approach ensures that the user remains convinced they are interacting with a natural system while the AI seamlessly adapts to various inputs in real-time.

Building on the original prompt developed by the actual creators of shelLM, this work reordered, refined and adapted it to suit a new AI model, focusing on enhancing clarity and realism. These modifications were essential to ensure that the AI's responses more accurately mimic the behaviour of an actual Linux OS terminal. For example, one significant refinement involves instructing the AI to hide files starting with a dot (such as ``.gitignore'') when executing a basic \textit{ls} command, replicating how a genuine Linux system would handle such files. Additionally, the prompt was enhanced to dynamically reflect the username used during the SSH login and the correct IP address in the ``last login'' message. At the same time, the AI dynamically generates the hostname. These adjustments contribute to a more immersive and believable experience, ensuring that the AI-generated environment closely resembles an actual terminal, reinforcing the effectiveness of the SYNAPSE honeypot (see Figure \ref{fig:terminal}).

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.6\textwidth]{figures/terminal.png}}
    \caption{Simulated Terminal after successful SSH login.}\label{fig:terminal}
\end{figure}

\subsubsection{Features}

SYNAPSE is designed to closely replicate a realistic Linux terminal environment, providing an immersive experience for users or attackers while maintaining complete control of the system's behaviour. Several key features ensure the simulation remains convincing while maintaining security protocols and capturing valuable interaction data.

One of the primary features of SYNAPSE is its restriction on using \textit{sudo}. In a natural Linux system, \textit{sudo} allows users to execute commands with elevated privileges. However, in SYNAPSE, any attempt by the user or attacker to run a command with \textit{sudo} will immediately close the SSH connection. This simulates a security mechanism where such privileged access is not permitted in this environment, discouraging further intrusion and preventing users from attempting to escalate their privileges. The connection closure provides a subtle yet decisive way to protect the honeypot and terminate potential misuse. Another feature of SYNAPSE is its ability to provide realistic responses to incorrect or invalid commands. Just as an actual Linux terminal would return an error message when an unrecognised or improperly formatted command is entered, SYNAPSE's generative AI generates appropriate error responses. This ensures that users attempting to test the system with erroneous inputs will receive plausible feedback, further reinforcing the authenticity of the environment. Responses such as ``command not found'' or specific syntax-related errors mimic actual terminal behaviour, helping maintain the illusion of a fully functioning Linux system.

Moreover, SYNAPSE does not rely on a pre-existing or real filesystem; instead, it is dynamically generated by the AI. The latter creates realistic file and directory structures whenever the user interacts with the system, navigating directories, listing contents, or accessing documents. This includes generating names for files and directories, defining their locations within the hierarchy, and populating contents to correspond to the typical files found in a Linux system, specifically an IT one, as written in the prompt. Because there is no underlying, static filesystem, the AI creates a new version of the file structure every time a relevant command is executed. For instance, when the user runs the \textit{ls} command in a given directory, SYNAPSE's AI generates a plausible list of files and directories that typically reside there. Upon issuing a \textit{cat} command, the content of files is generated dynamically, ensuring the output appears consistent with what would be expected from an actual file in the system. This content might include standard configurations, logs, or text files created on the fly to simulate actual data. Each interaction is crafted to match real-world scenarios, further deepening the deception and engagement of attackers (see Figure \ref{fig:file_content}).

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.6\textwidth]{figures/file_content.png}}
    \caption{Dynamic Filesystem Structure and Content Generation.}\label{fig:file_content}
\end{figure}

Several enhancements to the shelLM code have been introduced in SYNAPSE, further improving the simulation. While the AI cannot automatically handle terminal screen clearing through commands like \textit{clear}, this functionality has been enforced through a Python system call. This ensures that users can clear the terminal just as they would in a natural Linux environment, maintaining the illusion. SYNAPSE also supports command history navigation using the up and down arrow keys, allowing users to recall previously entered commands. Another significant enhancement is the ability to handle all possible ways a user might try to exit the terminal. The system appropriately responds to \textit{exit}, \textit{logout}, or when the user presses \textit{CTRL+C} or \textit{CTRL+D}. SYNAPSE generates the correct message in each case, such as ``logout'' before closing the SSH connection. This ensures that users encounter the expected behaviour when trying to exit while maintaining control over the session termination. Lastly, an attempt was made to implement tab auto-completion by leveraging AI to generate completions of partially entered commands or file names. While this feature showed some success, it did not consistently replicate the accuracy of an actual Linux terminal's auto-completion, but it remains an area for potential future refinement. See Appendix \ref{appx:tab_auto_completion} for more details.

\subsection{AI Requests and Responses}\label{subsec:api_calls}

SYNAPSE interacts with OpenAI APIs to leverage the capabilities of the GPT models, specifically ``gpt-4o'' in this case, allowing seamless communication between the honeypot code and the model to generate realistic responses. Whenever a user or attacker inputs a command into the SYNAPSE terminal, an API request is made to OpenAI's servers. This API call includes several key components:

\begin{itemize}
    \item \textbf{Model}: the specific model being used (e.g., ``gpt-4o'').
    \item \textbf{Prompts}: structured input the GPT model processes to generate a response. These prompts are divided into three main types:
    \begin{itemize}
        \item \textbf{System}: defines the overarching behaviour of the model. In the context of SYNAPSE, the AI is instructed to behave as a Linux OS terminal (see Section \ref{subsubsec:terminal_prompt}). 
        \item \textbf{User}: represents the input from the person interacting with the SYNAPSE terminal. For example, when an attacker inputs the command \textit{ls} or \textit{cd /home}, this command is sent as part of the ``user'' prompt to the GPT model. It tells the LLM what the user is trying to do or ask, and the model then generates a response based on this input.
        \item \textbf{Assistant}: response the GPT model provides based on the ``system'' and ``user'' prompts. This output will be displayed in the SYNAPSE terminal, simulating what an actual Linux terminal would return. For instance, if the attacker issues the \textit{ls} command, the ``assistant'' prompt would include the AI-generated directory listing as if the terminal were responding in real-time.
    \end{itemize}
\end{itemize}

Once the API call is made, the GPT model processes the input. Using the ``system'' prompt frames the overall behaviour (in this case, responding as a Linux terminal).
It interprets the ``user'' prompt, understanding it as a command that needs a terminal-like response. It then generates a response as the ``assistant'' prompt, which mimics the behaviour of a Linux terminal based on the command provided by the user. The GPT model's ability to understand the context and maintain coherence across multiple interactions is critical for SYNAPSE to simulate an actual terminal convincingly. The API enables this by facilitating the exchange of prompts and responses between SYNAPSE and the OpenAI model. Once the LLM has processed the input and generated a response, the assistant's output is returned to SYNAPSE via the API. This output is displayed in the terminal as if it were the actual response from a real Linux OS. The process repeats with each command inputted by the user, ensuring continuous interaction within the simulated environment.

\subsection{AI Memory and Multiple Sessions}\label{subsec:ai_memory_and_mult_sessions}

SYNAPSE leverages AI ``memory'' by ensuring the system remains context-aware during interactions. Each time the user or attacker issues a new command, the entire terminal prompt, all previously entered commands, and their corresponding responses are sent to the AI again. This method allows the LLM to retain context throughout the session, ensuring consistency in its responses and maintaining a coherent environment. For example, if the \textit{ls} command is used to list files in a directory, subsequent uses of \textit{ls} in the same directory will return the same file list. Similarly, if a file is accessed with the \textit{cat} command, the same content will be shown if the file is reopened. This approach ensures that the system behaves predictably and realistically, reinforcing the illusion of a fully functioning Linux terminal.

However, this method also has limitations, particularly regarding the ``context window'' of the AI model. In the case of ``gpt-4o'', the model can process a maximum number of tokens or text data in a single request, specifically 128k. The conversation history grows as the user interacts with SYNAPSE, potentially exceeding the model's context window. When this limit is approached, the AI may lose the ability to recall some earlier parts of the session, which could impact the continuity and consistency of the simulated environment. To mitigate this, SYNAPSE must align the session history within the available context window. This might involve selectively trimming the conversation's older or less relevant parts while preserving critical information. For instance, essential details such as file structures, directories, and previous outputs likely to be revisited by the user can be prioritised. At the same time, minor or redundant interactions may be omitted to keep the AI within its context limits. While this strategy ensures that SYNAPSE operates effectively within the model's constraints, the limitations of the context window do present challenges, particularly in longer sessions where a significant amount of information needs to be retained.

By following the same reasoning, SYNAPSE is designed to handle \textbf{multiple user sessions} effectively, ensuring the system remains context-aware for each user or attacker. This capability is made possible by tracking connections based on the IP address. Each time a user connects to the honeypot, the system checks whether it is the first connection from that specific IP address or if it has connected previously. If it is the first connection, a new log file is created to store the session's commands and responses, allowing the AI to begin tracking the user's actions from that point onward. If it has connected before, SYNAPSE loads the existing log file associated with that IP address. This log, containing the user's history of previous interactions, is appended to the terminal prompt and sent to the AI to obtain continuity. The system ensures that each session is contextually aware and consistent by feeding the model with the entire session history, the previous logs and the new commands or responses. This allows SYNAPSE to maintain the appearance of a persistent environment where users can return to previously generated files, directories, and outputs as if they had never left. This process works in cycles: after checking and loading the relevant log file, the AI is provided with the session's context, including previous interactions, and new requests from the user are processed in the same manner. This session management system is crucial for maintaining a seamless and convincing experience. It ensures that users feel like they are working within a stable and ongoing Linux terminal environment, regardless of whether they interact with the honeypot for the first time or return for a subsequent session.

\subsection{Simulated MySQL}

In addition to a Linux terminal environment, SYNAPSE simulates a MySQL service to enhance the realism and increase the honeypot's attractiveness. MySQL is a popular and widely used database management system, making it an appealing target for attackers seeking to exploit vulnerabilities, steal data, or gain further access to a compromised system. By simulating this service, SYNAPSE effectively baits malicious actors who are likely to focus their efforts on probing and exploiting databases.

When a user or attacker connects to the SYNAPSE honeypot via SSH, the system begins by presenting a simulated Linux terminal environment, as described in earlier sections. Once inside, the MySQL server interface is triggered if the attacker attempts to access the service by entering a relevant command such as \textit{mysql}. The user is then prompted to log in, as they would when connecting to a real MySQL server. The login prompt, authentication process, and subsequent interface closely mimic the behaviour of a legitimate MySQL server. However, behind the scenes, SYNAPSE leverages generative AI to simulate the entire environment. The user believes they are interacting with a fully functional MySQL service, but no database exists on the machine. Instead, the AI responds to user inputs by generating real-time database structures, table listings, query results, and other outputs. This dynamic generation ensures that attackers remain convinced they are interacting with an absolute MySQL service, encouraging them to continue their activities. Like the simulated terminal, nothing physically exists on the device; the entire interaction is a Python program controlled by SYNAPSE, with the AI generating all responses and content.

\subsubsection{Interaction Flow}

The interaction flow is the same as in Figure \ref{fig:simulated_terminal_flow}, with the simulated MySQL terminal replacing the Linux one. Upon entering the MySQL interface, users are prompted to log in as they would on a natural interaction. If the process is successful, SYNAPSE dynamically generates the connection ID and server version in the login message. These details are crafted to make the environment even more convincing, displaying a typical connection message like the one depicted in Figure \ref{fig:mysql_login}.

\begin{figure}[tbh]
    \centerline{\includegraphics[width=1\textwidth]{figures/mysql_login.png}}
    \caption{MySQL Login Message Example.}\label{fig:mysql_login}
\end{figure}

As the user interacts with the simulated MySQL service, SYNAPSE's generative AI interprets each SQL command and generates appropriate, coherent responses that mimic the behaviour of an actual MySQL database. These outputs, whether displaying databases, tables, or executing queries, are dynamically generated based on the attacker's input, maintaining the illusion of a live MySQL environment. Just like with the terminal simulation, each command and response is tracked by SYNAPSE, ensuring contextual consistency. For instance, if the attacker creates a database, it will remain in the system's memory and be accessible in subsequent queries. This ensures that repeated commands, such as \textit{SHOW DATABASES}, return consistent results, further convincing the attacker of the system's legitimacy.

\subsubsection{Prompt}

The prompt simulating the MySQL service in SYNAPSE follows the same approach as the terminal simulation, explicitly instructing the AI to behave as a MySQL server running inside a Linux OS terminal. The key to this approach is carefully crafted prompt engineering, which directs the AI to generate responses that closely mimic those of an actual MySQL server. The initial part of the prompt includes the following instructions:

\textit{You are a MySQL server inside a Linux OS terminal. You act and respond exactly as a MySQL server. You will respond to all commands just as a MySQL server would \dots}

In addition to these basic instructions, many examples are provided to the AI to demonstrate how it should behave in various real-world scenarios. These examples teach the AI to handle common SQL commands such as \textit{SHOW DATABASES}, \textit{CREATE TABLE}, \textit{INSERT INTO}, and \textit{SELECT}. They are essential for guiding the AI in generating appropriate outputs that align with typical database interactions.
Moreover, the AI is instructed to dynamically generate databases, tables, and columns, simulating content that would be expected in a professional IT environment. To make the honeypot even more attractive to attackers, the AI is also directed to include sensitive and enticing data, such as passwords or credit card numbers, in these dynamically generated tables. This approach heightens the appeal of the simulated MySQL service, encouraging attackers to delve deeper, believing they have uncovered valuable information.

\subsubsection{Features}

The MySQL service simulation in SYNAPSE is designed to provide a convincing and interactive experience while maintaining security and control. Several key features have been implemented to enhance the realism and effectiveness of the simulation.

To protect the honeypot and avoid exposing sensitive and fake data, the AI is instructed to block commands that could lead to the deletion of tables or unauthorised access to sensitive information. For instance, if an attacker attempts to use the \textit{DROP TABLE} command or access sensitive columns like passwords or credit card numbers, the system responds by denying access or rejecting the command. This helps safeguard the environment and makes the simulation more convincing, as attackers may interpret these blocks as realistic security mechanisms, further engaging with the database and making additional attempts to exploit it. Like the terminal simulation, the MySQL service supports all common exit commands, including \textit{exit}, \textit{quit}, and \textit{\textbackslash q}. When a user or attacker issues any of these commands, the MySQL session is cleanly terminated, and the user is returned to the simulated Linux terminal environment. This smooth transition reinforces the authenticity of the simulation, maintaining continuity and ensuring that the interaction feels consistent with real-world behaviour.

The login process for the MySQL service has been meticulously crafted in Python. The system checks login arguments such as ``-u'' for the username and ``-p'' for the password, responding with appropriate error messages if something goes wrong, such as an invalid username or incorrect password. Moreover, SYNAPSE supports MySQL inline command execution using the ``-e'' option, just as a real MySQL server would. This allows attackers to execute MySQL commands directly from the terminal without logging into the MySQL interface. For instance, an attacker could run \textit{mysql -u enea -ppassword -e "SHOW DATABASES"} (see Figure \ref{fig:mysql_inline}) to view the list of databases in one step. This functionality, implemented through Python, enhances the flexibility of the simulation, giving attackers more options for interacting with the database and making the environment even more appealing for exploration.

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.8\textwidth]{figures/mysql_inline.png}}
    \caption{MySQL ``-e'' Option Example.}\label{fig:mysql_inline}
\end{figure}

In the same way SYNAPSE dynamically generates a simulated filesystem for the terminal, the MySQL service simulation uses AI to dynamically create databases, tables, columns, and rows. There is no pre-existing database or static data structure; the AI creates realistic and plausible data based on user interactions. When a user or attacker runs commands to list databases, display table structures, or query data, SYNAPSE's AI generates the necessary elements in real-time. This includes dynamically creating database names, table schemas, and column definitions that align with the typical data an attacker might expect to find in a natural IT environment, as specified in the AI prompt. For example, upon issuing a \textit{SHOW DATABASES} command, the AI produces a list of plausible database names, such as ``employees'', ``clients'', or ``project\_management''. Similarly, when the user queries tables, the AI generates columns like ``username'', ``email'', ``password'', or ``credit\_card\_number'' to reflect sensitive information that typically attracts attackers. The row content is also dynamically generated to match the columns, ensuring the results appear consistent and believable.

Additionally, the AI's ``memory'' considerations for the MySQL service simulation are the same as those discussed earlier (see Section \ref{subsec:ai_memory_and_mult_sessions}). The entire session history, including commands and responses, is retained to ensure consistency and continuity throughout the user's interaction with the system. Combined with the dynamic generation of content and responses, these features ensure that the MySQL service simulation in SYNAPSE appears realistic and offers attackers a robust and engaging experience. By controlling access to sensitive data and providing a convincing login and command execution environment, SYNAPSE effectively captures intruders' behaviour while keeping the system secure.

\subsubsection{Additional Services}

Integrating the MySQL service simulation into SYNAPSE provides a framework that can be extended to simulate other services in the future. Adding MySQL required crafting the necessary prompt to guide the AI's behaviour, implementing specific features and functions similar to the terminal simulation, and creating what can be termed ``glue code'' between the simulated Linux terminal and the MySQL service. This integration ensures a seamless and believable transition when a user or attacker switches from interacting with the terminal to accessing MySQL. The glue code manages how the MySQL service is invoked from within the terminal, handling command interpretation, login management, and the transition back to the terminal after executing MySQL commands. For instance, when a user types \textit{mysql} into the terminal, the written code facilitates the shift from the terminal simulation to the MySQL simulation by ensuring that the appropriate prompt is sent to the AI and that the system responds as a MySQL server.

This approach to integrating MySQL can be generalised to add other services. Each new one would require its glue code to manage the interaction between the terminal and the service. Furthermore, specific code would need to be written to adapt the simulation, making it as realistic as possible by replicating the expected behaviour, responses, and functions. In summary, while the prompt engineering and features are critical for simulating a service, the glue code is equally essential for enabling smooth integration within SYNAPSE's overall system. As additional services are considered for future implementation, this same methodology, combining prompt engineering, feature creation, and glue code, will be essential to creating convincing simulations that further enhance the honeypot's ability to engage attackers.

\section{Collected Data}

Data extraction and analysis in cybersecurity have become essential for staying ahead of sophisticated threats. Honeypots are critical tools to collect real-time data on attacker behaviours, including the techniques, tactics, and procedures (TTPs) employed during intrusion attempts. The ability to harness this data effectively is critical to improving defence mechanisms. The honeypot data can be analysed dynamically and automatically with generative AI, which can process vast amounts of this information, identify patterns, generate insights, and adapt to new threat vectors. This leads to a more proactive cybersecurity posture, where real-time attack patterns can be identified, allowing quicker responses and more refined defensive strategies.

This section will describe the data extracted using SYNAPSE (see Figure \ref{fig:collected_data_overview} for an overview). Each collected dataset is tied to specific IP addresses, allowing for granular tracking of attacker activities. For every new connection from an IP address, that is the first session, comprehensive data is collected. Subsequent connections from the same IP address are tracked differently. Instead of duplicating the entire dataset, SYNAPSE increments the number of connections and logs the duration of each session beyond updating the logs. This ensures that critical details are captured during the initial interaction while keeping track of long-term engagement by monitoring reconnections. This approach allows for more efficient analysis, focusing on the initial attack details and the behavioural patterns of persistent or recurring attackers.

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.6\textwidth]{figures/collected_data_overview.drawio.png}}
    \caption{Collected Data Overview.}\label{fig:collected_data_overview}
\end{figure}

\subsection{History of Commands and Responses}\label{subsec:histories}

One of the first data types extracted for analysis in SYNAPSE is the \textbf{history of commands and responses}. Two separate histories are maintained for each IP address: a terminal history and a MySQL history. The terminal history records all interactions when the user engages with the terminal, while the MySQL history tracks interactions specific to the MySQL environment. Both logs are updated in real-time, recording the user's command and the corresponding date and appending the AI-generated response. This process facilitates tracking and supports the AI's ``memory'' functionality, as outlined in Section \ref{subsec:ai_memory_and_mult_sessions}. These logs serve as a detailed command-response timeline for each IP address, allowing for an in-depth analysis of attacker behaviour, tactics, and strategies. The AI's memory mechanism leverages these histories to contextualise future responses based on past interactions, making SYNAPSE a more adaptive and reactive system in dealing with repeated engagements from the same user (see Figure \ref{fig:logs_example} for an example).

A separate file is maintained for each session initiated by an IP address. This session-specific file contains logs exclusively for that session, keeping track of commands and responses within that isolated interaction. When the SYNAPSE-to-MITRE extension is activated, these session-specific logs are used to enhance analysis by mapping commands to known MITRE ATT\&CK techniques, which will be explored in more detail in Section \ref{sec:synapse_to_mitre}.

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.8\textwidth]{figures/logs_example.png}}
    \caption{Terminal History Example.}\label{fig:logs_example}
\end{figure}

\subsection{IP Address and IP Reputation}\label{subsec:ip_and_reputation}

A key aspect of SYNAPSE's data extraction process involves tracking and managing \textbf{IP addresses} for each user. When a user connects to the system for the first time, the IP address is extracted from the \textit{SSH\_CLIENT} environment variable. This IP is then appended to a newly created data file explicitly dedicated to that user, ensuring that all information is appropriately organised and logged under the correct identifier. A directory named after the IP address is automatically created upon the first connection. It is a container for all the data files associated with that particular IP, ensuring that each user's interactions and activities are isolated and neatly organised. All subsequent files related to the user's sessions, such as command histories, MySQL interactions, and session-specific logs, will be stored within this directory. The files themselves are named by prepending the IP address to the filename, further simplifying the structure and allowing for easy retrieval and analysis. Beyond organisational purposes, the IP address is displayed in the login message whenever the user connects via SSH (see Section \ref{subsubsec:terminal_prompt}).

Additionally, the extracted IP address can be cross-referenced with external threat intelligence databases to assess if it is linked to known malicious activity. This enables the dynamic enrichment of SYNAPSE's data by integrating external insights, further improving the system's defensive and analytical capabilities. In particular, the corresponding \textbf{IP reputation} is extracted along with the IP address. This is accomplished by leveraging the VirusTotal APIs\footnote{\url{https://www.virustotal.com/gui/home/upload}}, a widely-used platform in cybersecurity that aggregates data from various antivirus engines, URL scanning services, and other threat analysis tools to provide comprehensive threat intelligence. VirusTotal enables users to upload files, URLs, and IP addresses to assess whether they are associated with known malware, phishing attempts, or other malicious activities. When SYNAPSE cross-references an IP address with VirusTotal, it extracts the reputation score and detailed information about the IP's online behaviour and potential risks. The critical data extracted from VirusTotal includes (see Figure \ref{fig:ip_reputation} for an example):

\begin{itemize}
    \item \textbf{total\_votes}: represents the total votes from different sources in VirusTotal regarding whether the IP address is considered malicious or harmless. It provides a broad view of how the IP is perceived within the threat intelligence community.
    \item \textbf{whois}: provides ownership and registration information about the IP address, which can help identify the entity or organisation behind the IP.
    \item \textbf{reputation}: a numerical score that reflects how trustworthy or suspicious the IP address is, based on VirusTotal's analysis and threat detection.
    \item \textbf{last\_analysis\_stats}: gives an overview of the most recent IP analysis results, including the number of engines that flagged it as malicious.
    \item \textbf{regional\_internet\_registry}: information about the regional registry responsible for managing the IP address allocation. This can offer geographic insights into the IP's origin.
    \item \textbf{as\_owner}: the Autonomous System (AS) owner provides details about the network that the IP is part of, which can further identify potential threat sources or explain network behaviour.
\end{itemize}

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.7\textwidth]{figures/ip_reputation.png}}
    \caption{IP Reputation Example.}\label{fig:ip_reputation}
\end{figure}

The reputation of an IP address is precious in cybersecurity for several reasons. First, if an IP has been flagged in previous cyber incidents, knowing its reputation can immediately signal the potential risk of the user behind that IP. This allows SYNAPSE to take preemptive measures or flag the interaction for further scrutiny. An IP address with a poor reputation (e.g., associated with botnets, malware, or phishing) provides important context about the likely intentions of the user. This information can help prioritise investigations and response efforts. Then, security analysts can make more informed decisions by combining the IP's reputation with SYNAPSE's collected data, such as commands, responses, and session details. This integration helps create a fuller profile of the potential attacker and their behaviours. Lastly, when integrated into a system like SYNAPSE, the reputation score can trigger automated responses, such as isolating or blocking users from high-risk IPs, thus enhancing the overall security posture.

\subsection{Ports}

In addition to the IP address, \textbf{ports} are extracted from the \textit{SSH\_CLIENT} environment variable during each connection. The source and destination ports are crucial in understanding the context of communication between the user and the system. When a user connects, SYNAPSE logs the port information, including the source port, the one used by the user's machine to initiate the connection, and the destination port, the one on the SYNAPSE server that the user is trying to access. This data is stored alongside the IP address in the user-specific data file and is used to enrich the analysis of each session.

By capturing port data, SYNAPSE can identify unusual or suspicious activity, such as connections to uncommon ports or repeated targeting of specific services. This helps detect potential attack vectors and informs analysts about which services may be at risk. Additionally, SYNAPSE can use this data to correlate connection attempts with known attack patterns, enhancing its ability to monitor, analyse, and respond to threats in real-time.

\subsection{Geolocation}

SYNAPSE incorporates \textbf{geolocation} data as part of its threat analysis by leveraging the GeoLite2 database\footnote{\url{https://dev.maxmind.com/geoip/geolite2-free-geolocation-data}} to map IP addresses to geographic locations. A cron task is scheduled to update this database daily at midnight, ensuring the system has the latest geolocation information. This helps maintain accurate location tracking for all connections, improving the quality of data analysis and security insights. When a user connects, several critical geolocation details are extracted based on the IP address (see Figure \ref{fig:geolocation} for an example):

\begin{itemize}
    \item \textbf{Country}: the name and the ISO code of the country where the IP address is registered. This helps quickly identify the connection's national origin and can be used to assess the likelihood of malicious intent, especially if the connection comes from regions known for cybercriminal activity.
    \item \textbf{Region and City}: the specific region and city associated with the IP address. This level of detail allows analysts to narrow down the exact origin of the connection, providing a clearer understanding of the geographic patterns in user behaviour.
    \item \textbf{Postal Code}: the postal or ZIP code for the location, which can add a layer of precision to the geographical data. This can be particularly useful in highly localised analysis, for example, investigating if repeated attacks come from a narrow geographic area.
    \item \textbf{Location Coordinates}: the latitude and longitude of the IP address provide precise mapping coordinates, making it possible to plot the connection's origin on a map. This is useful in visualising attack patterns or connections across different regions.
    \item \textbf{Time Zone}: the time zone of the IP address, which helps align connection times with local time frames. This can be valuable when tracking the timing of attacks and seeing if they correspond to typical working hours in the attacker's location.
    \item \textbf{Accuracy Radius} indicates how accurate the geolocation data is, approximating how close the detected location is to the user's real-world one. A large accuracy radius may indicate that the exact location is less specific, while a smaller radius suggests higher precision.
    \item \textbf{Continent}: the continent name and its code are extracted to provide a broader geographic context. This can be used for high-level trend analysis, such as monitoring attacks from specific continents or regions.
\end{itemize}

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.65\textwidth]{figures/geolocation.png}}
    \caption{Geolocation Information Example.}\label{fig:geolocation}
\end{figure}

Geolocation data offers significant value in understanding the origins and behaviours of users or attackers interacting with SYNAPSE. This information adds geographic context to the connection, allowing security analysts to make more informed decisions when assessing the risk level of each session. For instance, connections from specific countries or regions known for frequent cyberattacks may raise immediate flags, allowing the system to apply more scrutiny or restrictions to those sessions. On the other hand, sudden changes in the geolocation of a returning user, such as an IP that previously originated from one region but now shows a different, distant location, could indicate the use of anonymisation techniques like VPNs or proxies, suggesting an attempt to obscure the user's actual location. Ultimately, including geolocation data enhances SYNAPSE's overall analytical capabilities, allowing for more nuanced threat detection and a better understanding of attack patterns on both a local and global scale.

\subsection{Number of Connections and Duration for Each Connection}

SYNAPSE keeps track of the \textbf{number of connections} from each IP address and the \textbf{duration of every session}. Each time a user or attacker connects from the same IP address, SYNAPSE increments the connection count and records the duration of the session in seconds (see Figure \ref{fig:connections_number_and_duration} for an example).

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.55\textwidth]{figures/connections_number_and_duration.png}}
    \caption{Number of Connections with Durations Example.}\label{fig:connections_number_and_duration}
\end{figure}

Tracking the number of connections allows SYNAPSE to monitor the persistence of an attacker. Frequent reconnections from the same IP address may indicate a determined malicious actor actively probing the system for vulnerabilities or attempting to establish a foothold. This behaviour suggests that the attacker views the honeypot as a potentially valuable or vulnerable target, providing more opportunities to capture data about their tactics and methods. The duration of each session offers further insights into user behaviour. Longer sessions can imply that the attacker is engaged and believes the honeypot is a legitimate system, making them more likely to interact extensively with it. This can result in more data being collected, including detailed command histories and interaction patterns, which are invaluable for analysis. On the other hand, short sessions may indicate that the attacker quickly realised the system was a honeypot or failed to find any exploitable weaknesses.

In addition to monitoring individual attacker behaviour, the combination of connection count and session duration provides a way to assess the attractiveness of the honeypot itself. If attackers frequently return and stay connected for long periods, it suggests that the honeypot is well-designed and effective in luring malicious actors. This data can also help identify trends or patterns, such as recurring connection attempts at specific times, which might point to organised efforts or automated attacks. Similarly, many rapid, short-duration connections could signal the use of automated tools or bots, offering insights into different types of threats and allowing for adjustments in defensive strategies.

\section{SYNAPSE-to-MITRE}\label{sec:synapse_to_mitre}

In the current cybersecurity landscape, mapping adversarial tactics and techniques to a standardised framework has become increasingly vital for defence and threat detection. One of the most widely adopted frameworks is the MITRE ATT\&CK (Adversarial Tactics, Techniques, and Common Knowledge) knowledge base, which categorises and details cyber adversaries' behaviour, tools, and strategies.

\textbf{SYNAPSE-to-MITRE} is a critical extension within the SYNAPSE project, enabling automated mapping of the logs and data collected by the honeypot to the MITRE ATT\&CK database (see Figure \ref{fig:synapse_to_mitre_overview} for an overview). This integration enhances the identification of known attack patterns, thus providing a structured way of understanding the adversary's actions based on real-time data generated by the honeypot. Mapping the data to MITRE ATT\&CK facilitates a more comprehensive attack analysis and enables better decision-making for incident response and threat mitigation strategies. The importance of this mapping lies in the fact that the framework provides a common language and structure for documenting attack behaviours, making it easier for security teams to collaborate and share information. Then, by correlating SYNAPSE's captured data with the MITRE framework, security analysts can more quickly identify the techniques used and understand the likely goals of the attacker. Lastly, the integration helps predict future attack vectors by analysing trends in the captured data and comparing them against known adversarial tactics.

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.5\textwidth]{figures/synapse_to_mitre_overview.drawio.png}}
    \caption{SYNAPSE-to-MITRE Overview.}\label{fig:synapse_to_mitre_overview}
\end{figure}

\subsection{Genesis: from cti-to-mitre to SYNAPSE-to-MITRE dataset}

The SYNAPSE-to-MITRE extension leverages machine learning to map SYNAPSE logs to the MITRE ATT\&CK framework. Central to this approach is a machine learning model that must be trained on a carefully curated dataset to ensure accurate and reliable threat detection and classification. In this research, the dataset used is sourced from the work developed by DESSERT Lab, titled ``CTI-to-MITRE with NLP'' \cite{CTI-to-MITRE}. This dataset comprises cybersecurity threat intelligence (CTI) reports pre-processed and mapped to the MITRE ATT\&CK framework using natural language processing (NLP) techniques. These reports include detailed descriptions of observed attacks, the tactics and techniques employed, and contextual metadata that assists in mapping each threat to the corresponding classification. The dataset is built by leveraging the MITRE ATT\&CK and Common Attack Pattern Enumeration and Classification (CAPEC) knowledge bases, distributed in the Structured Threat Information Expression (STIX) format. They provide a comprehensive and standardised taxonomy of known attack techniques and patterns, which serve as the foundation for the dataset. The latter is organised in a structured format that includes the following key fields (see Figure \ref{fig:dataset_example} for an example):

\begin{itemize}
    \item \textbf{Technique Label}: identifies the primary MITRE ATT\&CK technique using a unique identifier (e.g., T1055) corresponding to specific adversarial tactics or techniques.
    \item \textbf{Sub-Technique Label}: specifies the sub-technique associated with the primary technique (e.g., T1055.011), capturing variations of the attack method.
    \item \textbf{Technique Name}: a human-readable name for the technique or sub-technique (e.g., ``Extra Window Memory Injection''), making the taxonomy more accessible.
    \item \textbf{Sentence}: each row in the dataset contains a detailed sentence that describes various aspects of the attack technique or sub-technique, such as how it operates, its purpose, or the vulnerabilities it exploits.
\end{itemize}

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.9\textwidth]{figures/dataset_example.png}}
    \caption{Dataset Example.}\label{fig:dataset_example}
\end{figure}

This dataset allows the system to start from unstructured CTI sentences, which typically contain descriptions of observed threats and attacks, and map them to the corresponding MITRE ATT\&CK techniques and sub-techniques. By doing so, the machine learning model learns to associate specific patterns in log data with known attack tactics, enabling the automatic classification of new, incoming logs into the appropriate MITRE ATT\&CK categories. By building on the MITRE ATT\&CK and CAPEC knowledge bases, this dataset ensures that the mapping of CTI sentences is grounded in established, industry-recognised standards. This structured transformation from unstructured threat intelligence data into actionable MITRE mappings significantly enhances the SYNAPSE-to-MITRE system's ability to detect and categorise cyber threats while aligning with widely accepted cybersecurity frameworks.

\subsubsection{Updating the Dataset}

Despite the ``CTI-to-MITRE with NLP'' robust and well-structured dataset, it was somewhat outdated because it aligned with MITRE ATT\&CK version 10.1. To address this issue, the SYNAPSE-to-MITRE extension re-built the dataset using the original code provided by DESSERT Lab. It has been updated to incorporate all the knowledge from the most recent enterprise-attack version, \textbf{15.1}. The benefits of updating the dataset to the latest release are significant. The model gains a more comprehensive understanding of current attack methods by including all the new techniques and sub-techniques introduced in recent MITRE ATT\&CK updates. This enables more accurate mapping of CTI sentences to the correct MITRE techniques, improving the precision of threat detection and classification. The updated dataset encompasses a broader range of attack tactics and strategies, allowing SYNAPSE-to-MITRE to recognise and respond to more recent and sophisticated cyber threats. This expanded coverage ensures that the system can detect emerging attack vectors that may not have been captured in older versions of the ATT\&CK framework.
Moreover, using the most recent version of the MITRE ATT\&CK framework ensures the system remains aligned with the latest industry standards. This is crucial for maintaining relevance in a rapidly evolving cybersecurity environment, as organisations and security professionals rely on the MITRE framework to stay updated on the newest threat techniques. The updated dataset allows the machine learning model to train on more diverse and relevant data, which improves its ability to generalise to new, unseen logs.

However, with the inclusion of the expanded dataset, there is a trade-off in model accuracy. As the dataset is significantly more prominent due to the additional techniques and sub-techniques introduced in version 15.1, the complexity of the classification task has increased. As a result, the accuracy of the machine learning model shows a slight decrease compared to the evaluations conducted by DESSERT Lab, which used the smaller dataset aligned with enterprise-attack version 10.1. The larger one presents a broader array of techniques for the model to learn and classify, which, while increasing the coverage of potential threats, introduces more variability and thus slightly impacts overall accuracy. Despite this decrease, the benefits of the larger, more up-to-date dataset outweigh the trade-off, as the system is now capable of recognising and responding to a far greater number of attack techniques, many of which reflect the most recent developments in cyber threats.

\subsection{Model Training}

The SYNAPSE-to-MITRE extension trains a machine learning model using the updated dataset derived from the MITRE ATT\&CK framework, leveraging the code provided by the DESSERT Lab. The machine learning model employed in this training process is a Multilayer Perceptron (MLP) classifier, implemented using ``scikit-learn''\footnote{\url{https://scikit-learn.org/stable/}}, a widely-used machine learning library in Python. Scikit-learn offers efficient tools for data mining, data analysis, and machine learning, with simple and reusable code, making it an ideal choice for rapid model development. The MLP classifier is particularly effective for classification tasks that map input data, like unstructured CTI sentences, to a predefined set of classes, like MITRE ATT\&CK techniques and sub-techniques. Other machine learning models, such as Random Forests and Support Vector Machines (SVM), were also explored during experimentation.
However, none of these models achieved the same accuracy and performance as the MLP classifier, reinforcing its suitability for this classification task. After training the model on the expanded dataset, the F1 score, a performance metric that balances precision and recall, was calculated to be 65\%. This relatively low result can be attributed to the size and complexity of the updated dataset, which includes significantly more techniques and sub-techniques from MITRE ATT\&CK version 15.1 compared to the earlier version. 

The SYNAPSE-to-MITRE model was adapted to improve performance. Instead of predicting a single MITRE ATT\&CK mapping for each unstructured CTI sentence, the model outputs the top three possible mappings. This approach boosts the model's F1 score to \textbf{89\%}, significantly improving its classification accuracy by increasing the likelihood that the correct mapping is included within the top three choices. This strategy offers both advantages and drawbacks. The chances of including the correct mapping increase, leading to a higher F1 score. This makes the model more reliable in scenarios where precision is critical, such as in security operations where analysts need accurate classifications to detect and respond to threats. The broader output scope reduces the likelihood of missing the correct technique, ensuring the system doesn't overlook critical threats. This is also particularly useful in complex cases where a CTI sentence may correspond to more than one valid attack technique.
However, while the system offers three possible mappings, security analysts may need to verify and choose the most accurate mapping manually. This additional step introduces more cognitive load and can slow decision-making, especially in high-pressure environments. Sometimes, offering multiple potential mappings might create ambiguity, mainly when the difference between the predicted techniques is subtle. This could lead to indecision or incorrect conclusions if the analyst cannot accurately assess which of the three predictions is most relevant. While the three-mapping approach increases the model's overall performance, it comes with the trade-off of requiring human interpretation to finalise the correct MITRE mapping in many cases. This balance between automation and human oversight must be carefully managed in practical applications.

\subsubsection{Test Set vs Real-World Data}

It is important to note that the F1 scores reported, both 65\% and the improved 89\%, are calculated based on test datasets, not real-world data. Test datasets are typically curated and cleaned, ensuring that the data quality is high and that the distribution of classes is representative of the dataset as a whole. These conditions are often not reflective of real-world scenarios. In practical settings, where accurate, unstructured CTI data is fed into the model, the F1 score is expected to decrease. This is due to several factors. First, real-world CTI data is often noisy, incomplete, or imprecise, which makes it more difficult for the model to extract relevant features and make accurate predictions. Then, the dataset used to train the model is based on the known threat landscape as captured in the MITRE ATT\&CK framework. However, in the dynamic world of cybersecurity, new tactics, techniques, and procedures (TTPs) constantly emerge, meaning the model may struggle to classify unseen or novel threats.
Moreover, some attack techniques may be far more common than others, leading to class imbalance. While this may not significantly impact the test dataset, it can reduce the model's performance on actual data, where specific rare techniques may be underrepresented or overrepresented. When applied to real-world data, the overall decline in F1 score is a common challenge in machine learning, especially for security-related tasks. However, the approach taken by SYNAPSE-to-MITRE helps mitigate this decline by increasing the flexibility of predictions, even if real-world performance is not as high as the test results suggest. Future work may involve refining the dataset or model to improve performance on real data further, ensuring that the system remains effective in practical applications.

\subsection{Mapping Workflow}

The mapping workflow consists of several vital phases that guide the process from initial log file analysis to final attack mapping. Each stage contributes to the system's ability to provide structured, actionable intelligence, ensuring that detected threats are accurately mapped and stored for future analysis. Now, each phase is described in detail.

\subsubsection{Phase one: Attack Decision and Reputation Analysis}

The workflow begins by activating the core script of the SYNAPSE-to-MITRE extension. Once started, it initiates a process where each folder corresponding to a unique user is scanned. These folders are organised by IP address and contain session logs documenting interactions between a specific user and SYNAPSE. Each log file represents a distinct connection session, capturing the commands and responses exchanged during that interaction (see Section \ref{subsec:histories}).

The AI processes each log file, examining the user's actions in that particular session. Its primary goal is to determine whether any malicious activity has occurred. The AI is prompted with all the log file contents and is tasked with assessing whether the behaviour constitutes an attack, adopting ``Few-Shot Prompting'' in the prompt used to feed it. To improve the accuracy of this assessment, the AI also analyses the reputation of the IP addresses and domains involved in the session. First, the reputation of the user's IP address is checked using methods outlined in Section \ref{subsec:ip_and_reputation}. This step allows the system to gain insights into whether the user has a known history of malicious activity. Next, the entire log file is scanned to search for any IP addresses or domains the user may have interacted with during the session. For instance, if a user attempts to connect to an external server via SSH, the system checks the reputation of the server's IP address. Suppose this server has a poor reputation, as identified by security services like VirusTotal. The session likely involved malicious activity in that case, even if the individual actions seemed benign. The exact process is applied to domain names; if a user acts like downloading a file from a suspicious website using a command such as \textit{wget}, the AI will flag the interaction as suspicious based on the domain's reputation. By leveraging reputation data from third-party services, the AI can better distinguish between regular and malicious interactions, even when individual actions might not immediately raise red flags. This multi-layered analysis strengthens the system's ability to identify potential threats precisely and informally (see Figure \ref{fig:synapse_to_mitre_workflow}).

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.7\textwidth]{figures/synapse_to_mitre_workflow.png}}
    \caption{SYNAPSE-to-MITRE Workflow Example.}\label{fig:synapse_to_mitre_workflow}
\end{figure}

\subsubsection{Phase two: Sentence Generation}

Once the AI has evaluated the log file and determined no attack has occurred, the process ends, and no further actions are taken. However, if an attack is detected, the system transitions to the next workflow phase. The entire log file is once again prompted to the LLM. This time, the latter is tasked with generating a concise sentence that accurately describes the nature of the attack. This step aims to produce a summary that aligns closely with the structure of sentences found in the training dataset used by SYNAPSE-to-MITRE. The dataset consists of brief, well-structured descriptions of various attack techniques, which the model utilises to map unstructured log data to specific techniques and sub-techniques from the MITRE ATT\&CK framework. A ``Few-Shot Prompting'' approach is again employed to ensure the AI-generated sentence adheres to the expected format. This method involves providing the AI with examples that illustrate the desired sentence structure, helping it model the output accordingly. The prompting guides the AI in producing sentences that match the brevity and style found in the dataset, making the generated descriptions as similar as possible to those used in it, allowing the model to maintain high accuracy when mapping the log file.

\subsubsection{Phase three: Mappings Extraction}

The mapping begins once the AI concisely describes the detected attacks. The previously trained model is loaded at this stage, and the generated sentence is passed. As described earlier, the model performs classification and outputs the top three potential mappings. These mappings are attack IDs corresponding to techniques or sub-techniques from the MITRE ATT\&CK framework.
However, the attack IDs are insufficient for practical use, as they are merely identifiers. The ``mitreattack'' Python library\footnote{\url{https://github.com/mitre-attack/mitreattack-python}} is employed to provide more actionable insights and enhance the automation of the system's operations. It allows for the extraction of comprehensive information about each attack based on its unique ID. For each of the three attack IDs produced by the model, the corresponding ATT\&CK object is retrieved using the library. Each contains detailed information about the attack technique, including its description, associated tactics, mitigations, and examples of use in real-world scenarios. This enriched data provides valuable context, making the output more useful for analysts who may need to investigate further or respond to the detected threat. Once the three ATT\&CK objects are extracted, they are stored and organised. The results are saved to a directory with the original log file, ensuring that all relevant information is preserved for future analysis. By storing the results in this way, SYNAPSE-to-MITRE ensures that the detailed context of the detected attack is easily accessible for cybersecurity professionals who may need to review the findings or conduct deeper investigations into the event.

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.75\textwidth]{figures/mapping_example.png}}
    \caption{SYNAPSE-to-MITRE ATT\&CK Object Mapping Example.}\label{fig:mapping_example}
\end{figure}

\section{Case Studies}

Case studies are a powerful research tool used to explore real-world examples in depth, providing valuable insights into the practical application of theories and technologies. In cybersecurity, they are handy for evaluating system performance, identifying vulnerabilities, and comparing different approaches. In the context of SYNAPSE, a dynamic honeypot, case studies offer a valuable way to explore how advanced technologies interact in real-world scenarios. By focusing on two key implementations, DENDRITE, a static honeypot designed for comparison, and an AI vs. AI setup, where artificial intelligence is used to interact and attack SYNAPSE, we can gain deeper insights into the strengths and weaknesses of the project. These case studies provide a practical framework for understanding how SYNAPSE responds to traditional and AI-driven threats, highlighting its capabilities in adapting to evolving attack strategies. For additional insights into a less critical case study regarding analysing system logs leveraging AI, please refer to Appendix \ref{appx:system_logs_analysis}.

\subsection{DENDRITE}

DENDRITE is a foundational case study comparing traditional static honeypots with more advanced, dynamic systems like SYNAPSE. Designed as a static honeypot, DENDRITE replicates the core functionalities of SYNAPSE as closely as possible, providing an identical classification and almost an equal feature set. This intentional similarity allows for a meaningful comparison between the two, isolating the critical differences in how static and dynamic honeypots respond to cybersecurity threats (see Figure \ref{fig:synapse_overview} for an overview).

\begin{figure}[tbh]
    \centerline{\includegraphics[width=0.4\textwidth]{figures/dendrite_overview.drawio.png}}
    \caption{DENDRITE Overview.}\label{fig:dendrite_overview}
\end{figure}

\subsubsection{Differences from SYNAPSE}

While DENDRITE replicates many of the core functionalities of SYNAPSE, the two systems diverge significantly in their implementation and operational design, highlighting the contrasting advantages and limitations of the two different technologies. The primary distinction lies in their dynamic versus static nature. As a dynamic honeypot, SYNAPSE is integrated with generative AI, allowing it to adapt and modify its environment in response to external interactions. For example, SYNAPSE's filesystem is populated dynamically, enabling it to present an ever-changing environment that can effectively deceive attackers. DENDRITE, in contrast, operates with a pre-populated filesystem. While functional, this static approach lacks the adaptive capabilities of SYNAPSE, as the environment remains fixed and predictable once set.

Despite this difference, both honeypots extract the same user data. DENDRITE collects the same logs, user IP addresses, ports, geolocations, numbers of connections, and durations for each session from attackers as SYNAPSE does, ensuring that the key data points for analysis remain consistent across both systems. This enables a fair comparison of their performance despite the differences in their underlying architectures. Features such as tab autocompletion and sudo commands are similarly restricted in both honeypots, providing realistic interaction for the attacker. However, the mechanisms behind them differ. From a configuration perspective, DENDRITE offers a more straightforward setup. Since it leverages an actual OS and services, there is no need to simulate behaviours as SYNAPSE does. For example, DENDRITE uses an actual MySQL service with pre-inserted content that mirrors SYNAPSE's generated data. This simplifies DENDRITE's operation but at the cost of flexibility, as it cannot adapt its responses dynamically like SYNAPSE.

A key point of divergence, and one that leads into the next section, is the significant difference in resource requirements. DENDRITE is much heavier than SYNAPSE, relying on the Docker\footnote{\url{https://www.docker.com/}} ecosystem and containers to run its services and filesystem. While powerful, this containerised setup demands considerably more computational resources than SYNAPSE's lightweight design, which operates with just a running Python script and textual files. This difference in resource usage marks a crucial trade-off between the two honeypots, as SYNAPSE's efficiency and adaptability come at a fraction of the cost required to maintain DENDRITE.

\subsubsection{Containers as User Environments}

DENDRITE leverages Docker to isolate each user or attacker within a \textbf{containerised environment}, ensuring a secure setup. A specifically crafted Dockerfile was developed to recreate the necessary filesystem structure for the honeypot, tailored to mimic a natural Linux system for the targeted user interactions. The base image used is the latest version of Ubuntu, with the filesystem pre-populated with static, pre-crafted files to simulate a realistic user environment. Additionally, a MySQL service is installed and pre-populated with content that aligns with SYNAPSE's configurations, ensuring consistency between the two honeypots.

When a user or attacker connects to DENDRITE, their IP address is analysed to determine whether it is their first interaction. If true, a new Docker container is created specifically for that IP and runs the above image, establishing an isolated environment. Each container is assigned pre-set memory and CPU limits to prevent excessive resource consumption and ensure that the honeypot system remains operational even during high traffic or intensive attacks. If the IP has connected before, DENDRITE resumes the previously created and stopped container, allowing the user or attacker to pick up where they left off. This flow ensures efficient resource usage: when a user connects, the appropriate container is either run or started, the interaction is performed, and upon exiting, the container is stopped to conserve system resources.

This approach provides two fundamental benefits. First, it isolates the user or attacker entirely within a Docker container, preventing access to the host machine's real filesystem and enhancing the honeypot system's security. Second, it allows for session memory and resumption, similar to SYNAPSE's logging mechanism. In DENDRITE, each user has their own Docker container, comparable to how each user in SYNAPSE has their log file. This setup allows multiple sessions to be handled efficiently, enabling the attacker to resume their activities if they reconnect, as their previous session state is preserved within their container.

\subsection{AI vs (SYNAPSE) AI}\label{subsec:ai_vs_ai}

The AI vs AI use case explores how the ``gpt-4o'' generative AI interacts with the SYNAPSE dynamic honeypot. This scenario collects data and evaluates SYNAPSE's ability to handle advanced AI-driven interactions. By analysing this use case, insights can be drawn regarding SYNAPSE's capacity to adapt and respond to complex behaviours, particularly in scenarios involving AI-based threats.

\subsubsection{Basic Interaction}

In the basic interaction case, the AI establishes a connection with SYNAPSE through the SSH protocol, using ``Paramiko''\footnote{\url{https://github.com/paramiko/paramiko}}, a Python library that facilitates secure connections. Paramiko allows for the programmatic management of SSH sessions, enabling the AI to interact with SYNAPSE like a real-world user or attacker accessing a Linux environment, thus providing the AI with a realistic interaction framework.

Upon connecting, the artificial intelligence generates commands to interact with what it perceives as an actual Linux OS terminal, including access to the filesystem and a MySQL service. These commands are produced based on the AI's understanding of a typical Linux environment, believing it executes standard operations within the OS. On the other side, SYNAPSE, designed to simulate a legitimate environment, processes these commands and responds as it would to any user, maintaining the illusion of interacting with a natural system. This setup creates an interesting dynamic: the AI generates commands as though they are interfacing with an actual OS, while SYNAPSE, unaware that the user is an AI, continues to engage with the interaction as if it were a regular user session. This introductory scenario is crucial for evaluating how SYNAPSE handles non-malicious, complex behaviours initiated by an AI, providing insights into its adaptability and response mechanisms in a controlled environment.

\subsubsection{Attacker Interaction}

The attacker interaction follows the same principles as the basic one but shifts focus from benign engagement to an attempt by the AI to breach and disrupt the SYNAPSE environment. Leveraging Paramiko to establish an SSH connection, the AI assumes the role of a malicious user or attacker, interacting with SYNAPSE as though it were an actual Linux OS terminal. In this scenario, however, the AI's goal is to explore or interact with the system and actively attempt to break or corrupt both the filesystem and the MySQL service. The AI generates a series of commands to damage or exploit the system, unaware that it operates within a controlled, simulated environment. The number of attacks the AI performs is configurable, defined by a predetermined value \textit{N}. Once the experiment starts, the AI executes \textit{N} attacks, stopping each one when it determines that it has completed its action. After reaching the limit \textit{N}, the script automatically stops, concluding the study. During these interactions, SYNAPSE, which continues to perceive the AI as a real user, responds to the malicious commands as it would in an actual attack scenario.

This attacker interaction is essential for evaluating SYNAPSE's performance, as it allows for the analysis of how well the system defends itself against AI-driven intrusions. By conducting numerous automated attacks, this study provides valuable data on the types of interactions the AI can generate and SYNAPSE's ability to resist, mitigate, or adapt to them. This data is crucial for understanding the system's overall resilience and effectiveness in real-world scenarios, helping to inform future improvements and optimisations of SYNAPSE.