
This chapter deals with how the traffic data can be sniffed on the server and its underlaying technology. The data collection system is based on three script: \cmd{1clock.py}, \cmd{tcpLife} and \cmd{tcpTracer}. The first one in a manager of the seconds. The two scripts, tcpLife and tcpTracer, take measures about traffic information of TCP connections. In order to understand them, it is necessary an introduction of eBPF.

\section{eBPF}
eBPF, or extended Berkley Packet Filter, is a technology that allows access from the user space to the Linux kernel. This allows the user to execute custom code within the kernel without the need to edit the kernel source code and recompile it, or load kernel modules. The aim is to make them easier to change and avoid long innovation cycle building blocks. 

\begin{wrapfigure}{L}{0.4\textwidth}
    \centering
    \includegraphics[width=0.38\textwidth]{5dataCollector/ebpfLogo.png}
    \caption{eBPF logo.}
    \label{fig:ebpfLOGO}
\end{wrapfigure}

Initially designed for packet filtering, it has been developed in the early 1990s for efficient packet filtering coding and described in a paper by Berkeley, McCanne and Jacobson. The BPF programs were used to accept or reject network packets based on specific criteria. Around the 2014 BPF began to transition to eBPF, redesigning its instruction set to be more efficient on 64 bit machines, adding new data structures, like the eBPF maps that can be accessed both in kernel and user space, facilitating data sharing and enhancing the interactivity. A systemcall bpf() was introduced to allow the user space program to interact with eBPF programs in the kernel. In order to simplify the most common tasks, helper function (or high-level functions) were added. Finally, one of the most important improvements was the addition of a verifier that ensures that the eBPF programs are safe to execute, preventing potential crushes or security breaches. To sum up, the concept of BPF has been extended allowing the writing of custom code to load into the kernel dynamically, which means that the program can interact with a process already running, permitting on-the-fly changes without any reboot \cite{rice2023learning}.

The main uses of eBPF are in the networking, security and monitoring applications. It makes easier the implementation of custom network policies, efficient packet filtering and load balancing. Tools like Cilium use it to provide advanced features for containerised environments. Moreover, security policies can be configured at the kernel level, providing visibility and control over the behaviour of the system. It is possible, in fact, to monitor systemcalls, track process activities and detect anomalous behaviours exploiting its potentiality for security purposes. Concerning the observability, it gives access to various kernel and user space events, providing insights into the system performance and helping in debugging \cite{rice2023learning}.


The programs are written in a restricted subset of C and compiled into bytecode that the kernel can execute. The restriction make sure that the programs are safe, verifiable and efficient. As an example, they can not use floating-point arithmetic, which is not supported in the kernel context, as well as dynamic memory allocation. Moreover, the programs are executed in a controlled environment where they are not allowed to perform dangerous operation that could compromise the stability or security of the kernel. As anticipated, a set of helper function is provided, both to help the user to access the kernel but also to avoid the use of system calls and many standard functions, that may indeed compromise the stability of the kernel. Finally, the program needs to have a deterministic behaviour, meaning that their execution will not interfere with the kernel's operations \cite{rice2023learning}.

Here it is a sample code comparison that illustrates the restrictions:

\begin{lstlisting}[caption=Typical C Program]
#include <stdio.h>
#include <stdlib.h>

int main() {
    int *array = (int *)malloc(10 * sizeof(int));
    for (int i = 0; i < 10; i++) {
        array[i] = i * 2;
    }
    free(array);
    return 0;
}
\end{lstlisting}

\begin{lstlisting}[caption=Equivalent eBPF Program]
#include <uapi/linux/bpf.h>
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, 10);
    __type(key, int);
    __type(value, int);
} my_map SEC(".maps");

SEC("prog")
int my_prog(void *ctx) {
    int key = 0;
    int value = 0;
    for (int i = 0; i < 10; i++) {
        key = i;
        value = i * 2;
        bpf_map_update_elem(&my_map, &key, &value, BPF_ANY);
    }
    return 0;
}

char _license[] SEC("license") = "GPL";
\end{lstlisting}

As noticeable it is avoided the use of dynamic memory allocation and instead of standard data structures like the array, it has been used a BPF map and a \cmd{bpf\_map\_update\_elem} helper function to update the elements in the map.


The programs can be attached to various type of events, such as network packet processing, system calls and tracepoints allowing the eBPF to be involved in many scenarios from monitoring the system performance to enhancing security \cite{rice2023learning}:
\begin{itemize}
    \item Network Events comprehend the XDP (eXpress Data Path) which is attached at the earliest point of the network stack, right after the network driver, used to monitor incoming traffic, TC (Traffic Control) which allows the manipulation of both incoming and outgoing traffic and Socket Operation, that permits the attachment to operations like accept, connect, send and receive allowing inspection and modification of socket-level data.
    \item Through Tracepoints, functions can be attached to specific systemcalls, statically defined, in the kernel code for monitoring and debugging, Kprobes attached to almost any kernel function and Uprobes similar to Kprobes but attached to user-space applications.
    \item Application Events to avoid modifying the application code. For this purpose, USDT (User Statically-Defined Tracing) attaching point are used.
    \item File System events related to operation like open, read, write and close used to monitor file system activities.
    \item Scheduler events to monitor CPU performance, like cache missing, CPU cycles and context switches.
    \item Control Groups events, allowing monitoring inside a cgroup and apply resource limits and policies specific to a group of processes.
    \item Process Events like LSM (Linux Security Modules) Hooks that are attached to security-sensitive points in the kernel, allowing the implementation of security policies.
\end{itemize}

Before attaching the program, it needs to be compiled and loaded into the kernel. To do so it is possible to use tools like \cmd{clang} to compile the eBPF code into bytecode and the \cmd{bpf} system call, libraries like \cmd{libbpf} or tools like \cmd{bpftool} to load the bytecode into the kernel. Once done, it is possible to attach the program to a hook point using the appropriate system calls or helper functions. The eBPF verifier checks the program before loading it into the kernel, ensuring its safety and correctness.

It follows the first code example produced for the project, from which it is possible to capture a fast and easy deployment of an eBPF program :

\begin{lstlisting}[language=Python, caption={Python script for loading and attaching eBPF program}, label={lst:pythonLauncher}]
#!/usr/bin/python3
from bcc import BPF
import socket
import os
from time import sleep
from bcc.utils import printb

interface = "lo"  # Loopback interface
# interface = "eth0"  # Ethernet interface (uncomment to use)
kernel_headers = "/usr/src/linux-headers-6.5.0-21-generic/include/"  # Path to kernel headers

# Load eBPF program from source file
b = BPF(src_file="basicTryCopied.bpf.c", cflags=["-I", kernel_headers])

# Load XDP program
fx = b.load_func("udp_counter", BPF.XDP)

# Attach XDP program to interface
BPF.attach_xdp(interface, fx, 0)

try:
    # Print trace output
    b.trace_print()
except KeyboardInterrupt:
    # On interrupt, retrieve data from BPF map and print
    dist = b.get_table("packet_info_map")
    for k, v in sorted(dist.items(), key=lambda item: item[0].value):
        packet = v
        print("Timestamp : %10d, SRC_IP : %10d, DST_IP : %10d, PROTOCOL : %d" % 
              (k.value, packet.src_ip, packet.dst_ip, packet.protocol))

# Remove XDP program from interface
BPF.remove_xdp(interface, 0)
\end{lstlisting}

\begin{lstlisting}[language=C, caption={eBPF program for UDP packet monitoring}, label={lst:ebpf Program}]
#include <linux/in.h>
#include <linux/udp.h>
#include <linux/ip.h>
#include <linux/if_ether.h>
#include <linux/bpf.h>
#include <bpf_helpers.h>

struct _packet {
    __u32 src_ip;
    __u32 dst_ip;
    __u8 protocol;
    __u64 timestamp;
};

BPF_HASH(packet_info_map, u64, struct _packet);

int udp_counter(struct xdp_md *ctx) {
    void *data = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;

    struct ethhdr *eth = data;
    if ((void *)eth + sizeof(*eth) > data_end)
        return XDP_PASS;

    struct iphdr *iph = data + sizeof(*eth);
    if ((void *)iph + sizeof(*iph) > data_end)
        return XDP_PASS;

    struct _packet packet;
    packet.src_ip = iph->saddr;
    packet.dst_ip = iph->daddr;
    packet.protocol = iph->protocol;

    if (iph->protocol == IPPROTO_UDP) {
        packet.timestamp = bpf_ktime_get_ns();
        packet_info_map.update(&packet.timestamp, &packet);
    }

    return XDP_PASS;
}
\end{lstlisting}

The script \ref{lst:ebpf Program} is an eBPF program that processes the incoming packets, store the destination IP address, the source IP address and the protocol of the packet in a data structure \cmd{\_packet}. Finally, if the packet is a UDP packet, the structure is added into a library with the timestamp as key. It is evident that the standard C structure and functions have been avoided in favour of helper functions like \cmd{BPF\_HASH(...)} and \cmd{bpf\_ktime\_get\_ns()}. 
The script \ref{lst:pythonLauncher} instead, is a smoother and faster alternative to the manual commands to compile, load and attach an eBPF program. If normally, it would have been done like this:

\begin{lstlisting}[language=bash, caption={Shell commands to compile the eBPF program}]
# Compile the eBPF program
clang -O2 -target bpf -c basicTryCopied.bpf.c -o basicTryCopied.bpf.o
\end{lstlisting}

\begin{lstlisting}[language=bash, caption={Shell commands to load and attach the eBPF program}]
# Attach the compiled eBPF object to the specified network interface
ip link set dev eth0 xdp obj basicTryCopied.bpf.o sec udp_counter
\end{lstlisting}

\begin{lstlisting}[language=bash, caption={Shell commands to detach the eBPF program}]
# Detach the eBPF program from the network interface
ip link set dev eth0 xdp off
\end{lstlisting}

\begin{lstlisting}[language=bash, caption={Shell commands to interact with BPF maps}]
# Interact with BPF maps using bpftool
bpftool map dump id <map_id>
\end{lstlisting}

With the BCC (BPF Compiler Collection) library for Python, it is faster.
The program is compiled and loaded with:

\begin{lstlisting}[language=Python, caption={Loading eBPF program with Python}]
b = BPF(src_file="basicTryCopied.bpf.c", cflags=["-I", kernel_headers])
fx = b.load_func("udp_counter", BPF.XDP)
\end{lstlisting}

attached with:

\begin{lstlisting}[language=Python, caption={Attaching eBPF program with Python}]
BPF.attach_xdp(interface, fx, 0)
\end{lstlisting}

and linked to the data structure with:

\begin{lstlisting}[language=Python, caption={Communication between Python script and eBPF program}]
try:
    b.trace_print()
except KeyboardInterrupt:
    dist = b.get_table("packet_info_map")
    for k, v in sorted(dist.items(), key=lambda item: item[0].value):
        packet = v
        print("Timestamp : %10d, SRC_IP : %10d, DST_IP : %10d, PROTOCOL : %d" % 
              (k.value, packet.src_ip, packet.dst_ip, packet.protocol))
\end{lstlisting}

It's now clear that eBPF represents an important resource in how to interact with the Linux Kernel, offering a myriad of possibilities for improving performance, security and observability.

\section{Container environment}
To work with eBPF it is necessary to set up the environment and install fundamental packages and libraries \cite{learning-ebpfGit}.

Since, in this project, the monitoring and collecting phase happens in the Server node, its Dockerfile include the installation of the required packages and libraries to make eBPF work.
The first block of interest is:

\begin{lstlisting}[language=Dockerfile, caption=Packages installation]
# Packages needed for the installation of libbpf and bpftool   
RUN apt-get install -y apt-transport-https ca-certificates curl clang llvm jq && \
apt-get install -y libelf-dev libpcap-dev libbfd-dev binutils-dev build-essential make && \
apt-get install -y linux-tools-common && \
apt-get install -y bpfcc-tools && \
apt-get install -y python3-pip
\end{lstlisting}
To be precise, \cmd{clang} and \cmd{llvm} are mandatory for compiling eBPF programs into bytecode. \cmd{libelf-dev}, \cmd{libpcap-dev} and \cmd{binutils-dev} are required for handling ELF files and packet capture, which are part of many eBPF applications. \cmd{bpfcc-tools} is a collection of tools built on the BCC framework.
The second block deals with the installation of \cmd{libbpf}:

\begin{lstlisting}[language=Dockerfile, caption=libbpf installation]
RUN git clone https://github.com/libbpf/libbpf && \
cd libbpf/src && \
make && \
make install && \
cd ../..
\end{lstlisting}

This library is crucial for the interaction with the eBPF programs, because it provides APIs to load, verify and interact with eBPF bytecode within the kernel.
The third block is about the installation of \cmd{bpftools} that is a library that relies on \cmd{libbpf} and provides more functionality, as an instance a command-line interface.

\begin{lstlisting}[language=Dockerfile, caption=Libbpf installation]
# Bpftool installation
RUN git clone --recurse-submodules https://github.com/libbpf/bpftool.git && \
cd bpftool/src && \
make install && \
cd ../..
\end{lstlisting}

There is one last step to perform. As anticipated in the Virtual Network chapter, in order to make eBPF work, are necessary some headers, that apparently need to be loaded at run time. For this reason, they have been inserted in the entry point script, together with the update of the repositories.

\begin{lstlisting}[language=bash, basicstyle=\ttfamily, keywordstyle=\color{blue}, commentstyle=\color{gray}, caption=Server entrypoint eBPF detail]
# Update the package repositories
apt-get update

# Install the necessary kernel headers for eBPF tools
apt-get install -y linux-headers-$(uname -r)  # needed to use the eBPF tools. Since they are often updated, it's necessary to update the repositories

# Mount the debug filesystem
mount -t debugfs none /sys/kernel/debug
\end{lstlisting}

The last command is meant to mount the debug filesystem that provides access to various information and tools needed by eBPF.

\section{Data Collector}
Rather than implementing a monitor tool from zero, it has been decided to use the same tool, provided by \cmd{bpftool}, used to elaborate the dataset, which are \cmd{tcpLife} and \cmd{tcpTracer}.

BCC provides several tools, for each possible layer. The image \ref{fig:BCCtools} gives a comprehensive vision about all the possible layers and functions.

\begin{figure}[hbt]
    \centering
    \includegraphics[width=1\linewidth]{5dataCollector/bcc_tracing_tools_2019.png}
    \caption{BCC tools (Source: \href{https://github.com/iovisor/bcc/blob/master/images/bcc_tracing_tools_2019.png}{IO Visor Project})}
    \label{fig:BCCtools}
\end{figure}

 As an instance, at the level of the Device Drivers, there are tools for monitoring block devices and network devices.
There is a block related to File System tools and System Call. Of course, the one that it is interesting for the purpose of detecting DoS attack is the whole network block, with more attention to the TCP/UDP blocks. This block offers the following tools:
\begin{itemize}
    \item \textbf{tcptop}: Displays the top active TCP sessions by bandwidth usage. It helps in identifying the most bandwidth-consuming connections.
    \item \textbf{tcplife}: Monitors the lifecycle of TCP connections, providing detailed information about connection duration and resource usage.
    \item \textbf{tcptracer}: Traces TCP connection events, such as connection establishment, data transfer, and termination. It provides a granular view of TCP activities.
    \item \textbf{tcpconnect}: Tracks TCP connection attempts, showing when and where connections are being initiated.
    \item \textbf{tcpaccept}: Monitors incoming TCP connections that are accepted by the server.
    \item \textbf{tcpconnlat}: Measures the latency of TCP connections from initiation to establishment.
    \item \textbf{tcpretrans}: Tracks TCP retransmissions, which can indicate network congestion or packet loss.
    \item \textbf{tcpsubnet}: Focuses on TCP connections within a specific subnet.
    \item \textbf{tcpdrop}: Monitors dropped TCP packets, which can affect network performance.
    \item \textbf{tcpstates}: Observes TCP state changes, helping in diagnosing issues related to TCP state transitions.
\end{itemize}

The tools tcpLife and tcpTracer have been designed to trace and monitor TCP connection, providing insights about their lifecycle and performance. The former, as the name suggest, keep track of the complete lifecycle of TCP connections, from initiation to closure, providing metrics such as the duration of the connection, bytes sent and received, the timestamp, the PID and the command and other information. Here it is an output sample:

\begin{lstlisting}[label=lst:lifeOutput,caption=tcpLife output]
TIME(s)   PID   COMM       LADDR           LPORT RADDR           RPORT TX_KB RX_KB MS
0.000000  22970 socat      10.0.1.20       51838 10.100.0.2      80        0     0 501.50
0.000125  22971 socat      10.100.0.2      80    10.0.1.20       51838     0     0 501.58
1.646757  22973 socat      10.0.1.20       51852 10.100.0.2      80        0     0 501.51
1.646898  22974 socat      10.100.0.2      80    10.0.1.20       51852     0     0 501.61
\end{lstlisting}

By means of these tools, programmers are able to analyse the network performances, monitoring the data transferred in the TCP connections, identifying the bottleneck and optimise the traffic, monitor the security and troubleshoot.
TcpTracer instead, provides information about TCP connection events, in which event is intended as connection establishment, data transfer and termination. It provides insights relative to the IP version used, the belonging command and PID, the type of event (open (C), accept (A), close (X)) and the source and destination IP address and port. Here it is a sample of output: 

\begin{lstlisting}[label=lst:tracerOutput,caption=tcpTracer output]
Tracing TCP established connections. Ctrl-C to end.
TIME(s)  T  PID    COMM             IP SADDR            DADDR            SPORT  DPORT 
0.000    C  4948   Socket Thread    4  10.156.4.134     130.192.55.225   41174  443   
0.673    C  27614  socat            4  10.0.1.20        10.100.0.2       55226  80    
0.000    A  22841  socat            4  10.100.0.2       10.0.1.20        80     55226 
0.501    X  27614  socat            4  10.0.1.20        10.100.0.2       55226  80    
0.001    X  27615  socat            4  10.100.0.2       10.0.1.20        80     55226 
4.228    X  4948   Socket Thread    4  10.156.4.134     130.192.55.225   41174  443   
0.401    X  3547   geoclue          4  10.156.4.134     52.34.56.182     35948  443   
7.338    C  3547   geoclue          4  10.156.4.134     52.34.56.182     50210  443   
\end{lstlisting}

Looking at the output, are evident the similarity with the data set. TcpLife can provide the transferred and received bytes, and the \cmd{MS} column is referred to the duration of the connection, which in the dataset is called \cmd{latency}, while tcpTracer can provide the number of open connection, closed connection. What is missed is a time value referred to the difference of time between two network events. Since the tools offered by \cmd{bpftools} are ready-made programs, it felt more easy and natural, obtain the same version of the program from the library BCC and modify it. The BCC library offers the possibility to compile, load and attach eBPF program straight from python code and as examples offers the \cmd{bpftools} in their python version. This will allow the editing of the tools in order to retrieve what is missing. In this case, it has been possible to add the time variable relative to the difference of time between events. Using the python version of the tools, also offered the possibility to synchronise the tools and record the information by packing them in chunks of a specific interval of time. This will allow the compilation of the average and the variance of each metric and have the same exact feature of the dataset. 

\subsection{TcpLife and tcpTracer: the eBPF section}
\label{par:tools-eBPF}
Before going in more detail, a small analysis of the eBPF tools will be performed. For a matter of simplicity, the analysis will be focused on the ipv4 connections
\subsubsection*{TcpLife}
First, the necessary headers are imported.
Next, some BPF maps are defined, to store timestamps (\cmd{birth}) and process the information related to TCP the connection (\cmd{whoami}). Moreover, the data structure to define the ipv4/v6 data are defined. Finally, to allow the program to communicate outside the result a \cmd{BPF\_PERF\_OUTPUT} is defined.

\begin{lstlisting}[language=C,caption={tcpLife data definition},label={lst:tcpLifedataDefin}]
struct id_t {
    u32 pid;
    char task[TASK_COMM_LEN];
};
BPF_HASH(birth, struct sock *, u64);
BPF_HASH(whoami, struct sock *, struct id_t);
// separate data structs for ipv4 and ipv6
struct ipv4_data_t {
    u64 ts_us;
    u32 pid;
    u32 saddr;
    u32 daddr;
    u64 ports;
    u64 rx_b;
    u64 tx_b;
    u64 span_us;
    char task[TASK_COMM_LEN];
};
BPF_PERF_OUTPUT(ipv4_events);
\end{lstlisting}

The program is defined mainly by the function \cmd{kprobe\_\_tcp\_set\_state}, which has as arguments a context pointer for the BPF program, a pointer to the socket structure and an \cmd{int} variable for the TCP state.
The function is defined in several sections.

In the first section, the process is identified. At first, the PID of the process is captured, together with the local port and the destination port (thanks to the socket structure). The packet received by the network card are filtered based on the port captured by means of macro \cmd{FILTER\_LPORT} and \cmd{FILTER\_DPORT}, and later on the PID with \cmd{FILTER\_PID}. The timestamp of the early state of the packet is captured and updated in the respective map. Finally, the name of the command is saved and updated in the correspondent map.

The second section, is related to the closing phase of the connection and the collection of its information. First of all, the duration of the connection is calculated, and as a consequence the respective entry in the \cmd{birth} map gets deleted. Then, the process is individuated by means of the macro \cmd{FILTER\_PID} and by doing so the socket in charge of it is identified. Finally, the information related to the connection, like the number of transferred and received bytes, are stored.

The last section is about the event submission, which means that the results are formatted in the data structure defined before (\cmd{ipv4\_data\_t} or \cmd{ipv6\_data\_t}). The connection is indeed filtered for the type of IP version and data are set in the correspondent structure. Finally, they are sent as output by means of the BPF structure in charge (\cmd{ipv4\_events} which is a \cmd{BPF\_PERF\_OUTPUT}).  

\begin{lstlisting}[language=C,caption={tcpLife section 3},label={lst:tcpLifesection3}]
[...]

u16 family = sk->__sk_common.skc_family;
FILTER_FAMILY

if (family == AF_INET) {
    struct ipv4_data_t data4 = {};
    data4.span_us = delta_us;
    data4.rx_b = rx_b;
    data4.tx_b = tx_b;
    data4.ts_us = bpf_ktime_get_ns() / 1000;
    data4.saddr = sk->__sk_common.skc_rcv_saddr;
    data4.daddr = sk->__sk_common.skc_daddr;
    data4.pid = pid;
    data4.ports = dport + ((0ULL + lport) << 32);
    if (mep == 0) {
        bpf_get_current_comm(&data4.task, sizeof(data4.task));
    } else {
        bpf_probe_read_kernel(&data4.task, sizeof(data4.task), (void *)mep->task);
    }
    ipv4_events.perf_submit(ctx, &data4, sizeof(data4));

} 
[...]
\end{lstlisting}

Another important aspect of the program, is the way the eBPF program is integrated in the python code. Initially, it is compiled and load by means of the BPF function. In order to allow the communication between with the \cmd{BPF\_PERF\_OUTPUT}, it is necessary to open the communication with the specific function and later an infinite loop is set to activate the transfer of data from the kernel space to the user space. Indeed, \cmd{ipv4\_events} is the \cmd{BPF\_PERF\_OUTPUT}, and \cmd{print\_ipv4\_event} is the function defined to handle the incoming events.

\begin{lstlisting}[language=C,caption={tcpLife Python integeration},label={lst:tcpLifePythonIntegration}]

b = BPF(text=bpf_text)
b["ipv4_events"].open_perf_buffer(print_ipv4_event, page_cnt=64)
while 1:
    try:
        b.perf_buffer_poll()
\end{lstlisting}

\subsubsection*{TcpTracer}
In the same way of tcpLife, at first an analysis of the headers included. It follows some definition: the necessary constant to distinguish the type of event (connect, accept, close); similarly to tcpLife the data structure to capture the information and the corresponded \cmd{BPF\_PERF\_OUTPUT} necessary to export data from kernel space to user space; finally the BPF maps to collect data inside the eBPF program, which are hash maps called \cmd{tuplepid\_ipvX} with key a \cmd{ipvX\_tuple\_t} and value the PID and command name of the current process (in which X goes for 4 or 6 depending on the IP version used). The struct \cmd{ipvX\_tuple\_t} takes note of basic information of the connection (source/destination IP, ports, and network namespace). The last hash map, called \cmd{connectsock}, maps an \cmd{u64} to a socket reference.

\begin{lstlisting}[language=C,caption={tcpTracer data definition},label={lst:tcpTracerdataDefin}]
#define TCP_EVENT_TYPE_CONNECT 1
#define TCP_EVENT_TYPE_ACCEPT  2
#define TCP_EVENT_TYPE_CLOSE   3

struct tcp_ipv4_event_t {
    u64 ts_ns;
    u32 type;
    u32 pid;
    char comm[TASK_COMM_LEN];
    u8 ip;
    u32 saddr;
    u32 daddr;
    u16 sport;
    u16 dport;
    u32 netns;
};
BPF_PERF_OUTPUT(tcp_ipv4_event);

// tcp_set_state doesn't run in the context of the process that initiated the
// connection so we need to store a map TUPLE -> PID to send the right PID on
// the event
struct ipv4_tuple_t {
    u32 saddr;
    u32 daddr;
    u16 sport;
    u16 dport;
    u32 netns;
};

struct pid_comm_t {
    u64 pid;
    char comm[TASK_COMM_LEN];
};

BPF_HASH(tuplepid_ipv4, struct ipv4_tuple_t, struct pid_comm_t);

BPF_HASH(connectsock, u64, struct sock *);
\end{lstlisting}


The function \cmd{trace\_connect\_v4\_entry} traces the entry point of an IPv4 TCP connection and take trace of it by means of the hash map \cmd{connectsock}. In other words, the connection is temporarily added to \cmd{connectsock} until the connection is ultimated.

\begin{lstlisting}[language=C,caption={tcpTracer connect entry},label={lst:tcpTracerconnectentry}]
int trace_connect_v4_entry(struct pt_regs *ctx, struct sock *sk)
{
  if (container_should_be_filtered()) {
    return 0;
  }

  u64 pid = bpf_get_current_pid_tgid();

  ##FILTER_PID##
  
  u16 family = sk->__sk_common.skc_family;
  ##FILTER_FAMILY##


  // stash the sock ptr for lookup on return
  connectsock.update(&pid, &sk);

  return 0;
}
\end{lstlisting}

In order to complete the connection, the socket needs to pass through another state. Once the syscall \cmd{connect} returns 0, the connection is considered open, and it is inserted in the \cmd{tuplepid\_ipv4} map. To do so, the function \cmd{read\_ipv4\_tuple} receives a tuple by reference, reads from the socket the information related to the connection and store them into the structure tuple. This function is used in other sections of the code to store the information related to the event.

\begin{lstlisting}[language=C,caption={tcpTracer connect return and read tuple},label={lst:tcpTracerConnectReturn}]
static int read_ipv4_tuple(struct ipv4_tuple_t *tuple, struct sock *skp)
{
  u32 net_ns_inum = 0;
  u32 saddr = skp->__sk_common.skc_rcv_saddr;
  u32 daddr = skp->__sk_common.skc_daddr;
  struct inet_sock *sockp = (struct inet_sock *)skp;
  u16 sport = sockp->inet_sport;
  u16 dport = skp->__sk_common.skc_dport;
#ifdef CONFIG_NET_NS
  net_ns_inum = skp->__sk_common.skc_net.net->ns.inum;
#endif

  ##FILTER_NETNS##

  tuple->saddr = saddr;
  tuple->daddr = daddr;
  tuple->sport = sport;
  tuple->dport = dport;
  tuple->netns = net_ns_inum;

  // if addresses or ports are 0, ignore
  if (saddr == 0 || daddr == 0 || sport == 0 || dport == 0) {
      return 0;
  }

  return 1;
}

int trace_connect_v4_return(struct pt_regs *ctx)
{
  int ret = PT_REGS_RC(ctx);
  u64 pid = bpf_get_current_pid_tgid();

  struct sock **skpp;
  skpp = connectsock.lookup(&pid);
  if (skpp == 0) {
      return 0;       // missed entry
  }

  connectsock.delete(&pid);

  if (ret != 0) {
      // failed to send SYNC packet, may not have populated
      // socket __sk_common.{skc_rcv_saddr, ...}
      return 0;
  }

  // pull in details
  struct sock *skp = *skpp;
  struct ipv4_tuple_t t = { };
  if (!read_ipv4_tuple(&t, skp)) {
      return 0;
  }

  struct pid_comm_t p = { };
  p.pid = pid;
  bpf_get_current_comm(&p.comm, sizeof(p.comm));

  tuplepid_ipv4.update(&t, &p);

  return 0;
}

\end{lstlisting}
The function \ref{tcpTracerCountConnection} is the actual function, in charge of defining the event as connect. When the state of the function is not established or closed, the data of the connection are saved into struct variable \cmd{tcp\_ipv4\_event\_t} and finally exported by means of the usual \cmd{BPF\_PERF\_OUTPUT}. Finally, the connection is deleted from the tuples' map. 

\begin{lstlisting}[language=C,caption={tcpTracer count connection},label={lst:tcpTracerCountConnection}]

int trace_tcp_set_state_entry(struct pt_regs *ctx, struct sock *skp, int state)
{
  if (state != TCP_ESTABLISHED && state != TCP_CLOSE) {
      return 0;
  }

  u16 family = skp->__sk_common.skc_family;
  ##FILTER_FAMILY##
  
  u8 ipver = 0;
  if (check_family(skp, AF_INET)) {
      ipver = 4;
      struct ipv4_tuple_t t = { };
      if (!read_ipv4_tuple(&t, skp)) {
          return 0;
      }

      if (state == TCP_CLOSE) {
          tuplepid_ipv4.delete(&t);
          return 0;
      }

      struct pid_comm_t *p;
      p = tuplepid_ipv4.lookup(&t);
      if (p == 0) {
          return 0;       // missed entry
      }

      struct tcp_ipv4_event_t evt4 = { };
      evt4.ts_ns = bpf_ktime_get_ns();
      evt4.type = TCP_EVENT_TYPE_CONNECT;
      evt4.pid = p->pid >> 32;
      evt4.ip = ipver;
      evt4.saddr = t.saddr;
      evt4.daddr = t.daddr;
      evt4.sport = ntohs(t.sport);
      evt4.dport = ntohs(t.dport);
      evt4.netns = t.netns;

      int i;
      for (i = 0; i < TASK_COMM_LEN; i++) {
          evt4.comm[i] = p->comm[i];
      }

      tcp_ipv4_event.perf_submit(ctx, &evt4, sizeof(evt4));
      tuplepid_ipv4.delete(&t);
  } else if [...ipv6Version...]
  return 0;
}
\end{lstlisting}

The manager of the events \cmd{accept} and \cmd{close} are simpler, because they are articulated in just one function. In both cases, in fact there is no need to pull the socket reference or tuple from any map, because it is related to connections already existing. In fact, when one of the connection goes through the correspondent state, which is whatever for the accept event and whichever except \cmd{oldstate == TCP\_SYN\_SENT || oldstate == TCP\_SYN\_RECV || oldstate == TCP\_NEW\_SYN\_RECV}, the event is set as accept or close. The reason why it is not waited for the \cmd{TCP\_CLOSE} state, is because after that state there is no trace of the socket and because after the connection has been opened (which are the cases of the state aforementioned) it is given that the connection will be closed.
Differently from the tcpLife, tcpTracer needs to be attached to a function by means of the \cmd{attach\_kprobe} function.

\begin{lstlisting}[language=C,caption={tcpTracer attachment points},label={lst:tcpTracerAttachmentPoints}]

if args.ipv4:
    b.attach_kprobe(event="tcp_v4_connect", fn_name="trace_connect_v4_entry")
    b.attach_kretprobe(event="tcp_v4_connect", fn_name="trace_connect_v4_return")
b.attach_kprobe(event="tcp_set_state", fn_name="trace_tcp_set_state_entry")
b.attach_kprobe(event="tcp_close", fn_name="trace_close_entry")
b.attach_kretprobe(event="inet_csk_accept", fn_name="trace_accept_return")
\end{lstlisting}

A synthetic and summarising diagram of the tool tcpTracer is show in the figure \ref{fig:tcpTracerMech}.

\begin{figure}[htb]
    \centering
    \includegraphics[trim=0in 7.5in 0in 0in, clip, width=0.95\textwidth]{5dataCollector/tcpTracer.pdf}
    \caption{TcpTracer mechanism}
    \label{fig:tcpTracerMech}
\end{figure}

%https://eunomia.dev/tutorials/13-tcpconnlat/#analysis-of-tcp_v4_connect-function
%https://elixir.bootlin.com/linux/latest/source/net/ipv4/tcp_ipv4.c#L1169 FUNCTION TO attach to count half open connection

\subsection{TcpLife and tcpTracer synchronisation}
\label{par:synchron}
Once understood how the tools work, it is the time to modify them to obtain the metrics required. The easy challenge is to change the representation of time in tcpTracer in order to obtain the difference of time between events, which is the attribute present in the dataset. As explained in the last paragraph, once data are ready, they are sent to the structure \cmd{BPF\_PERF\_OUTPUT} side eBPF and by means of the \cmd{open\_perf\_buffer} function they are linked to an output manager function. In this context, the time difference is managed by means of python code, by simply computing the time as the current time minus the time of the previous events \cmd{tdif1 = ((event.ts\_ns - start\_ts) / 1000000000.0 ) - tbef1; tbef1 = ((event.ts\_ns - start\_ts) / 1000000000.0 )}. In case of the first event \cmd{start\_ts} is set as 0.

\begin{lstlisting}[language=C,caption={tcpTracer section1},label={lst:tcpTracerSection1}]

b["tcp_ipv4_event"].open_perf_buffer(print_ipv4_event)

def print_ipv4_event(cpu, data, size):
    event = b["tcp_ipv4_event"].event(data)
    global start_ts

    #added by Girasolo
    global tdif1                                                            # Used to calculate the difference of time between a open/accept/close event and the next
    global tbef1
    [...]
    elif args.socketconnection:                                              # Print on socket channel
        global client_socket
        message = ""
        if args.timestamp:
            if start_ts == 0:
                start_ts = event.ts_ns
                #added by Girasolo
                tdif1 = 0
                tbef1 = (event.ts_ns - start_ts) / 1000000000.0
            else:
                tdif1 = ((event.ts_ns - start_ts) / 1000000000.0 ) - tbef1
                tbef1 = ((event.ts_ns - start_ts) / 1000000000.0 )
    [...]

\end{lstlisting}

The bigger problem to solve, was the synchronisation of the two script, tcpLife and tcpTracer. To do so, it has been developed a manager script called 1clock. The script is divided in several sections: the receiving part, the timing and synchronisation part and the log part. The definition of variable and function will be treated together with its part.
The first section to be analysed is the receiving section. The receiving block is composed of the launching of the scripts, the establishing of a connection to communicate with them and the receiving function.
To run the processes has been opted for the function \cmd{life = subprocess.Popen(['python3', 'tcplife.py', '-sc'])}. As noticeable the script tcpLife, but the same is for tcpTracer, are run with the argument \cmd{-sc}. Originally, both the script, did not have this option. It was indeed added to allow the communication with the manager. To do so it has been chosen, as inter-process communication method, the communication through socket.  
The manager opens the communication on a fixed port on the local host. The scripts, instead, have been modified adding the argument \cmd{-sc} by means of the function \cmd{parser.add\_argument("-sc","--socketconnection", action='store\_true', [...])}. Once the processes are started, they connect to the socket.
Afterwards, the connections are accepted by the manager with \cmd{client\_socketLife, client\_addressLife = server\_socketLife.accept()}. Once, the system is ready to start a receiver thread for each process is run.
    
\begin{lstlisting}[language=C,caption={1clock receiving section},label={lst:1clockReveiving}]
def tcplife_receiver(client_socket):
    global mean, variance                                                       # Global so that they can comunicate the result to the thread resposible of the sending
    lifedata = []                                                               # List of tuple (tx_kb, rx_kb, ms)
    print("life thread works")
    try:
        while True:
                # Receive data from the client
                data = receive_line(client_socket)                              # Receive a line from the tcplife socket
                if not data:
                    break
                data = data.decode()
                if data == "SIGNAL HERE ---":                                   # If this string is received it means that tcplife received the sigusr1 after the 25 secs
                    if lifedata != []:
                        mean, variance = compute_mean_variance(lifedata)        # Compute mean and variance of (tx_kb, rx_kb, ms)
                    barrier_result.wait()                                       # Barrier1: communicate that the job is done
                    barrier_result.wait()                                       # Barrier2: wait for the data to be sent to fluentd
                    lifedata = []                                               # Get ready for the next interval
                    mean, variance = [(0,0,0),(0,0,0)]
                else:
                    data = data.strip().split()                                 # Collect data from tcplife
                    rx_kb = float(data[7])
                    ms = float(data[8])
                    lifedata.append((tx_kb, rx_kb, ms))
    finally:
        client_socket.close()                                                   # If something goes wrong or the thread is closed, close the socket

def tcptracer_receiver(client_socket):
    global mean_time, variance_time, open_connections, closed_connections       # Global so that they can comunicate the result to the thread resposible of the sending
    tracerTimes = []                                                            # List of difference of time
    print("tracer thread works")
    try:
        while True:            
            data = receive_line(client_socket)                                  # Receive data from the tcptracer socket
            #print("TRACER\tReceived:", data.decode())
            if not data:
                break
            data = data.decode()
            if data == "SIGNAL HERE ---":                                       # If this string is received it means that tcptracer received the sigusr1 after the 25 secs
                if tracerTimes != []:
                    mean_time = np.mean(tracerTimes)                            # Compute mean and variance of difference of time
                    variance_time = np.var(tracerTimes)
                barrier_result.wait()                                           # Barrier1: communicate that the job is done
                barrier_result.wait()                                           # Barrier2: wait for the data to be sent to fluentd
                tracerTimes = []
                open_connections = 0
                closed_connections = 0
                mean_time ,variance_time = 0,0
            else:
                data = data.strip().split()                                     # Collect data
                if data[1] == 'C':
                    open_connections += 1
                elif data[1] == 'X':
                    closed_connections += 1
                tracerTimes.append(float(data[0]))
    finally:
        client_socket.close()                                                   # If something goes wrong or the thread is closed, close the socket

\end{lstlisting}

The tcpLife thread receiver is a loop that receive a line with a custom function and append the data received in a list called \cmd{lifedata}, in order to compute the mean and the variance of the time interval required. TcpTracer does the same except the variables \cmd{open\_connection} and \cmd{closed\_connection} which are counters, so they just need to be set equal to zero when the time interval expires. To allow the communication with the main program or with other threads, the global variables have been chosen as solution. 

The next phase to analyse is the log part. In order to send the log to Fluentd located in the admin node, it has been used a custom thread called \cmd{sendResult} and an object called \cmd{sender} from the library \cmd{fluent}. The object sender has a creator called FluentSender that allows the creation of a logger with a specific tag and that send logs to a specific host in a specific port. The sending thread is a loop that set the variable in a structure and send them to the log collector using the function \cmd{emit()} in which another tag can be added to the message.

\begin{lstlisting}[language=C,caption={1clock sending thread},label={lst:1clockSendig}]
def sendResult():
    """
    Send the dos.log entry to fluentd
    """
    global mean, variance, open_connections, closed_connections, mean_time, variance_time, logger           # Global variable, so that they get modified by other thread and sent here to fluentd 
    while True:
        barrier_result.wait()                                                                               # Barrier1: waits for tcplife and tcptrace to send their data
        #mediaBT	mediaRX	varBT	varRX	medialat	varlat	Open	Close	mediadift	varDifT	Attack
        dataToSend = {
                'final result' : socket.gethostbyname(socket.gethostname()),
                'tx_mean' : mean[0],
                'rx_mean' : mean[1],
                "tx_var" : variance[0],
                "rx_var" : variance[1],
                "ms_mean" : mean[2],
                "ms_var" : variance[2],
                'open_connections' : open_connections,
                'closed_connections' : closed_connections,
                "mean_time" : mean_time,
                "variance_time" : variance_time
                }                                                                                           # Collect data in a library
                # Send data to Fluentd
        print(dataToSend)
        logger.emit('tracer.logs', dataToSend)                                                              # Send it with the right tag
        barrier_result.wait()                                                                               # Barrier2: Tell tcplife and tcptrace that can go to the next set of data
\end{lstlisting}

As noticeable in the sending part, a barrier called \cmd{barrie\_result} is used. Its purpose is of synchronisation with the receiving thread. Let's get now in the synchronisation phase. To synchronise the three thread has been defined a barrier with a counter of 3, which means that to get it unlocked all the three thread need to execute the function \cmd{barrier\_result.wait()}. As noticeable, the same barrier has been positioned twice in the \ref{lst:1clockSendig} and twice in the \ref{lst:1clockReveiving}. The first barrier makes the sending thread to wait for the thread receivers to obtain the data and compute the requested mean and variance. The second one, instead, makes the receivers to wait for the sending thread to transmit the data, in order to be able to use the global variables again.
As anticipated, it is also required the synchronisation of the monitor tools together with the receiver, for the aim of computing the mean and variance of several features according to a specific time interval, which has been set as 25 seconds. To do so, two other threads, that work as a couple, are launched by the manager script: \cmd{signal\_thread = threading.Thread(target=manageEvent, args=[event])} and \cmd{signal\_process = mp.Process(target=send\_signal, args=([p.pid for p in subprocesses], event))}. The first one is a loop that waits for an event to be set, by the second one. Once it has been set, it sends a SIGUSR1 signal to the process tcpLife and tcpTrace. Finally, unset the event with the function \cmd{event.clear()}. The second thread, instead, is the real clock of the manager. This loop, in fact, waits for 25 seconds and then set the event as true, for the purpose of unlock the SIGUSR1 signal sender. This is done in order to keep the clock of the time interval in the manager and synchronise the processes with it.    

From the processes side, they handle the signals in similar way. When a SIGTERM is received, they terminate the process and close the connection. When they receive the SIGUSR1 signal, they send a special message to the manager containing the string "SIGNAL HERE ---". When the manager receives this string, it starts the procedure of computation of mean and variance (snippet \ref{lst:1clockReveiving}). 

At this point, it is just necessary to launch the script \cmd{1clock} to collect and send to the Fluentd node all the required data. 

\begin{figure}
    \centerline{\includegraphics[width=0.8\textwidth]{5dataCollector/dataCollectorSummary.pdf}}
    \caption{Data collector diagram}
    \label{fig:dataCollectorDiagram}
\end{figure}