# TSTAT Aggregation Script

This repository contains a PySpark-based script for aggregating network flow data generated by Tstat.  
The script processes the TCP and UDP log files of a given directory, detects the corresponding services, computes hourly traffic aggregations per client IP, and exports the results to CSV files.

## Overview

The script performs the following steps:

1. Reads all the .out directories that match the argument regex and for each of them it extract these files:
   - `log_tcp_complete.gz`
   - `log_tcp_nocomplete.gz`
   - `log_udp_complete.gz`
2. Identifies network services using regular expressions defined in `services.json`.
3. Aggregates data per client IP (`c_ip`), service, protocol and hour range.
4. Exports the aggregated results as a CSV file in a folder named after the date of the data.

## Environment setup

The project should be executed in a Python virtual environment.

### 1. Create and activate the virtual environment

```bash
python3 -m venv venv_agg
source venv_agg/bin/activate
```

### 2. Install dependencies

Once the virtual environment is activated, install the required Python packages listed in `requirements.txt`:

```bash
pip install -r requirements.txt
```

## Execution

Run the script by providing the input directory glob and the services.json configuration file.
The script accepts the following arguments:
- input_glob – input directory glob containing .out directories
- services_json – path to the services.json file used for hostname → service classification
- --output / -o – optional output directory (default: output)

Example command:

```bash
python aggregate.py file:///share/smartdata/tstat/EUTELSAT/2023/05-May/2023_05_13_*.out ./services.json --output ./output
```

### Command-line help

You can display the full list of options using:

```bash
python aggregate.py --help
```

Example output:

```bash
usage: aggregate.py [-h] [-o OUTPUT] input_glob services_json

Aggregate Tstat logs into hourly user-level summaries.

positional arguments:
  input_glob     Input directory glob (e.g. file:///data/2023_05_13_*.out/)
  services_json  Path to services.json used for hostname → service classification

optional arguments:
  -h, --help     show this help message and exit
  -o, --output   Output directory for aggregated CSV files (default: output)
```

## Output

After execution, the script generates an output directory inside the specified output folder (default: `output/`).  
The directory name corresponds to the date extracted from the input path and contains the aggregated CSV file.

Example output structure:

```
output/
└── 2023_05_13/
    └── aggregation.csv
```

The resulting CSV file contains the following columns:

| Column | Description |
|------|------|
| `c_ip` | Client IP address |
| `hour_bin` | Hour interval of the aggregation (e.g. `13-14`) |
| `service` | Detected application service |
| `protocol` | Transport protocol (`TCP`, `UDP`, or `TCP_NC`) |
| `bytes_down` | Total bytes downloaded by the client |
| `bytes_up` | Total bytes uploaded by the client |
| `flows` | Number of flows observed in the aggregation window |

Each row represents the total traffic aggregated per **hour, client IP, detected service, and protocol**.
