# Architecture Overview

This document describes the implemented architecture of the REST Sun Grass backend as it exists in this repository.

## Contents

- [System context](#system-context)
- [Main components](#main-components)
- [Internal backend structure](#internal-backend-structure)
- [Core backend service class diagram](#core-backend-service-class-diagram)
- [Solar service package structure diagram](#solar-service-package-structure-diagram)
- [API surface](#api-surface)
- [Execution model](#execution-model)
- [Workflow sequence](#workflow-sequence)
- [State and persistence](#state-and-persistence)

---

## System Context

```text
┌───────────────────────────────────────────────────────────────────┐
│ Clients                                                           │
│ - Swagger / curl for REST workflows                               │
└───────────────────────────────┬───────────────────────────────────┘
                                │
                  HTTP          │       
                                │
┌───────────────────────────────▼───────────────────────────────────┐
│ FastAPI backend                                                   │
│ - /auth, /users, /solar, /health                                  │
│ - service layer orchestration                                     │
│ - job/result metadata persistence                                 │
└───────────────┬───────────────────────┬───────────────────────────┘
                │                       │
                │                       │
     ┌──────────▼──────────┐   ┌────────▼────────┐
     │ PostgreSQL          │   │ Valkey          │
     │ users, sessions,    │   │ session-scoped  │
     │ files, jobs,        │   │ workflow cache  │
     │ results             │   └─────────────────┘
     └─────────────────────┘
                │
                │
┌───────────────▼───────────────────────────────────────────────────┐
│ Execution and data dependencies                                   │
│ - Actinia REST API for auth and workspace provisioning            │
│ - GRASS execution inside the Actinia container via docker exec    │
│ - Open-Meteo for optional weather ingestion                       │
│ - Shared filesystem for Parquet artifacts and GRASS workspaces    │
└───────────────────────────────────────────────────────────────────┘
```

## Main Components

| Component | Role |
|-----------|------|
| **FastAPI app** | Exposes authentication, user, terrain, radiation, weather, and batch PV endpoints; wires routers and returns docs pages |
| **Service layer** | Implements the stage-based workflow: terrain import, smoothing, horizon, radiation, Parquet artifact generation, weather preparation, and PV simulation |
| **PostgreSQL** | Persists `users`, `sessions`, `files`, `jobs`, and `results` records |
| **Valkey** | Caches per-session workflow context such as DSM, slope/aspect, horizon outputs, `points_static`, `suitable_areas`, and `area_clearsky_means` paths |
| **Actinia REST API** | Used for credential validation and for ensuring locations/mapsets exist |
| **Actinia container / GRASS** | Runs most geospatial commands through direct in-container GRASS execution |
| **Filesystem artifact store** | Holds Parquet outputs, uploaded weather files, temporary files, and shared data exchanged between pipeline stages |
| **Open-Meteo** | Optional external weather source used to build simulation-ready weather Parquet artifacts |
| **`run_cosim.py` CLI** | Step-based co-simulation entry point for single-area or multi-area runs using session context or manual paths |
 
## Internal Backend Structure

The following diagram shows the main internal backend layers, from API routers to services, core helpers, repositories, and ORM models, together with their relationships to the external runtime dependencies.

![Backend module structure](diagrams/backend-module-structurediagram.png)

*Figure 1: Internal backend module structure, including the main application layers and their connections to Actinia, GRASS/Docker, and PostgreSQL.*

In this structure, incoming requests enter through the API router layer and are delegated to the service layer. The services then rely on core helpers, utility runners, repositories, and ORM models to manage session-aware workflow execution and to interact with Actinia, GRASS/Docker, PostgreSQL, and the filesystem-backed artifact store.

## Core Backend Service Class Diagram

The following diagram provides an implementation-oriented UML-style overview of the backend structure. It shows the main API modules, authentication and user services, grouped solar services, ORM models, repositories, core utilities, and external systems used by the framework.

![Core backend service class diagram](diagrams/backend-core-service-class-diagram.png)

*Figure 2: Core backend service class diagram.*

## Solar Service Package Structure Diagram

The following diagram expands the internal organization of `app.services.solar`. It shows the terrain, radiation, artifact, weather, and photovoltaic service groups and the main dependency flows between them. It also distinguishes the workflow-based photovoltaic branch from the fast horizon-based simulation branch.

![Solar service package structure diagram](diagrams/solar-service-package-structure-diagram.png)

*Figure 3: Solar service package structure diagram.*

## API Surface

The backend is organized into a small number of top-level route groups:

| Prefix | Purpose |
|--------|---------|
| `/health` | Simple application liveness response with app name/version |
| `/redoc` | ReDoc |
| `/docs` | Swagger UI |
| `/auth` | Login, refresh, logout, and current-session identity |
| `/users` | Self-service profile/preferences and admin user operations |
| `/solar` | Stage-oriented terrain, radiation, weather, and batch PV workflow endpoints |


Within `/solar`, the current REST workflow is split into these stages:

| Area | Endpoints |
|------|-----------|
| **Terrain** | `/upload-real-dsm`, `/upload-and-synthetic-dsm`, `/dsm-smooth-and-slope-aspect`, `/horizon`, `/solar-radiation` |
| **Radiation and artifacts** | `/points-static-parquet`, `/suitable-areas-parquet`, `/area-clearsky-means-parquet` |
| **Weather** | `/weather/open-meteo` |
| **Photovoltaic** | `/real-sky-batch`, `/fast-horizon-pv-simulation` |

Co-simulation is not exposed as a REST endpoint. It is implemented as a service/library and surfaced through [`backend/scripts/run_cosim.py`](../backend/scripts/run_cosim.py).

## Execution Model

The backend uses a hybrid integration model with Actinia:

1. **REST calls** are used where Actinia behaves like a remote service.
   Authentication validates credentials against Actinia and workspace setup ensures locations and mapsets exist.
2. **Direct container execution** is used for most GRASS processing.
   Services call an internal runner that executes `docker exec ... grass ... --exec ...` inside the configured Actinia container.
3. **Shared filesystem exchange** is used between pipeline stages.
   Large intermediate and final artifacts are written to disk and referenced later through Valkey and database job metadata rather than passed through HTTP responses or stored in PostgreSQL.

This is an important implementation detail: the backend is not only an HTTP client of Actinia. It also depends on local access to the Actinia container runtime.

## Workflow Sequence

The implemented end-to-end pipeline is best understood as a sequence of stateful session-scoped stages:

```mermaid
flowchart TD
    A[1. Login<br/>create session] --> B1[2a. Upload<br/>real DSM]
    A --> B2[2b. Build<br/>synthetic DSM]
    B1 --> C[3. Smooth DSM<br/>derive slope/aspect]
    B2 --> C
    C --> D[4. Compute<br/>horizon rasters]
    D --> E[5. Run r.sun<br/>radiation]
    E --> F[6. Build Parquet<br/>artifacts]
    F --> G[7. Prepare<br/>weather]
    G --> H[8. Run PV<br/>simulation]
```

### 1. Session and identity

- `POST /auth/login` validates the supplied Actinia credentials.
- The backend synchronizes the local user record, creates a session row, and issues access and refresh JWTs.
- The session id becomes the canonical workflow mapset name: `session_<sid>`.

### 2. Terrain ingestion

The terrain stage starts from two high-level scenarios:

- **Scenario 1: a real DSM already exists**. The user has a real DSM with 1 m resolution or finer and uploads it together with building footprints through [`/solar/upload-real-dsm`]. In this path, the backend imports the DSM into GRASS and the workflow continues with smoothing and slope/aspect generation.
- **Scenario 2: a real DSM does not exist**. The backend must first construct a synthetic DSM through [`/solar/upload-and-synthetic-dsm`]. This synthetic path has two supported input modes: 
  - the uploaded building-footprint layer already contains a building-height column, so the synthetic DSM can be derived directly from vector attributes;
  - the footprint layer does not contain building heights, so the backend first derives building heights from a coarser DSM and DTM, then uses those heights to build the synthetic DSM.

After either path produces the DSM used for the session, the workflow converges on the same downstream stages: DSM smoothing, slope/aspect generation, horizon computation, radiation analysis, and artifact generation.

### 3. Terrain derivatives

- [`/solar/dsm-smooth-and-slope-aspect`] resolves the session DSM context and creates smoothed DSM, slope, and aspect rasters.
- [`/solar/horizon`] computes the directional horizon raster set used by later radiation and PV calculations.

### 4. Radiation

- [`/solar/solar-radiation`] orchestrates `r.sun` analysis over a predefined annual sampling schedule rather than over all 365 days.
- The service records output rasters and persists a result manifest so downstream stages can resolve them consistently.

The radiation workflow uses one representative day for each month:

| Month | Jan | Feb | Mar | Apr | May | Jun | Jul | Aug | Sep | Oct | Nov | Dec |
|-------|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|-----|
| Date | 17th | 16th | 16th | 15th | 15th | 11th | 17th | 16th | 15th | 15th | 14th | 10th |
| DOY | 17 | 47 | 75 | 105 | 135 | 162 | 198 | 228 | 258 | 288 | 318 | 344 |

For each representative day, `r.sun` is evaluated from `04:00` to `22:00` with a 15-minute interval. In the implementation, this gives 73 time steps per sampled day (`4.00, 4.25, 4.50, ..., 22.00`), where each `0.25` hour step is equal to 15 minutes.

### 5. Artifact generation

Radiation outputs are converted into analysis-ready artifacts in three main steps:

- [`/solar/points-static-parquet`] builds `points_static.parquet` plus clear-sky timeseries datasets. On the host side, these are written under `FILE_STORAGE_ROOT/points_clearsky/<location>/<mapset>/`, with the main files stored as `points_static.parquet` and `points_clearsky_timeseries.parquet/`.
- [`/solar/suitable-areas-parquet`] builds or imports suitable roof polygons and writes `suitable_areas.parquet` to `FILE_STORAGE_ROOT/suitable_areas/<location>/<mapset>/suitable_areas.parquet`.
- [`/solar/area-clearsky-means-parquet`] aggregates point-level clear-sky timeseries to area-level datasets and writes them under `FILE_STORAGE_ROOT/area_clearsky_means/<location>/<mapset>/`.

These artifact stages are a core architectural boundary: they convert GRASS-native outputs into Parquet datasets used by later PV workflows.

### 6. Weather preparation

Weather can enter the system in two ways:

- Fetch from Open-Meteo with [`/solar/weather/open-meteo`], which writes a weather Parquet artifact to `FILE_STORAGE_ROOT/weather_openmeteo/<location>/<mapset>/`.
- Upload a weather file directly to the batch PV endpoint, where it is validated and staged on the host under `FILE_STORAGE_ROOT/uploads/<location>/<mapset>/` before batch simulation runs.

### 7. Photovoltaic simulation

Three PV execution modes exist in the codebase:

- **Real-Sky Batch REST mode** via [`/solar/real-sky-batch`], which:
  - resolves session-scoped `suitable_areas` and `area_clearsky_means` inputs,
  - loads uploaded or resolved weather,
  - writes partitioned real-sky results (partitioned by `timestamp_clear`),
  - persists a `jobs` row and associated `results` manifest entries.
  - produces output columns specific to the batch model schema (`timestamp_clear`, `dni_clear`, `dhi_clear`, `ghi_clear`, `k_dir`, `k_diff`, etc.).

- **Fast Horizon PV REST mode** via [`/solar/fast-horizon-pv-simulation`], which:
  - depends on existing horizon-analysis rasters and suitable areas,
  - reuses external PVGIS horizon profiles merged with local horizon data,
  - converts local GRASS `r.horizon` directions and GRASS-derived `aspect_mean` to pvlib/PVGIS azimuth convention before pvlib calculations,
  - performs centroid-based, horizon-aware pvlib simulation directly on each area,
  - writes one timeseries Parquet file per suitable area (not partitioned),
  - persists a `jobs` row and associated `results` manifest entries.
  - produces output columns aligned with the fast-horizon logic (`solar_azimuth`, `solar_elevation`, `horizon_elevation`, `dni_adjusted`, `poa_global`).

- **Co-simulation mode** via [`backend/scripts/run_cosim.py`], which:
  - runs one time step at a time using the batch model logic,
  - supports a single area, a selected set of areas, or all areas,
  - can resolve context from Valkey using `(user_id, session_id)` or from manually supplied paths,
  - returns terminal JSON rather than a persisted REST job.

## State and Persistence

### PostgreSQL

The relational model currently stores:

- **Users**: Actinia-linked identity plus profile/preferences.
- **Sessions**: refresh token state, expiry, revocation, device/IP metadata.
- **Files**: uploaded/staged file metadata.
- **Jobs**: stage execution status and JSON parameters.
- **Results**: normalized result-manifest entries pointing to produced artifacts.

This means PostgreSQL is the source of truth for session lifecycle and job history, but not for large geospatial outputs.

### Valkey

Valkey is used as a short-lived workflow accelerator. The current cache stores stage-specific session context for:

- DSM inputs
- smoothed DSM, slope, and aspect
- horizon output base and raster list
- `points_static.parquet` and its timeseries root
- `suitable_areas.parquet`
- `area_clearsky_means` output root

These entries let later stages resolve prerequisites quickly without scanning all historical jobs first.

### Filesystem Layout

The artifact store is rooted at `FILE_STORAGE_ROOT` and, in the local compose setup, is typically bind-mounted under `data/actinia-data/userdata`.

Important paths used by the current services include:

```text
data/
└── actinia-data/
    ├── grassdb/                      Shared GRASS database roots
    ├── userdata/
    │   ├── suitable_areas/
    │   │   └── <location>/<mapset>/suitable_areas.parquet
    │   ├── area_clearsky_means/
    │   │   └── <location>/<mapset>/...
    │   ├── real_sky_results/
    │   │   └── <location>/<mapset>/... (partitioned by timestamp_clear)
    │   ├── fast_horizon_pv_results/
    │   │   └── <location>/<mapset>/suitable_area_id-<id>.parquet
    │   ├── weather_openmeteo/
    │   │   └── <location>/<mapset>/openmeteo_*.parquet
    │   └── ... additional uploaded or generated workflow artifacts
    └── workspace/
        ├── temp_db/                  Temporary GRASS workspaces
        └── tmp/                      Intermediate staging files
```

Some artifact paths are host-side filesystem paths, while GRASS resources are referenced separately through `grass://<location>/<mapset>/...` entries in result manifests.
