\section{Meteorological Data}
Meteorological data are divided in raw and synthetic data.
Raw meteorological data often comes in a temporal-spaced grid-based format. The density of the grid is a discriminatory feature, reflecting into the precision of mapping phenomena at a local-regional level.\\
Every cell in the grid (vertex of a cell) derives from a \textit{data assimilation} process, which, starting from a set of initial weather conditions as measured by ground stations, satellites, planes, boats etc., allows to project them in a regular format, at a particular point in time, leveraging on systems of physical equations.\\
Meteorological forecasts by means of physical models fall into this category, and are divided in:
\begin{itemize}
    \item Short-term or 15-days forecasts
    \item Medium-term or \textit{sub-seasonal} forecasts: quite a recent concept, they analyze periods covering up to 6 weeks forward in the future
    \item Long-term or \textit{seasonal} forecasts: adopted to get an overview over the next season
\end{itemize}
Dealing with chaotic systems, short-term forecasts are reliable just for few days after the forecasting date. This problem is typically addressed as ``butterfly effect'', where a minor change in the aforementioned initial conditions leads to potentially divergent forecasts in longer time horizons.\\
Synthetic data relies instead on the use of statistical models, typically inferring quantities from raw meteorological measures. For instance, this is the case for most of the forecasts of energy variables (e.g. wind/solar power generation derived respectively from wind speed/solar irradiance measures).\\
Further, synthetic data is often useful for data augmentation purposes, allowing to project data far in the past, where either raw measurements are missing or there was not the possibility to obtain them (e.g., a standard example is represented by renewable power generation when no renewable plant was still existing).\\
A further necessary differentiation refers to the two types of forecast models: \textit{operational} and \textit{ensemble} models. The former is a singular fine-tuned instance, usually representing the best estimation of the forecast. It is a deterministic model, leveraging on the best knowledge we have of the initial conditions, by combining and comparing several different measurement sources. It usually exploits the best resolution, both vertically and spatially, as well as the best physical parametrization of the underlying model. However, initial conditions are likely to differ from the ground truth. For this reason, ensemble models are introduced. A set of initial conditions is obtained by perturbing them with noise, thus allowing to explore the space of errors. Each of these ``members'' (i.e., a singular configuration of the ensemble) is plug into a model, typically on a lower resolution, due to computational constraints. Sometimes the model can be perturbed too, to increase the robustness. As a consequence, an ensemble of forecasts is obtained, which are typically used to get some insights into longer-term trends and patterns. Figure \ref{fig:opens} reports an illustration of the difference between the two categories.
\begin{figure}[!htbp]
    \centering
    \includegraphics[width = .75\textwidth]{imgs/OpEnsModels.png}
    \caption{Operational and Ensemble models in a 15-days forecast scenario. In \textbf{black} operational model, in \textcolor{red}{red} the mean of the ensemble. It is worth to notice the increasing volatility of the members of the ensemble for further forecasted dates}
    \label{fig:opens}
\end{figure}

In terms of data availability, forecasts are mostly proprietary measurements, coming in quite expensive solutions. For the scopes of this project, we only rely to open data, whether they are historical or raw re-analyzed collections.
\subsection{Historical Data: ERA5 and Energy Variables}\label{sec:meteo_data}
Our main data source of reference is Climate Data Store \parencite{climate_data_store}, a web platform exposing proper APIs and allowing to customize the data requests according to the needs. It is a database containing re-analyzed raw data, offering a vast pool of well-noted meteorological models.\\
For the first part of the project, we use the standard \textit{ERA5} dataset on single pressure-levels \parencite{hersbach2020era5, era5}, collecting daily observations from 1979, up to the present.\\
The reference period we choose to examine is the winter-time, restricted to the December-January-February (DJF) months. In this way, we ensure to have similar conditions, avoiding any possible effect of seasonality.\\
Under the geographic point of view, we consider the North-Atlantic/European region \parencite{2011ClDy...36...19C, Falkena2020RevisitingTI, grams, van_der_Wiel_2019}, corresponding to an area of over 40 million km$^2$ (latitude in $[20^{\circ}, 80^{\circ}$], longitude in $[-90^{\circ}, 30^{\circ}]$) and ensuring to capture the entire regional envelope of the weather regimes.
The regular grid of the dataset is composed of $115'921$ points, for a density of $0.25$ lat-lon degrees (about $30$ km). We place high importance on the high density characterizing the dataset, since it allows us to adopt more complex techniques than what the literature exposes.\\
For the quantification part of the dissertation, we take advantage of secondary weather variables coming from ERA-5, such as Temperature, Solar Irradiance, Sea Level Pressure, Wind Speed, as well as raw and synthetic energy data directly obtained from TSO (Transmission System Operator), a network of European operators providing the measurements on their grid (some examples are RTE in France, Terna in Italy). Table \ref{tab:energy_vars} reports the set of variables we take into account.
\begin{table}[!htbp]
\centering
\resizebox{\textwidth}{!}{%
\begin{tabular}{|c|c|c|c|}
\hline
\rowcolor[HTML]{FFDE66} 
\multicolumn{4}{|c|}{\cellcolor[HTML]{FFDE66}\textbf{Energy Variables}}               \\ \hline
\rowcolor[HTML]{FFFFFF} 
Wind Offshore Obs (GW)*      & Wind Onshore Obs (GW)    & Wind Capacity (GW)    & Wind Load Factor    \\ \hline
\rowcolor[HTML]{FFFFFF} 
Solar Photo Obs (GW)        & Solar Thermal Obs (GW)*   & Solar Capacity (GW)   & Solar Load Factor   \\ \hline
\rowcolor[HTML]{FFFFFF} 
Hydro Run of River Obs (GW)* & Hydro Reservoir Obs (GW)* & Hydro Capacity (GW)   & Hydro Load Factor   \\ \hline
\rowcolor[HTML]{FFFFFF} 
\multicolumn{4}{|c|}{\cellcolor[HTML]{FFFFFF}Load}                                    \\ \hline
\rowcolor[HTML]{FFDE66} 
\multicolumn{4}{|c|}{\cellcolor[HTML]{FFDE66}\textbf{Weather Variables}}              \\ \hline
\multicolumn{1}{|l|}{Temperature ($^\circ$C)} & \multicolumn{1}{l|}{Precipitation (mm)} & \multicolumn{2}{l|}{Solar Irradiance (W/m$^2$)} \\ \hline
\multicolumn{2}{|l|}{Wind Speed - Sea Level (m/s)} & \multicolumn{2}{l|}{Wind Speed - 100m (m/s)} \\ \hline
\end{tabular}%
}
\caption{Set of energy variables coming from ERA-5 dataset and TSO\\
*variables not available for all the considered countries}
\label{tab:energy_vars}
\end{table}
\subsection{Weather Regimes - Météo-France}
In modeling weather regimes, a definition of ground truth does not properly exist. This is evident across the numerous studies, where results tend to be slightly misaligned.\\
Weather regimes are phenomena which have been only recently adopted for high-level applications of side sectors, therefore freely available datasets tend to be scarce.\\
Météo-France \parencite{meteofrance} is a national weather agency, which offers us the access to their historical collection of daily weather regimes from 1991 up to the present, as predicted by their ensemble centroid-based model. For our scopes, we decide to plug them into our a-posteriori validation framework, after we empirically validate their reliability.\\ 
As it is discussed in the next sections, we include in our results a probabilistic characterization of the ensemble results, although we specify how this is only determined by a pure frequentist assumption.\\
Figure \ref{fig:MF} reports the centroids of the four weather regimes Météo-France predicts.
\begin{figure}[!htbp]
    \centering
    \includegraphics[width = .65\textwidth]{imgs/MF_centroids.png}
    \caption{Visualization of the four regimes in the North-Atlantic/European zone, as predicted by Météo-France \parencite{meteofrance}. It is possible to identify: AR (top left); NAO+ (top right); NAO- (bottom left); SB (bottom right)}
    \label{fig:MF}
\end{figure}
\subsection{Sub-seasonal Forecasts}\label{sec:s2s}
Sub-seasonal forecasts entail a new perspective in data-driven energy markets. Since large-scale configurations tend to be much more predictable and recurrent, medium-term forecasts represent an opportunity in several application domains, thus having recently led to a major development of sub-seasonal models.\\
Under a meteorological point of view, the stationarity (and thus the predictability) of a regime happens due to events occurring in the Stratosphere, and partially blocking the regime circulation. In our study, we aim at detecting some trends in energy variables by correlating the forecasted weather regimes, with the historical quantification we perform on them.\\
Data for sub-seasonal forecasts are retrieved by the ECMWF (European Centre for Medium-Range Weather Forecasts) \parencite{p17442}, releasing ensemble models' predictions twice a week. Again, in processing this collection we adopt a frequentist perspective to enable a probabilistic modeling on the problem.\\
In their modeling, they adopt a 5-clusters model (four standard clusters, and an additional ``Unknown'' cluster), which we easily reconduct to a canonical 4-clusters assumption, due to reasons which will be clear later on.\\
Figure \ref{fig:ECMWF} offers a snapshot on how sub-seasonal forecast looks like.
\begin{figure}[!htbp]
    \centering
    \includegraphics[width = .65\textwidth]{imgs/ECMWF.png}
    \caption{Visualization of the 45-days sub-seasonal forecast provided by ECMWF \parencite{p17442}}
    \label{fig:ECMWF}
\end{figure}
