Energy Consumption Prediction

Developing a Custom Attention Mechanism for Multivariate Time Series Forecasting This thesis work focuses on the development of a new attention mechanism to tailor the Transformer architecture to multivariate time series. In particular, the performance of two attention mechanism were compared. The first is the classical attention mechanism commonly used in ’vanilla’ transformers. The second, that we will refer to a ’cross-attention’, is a modification aimed to improve the projection representation in the attention to learn the most critical features for multivariate time series forecasting. Therefore, three mains use case projections Embedding were introduced. The first tested the influence of the embedding dimension in capturing the long terms dependence and long time, however it could have a better reactivities in the early event. The Second concerns the intra-day variations improving the model comprehension of daily cycles, the third capture the long tendances and potentially the weekly schemas. The main challenge is to not dilute the attention of transformer to the most relevant points by increasing the sequences length and up the model complexity. Furthermore, we propose a solution to mitigate the capacity of model to learns to copy patterns in the data rather than learning dynamics of time series, which is one of common time series forecasting issue. This thesis explores a solution based on the state-of the art transformer algorithms model and the comparison of different Attention mechanism characterized by different complexity, computations time and accuracy result in the defined use cases.