In this thesis, we investigate the incorporation of causality into machine learning models, particularly focusing on large dimensional data. The work starts with an exploration of linear regression within machine learning, highlighting its connections with linear response theory, a field of statistical physics. We derive a closed-form solution for the empirical response function, accounting for bias and variance. The study extends to high-dimensional data, addressing the complexities of identifying causal relationships and the curse of dimensionality. Numerical simulations show that observational causality can be inferred in high-dimensional contexts with linear dependencies among variables. This thesis bridges principles from statistical mechanics and machine learning, offering new perspectives on causal relationships in complex systems.