In Deep Learning, the choice of numerical optimization algorithms significantly influences training dynamics, convergence behavior and computational efficiency. This thesis presents a systematic and reproducible benchmark of widely adopted first-order optimization methods. The study integrates a theoretical analysis of their mathematical foundations with an extensive empirical evaluation conducted across heterogeneous tasks, using a unified task–dataset–model–metric framework. Particular emphasis is placed on convergence properties, predictive performance, and runtime efficiency under realistic computational constraints. The experimental protocol is deliberately designed to ensure reproducibility in low-budget and time-constrained environments, thereby enhancing the practical relevance of the findings.The results aim to provide structured and context-aware guidance for researchers and practitioners in selecting appropriate optimization strategies for diverse Deep Learning applications.