Efficient implementation of feedback control-based optimization algorithms

This thesis is a satellite continuation to [4] in which the authors introduce a technique to solve equality constrained minimization problems (such as the training of a Re- current Neural Network) via a modified version of the standard gradient algorithm, generating a fictitious system that is controlled with the Lagrange multipliers of the problem and has the constraints violation as its output. Their work introduces two control strategies. The first one consists in using a PI controller, while the second employs feedback linearization to define the lagrange multipliers vector in such a way that makes possible to directly push the constraint violation towards zero using the derivative of the constraints as input. The PI ap- proach has been thoroughly discussed in [7]. The feedback linearization approach while promising, shows limitations due to its expensiveness in terms of operations needed (in particular, for a matrix inversion), which scales badly when the number of constraints and optimization variables goes up. The aim of this work is to explore and experiment strategies to make the Feedback Linearization Constrained Multipliers Optimization (FLCMO) algorithm faster, via three ways: Smart usage of geometric decompositions to alleviate the complexity of a single step of the algorithm, the adoption of an adaptive stepsize to limit the number of steps needed for the algorithm to converge, and lastly with alternative approaches that use only a small fraction of the training data for a single optimization step. The results of these new algorithms are confronted with the standard version to mea- sure the improvement. The work also explores additional algorithmic approaches that, despite not achieving the performance levels of the more successful ones, have potential for improvement and could be enhanced to reach similar levels of effective- ness. [4] V. Cerone, S. M. Fosson, S. Pirrera, and D. Regruto. A new framework for constrained optimization via feedback control of lagrange multipliers. arXiv.org, 2024. doi: 10.48550/arxiv.2403.12738. [7] A. Re. A dynamical systems theory approach for gradient-free training of recurrent neural networks. Master’s thesis, Università degli studi di Torino, 2024.