Politecnico di Torino (logo)

Using random subspace training to investigate deep neural networks

Lorenzo Pacchiardi

Using random subspace training to investigate deep neural networks.

Rel. Alfredo Braunstein. Politecnico di Torino, Corso di laurea magistrale in Physics Of Complex Systems (Fisica Dei Sistemi Complessi), 2018


Despite their ubiquitous applicability and stunning performances, neural networks lack of understanding and interpretability. Many research efforts are nowadays dedicated to tackle this tasks from a theoretical point of view, trying to explain empirical observations on the improvements given by certain architectures and techniques, and hoping that a better comprehension may lead to faster and more performing algorithms.This work is inserted in this research track. An approach to investigate the importance of the abundant number of parameters of a network, is presented, based on the estimation of the intrinsic dimension of the objective landscape by restricting the training algorithm to a random lower-dimensional subspace of the full parameters space. Subsequently, the same technique is applied separately to the parameters of every network layers. In this way, the contribution to the final performance of the network of each layer can be analyzed, and the number of degrees of freedom necessary for it to perform its function can be found. Moreover, such investigations can provide insights on the properties of the objective landscape; specifically, it was experimentally observed that these properties are quite similar across the hyperplanes spanned by the layers' parameters, independently on the type of transformation performed by the layer or its position in the architecture. This suggests that the number of degrees of freedom in the network may be more important than their specific position.

Relators: Alfredo Braunstein
Academic year: 2017/18
Publication type: Electronic
Number of Pages: 49
Additional Information: Tesi secretata. Full text non presente
Corso di laurea: Corso di laurea magistrale in Physics Of Complex Systems (Fisica Dei Sistemi Complessi)
Classe di laurea: New organization > Master science > LM-44 - MATHEMATICAL MODELLING FOR ENGINEERING
Ente in cotutela: Université de Paris-Sud (Paris XI) (FRANCIA)
Aziende collaboratrici: UNSPECIFIED
URI: http://webthesis.biblio.polito.it/id/eprint/8029
Modify record (reserved for operators) Modify record (reserved for operators)