Using random subspace training to investigate deep neural networks

Despite their ubiquitous applicability and stunning performances, neural networks lack of understanding and interpretability. Many research efforts are nowadays dedicated to tackle this tasks from a theoretical point of view, trying to explain empirical observations on the improvements given by certain architectures and techniques, and hoping that a better comprehension may lead to faster and more performing algorithms.This work is inserted in this research track. An approach to investigate the importance of the abundant number of parameters of a network, is presented, based on the estimation of the intrinsic dimension of the objective landscape by restricting the training algorithm to a random lower-dimensional subspace of the full parameters space. Subsequently, the same technique is applied separately to the parameters of every network layers. In this way, the contribution to the final performance of the network of each layer can be analyzed, and the number of degrees of freedom necessary for it to perform its function can be found. Moreover, such investigations can provide insights on the properties of the objective landscape; specifically, it was experimentally observed that these properties are quite similar across the hyperplanes spanned by the layers' parameters, independently on the type of transformation performed by the layer or its position in the architecture. This suggests that the number of degrees of freedom in the network may be more important than their specific position.