Plasticity across neural hierarchies in artificial neural network

Deep neural networks can extract a hierarchy of relevant features from the data that can be used both for classification and generation task, reaching state-of-the-art performance in object/speech recognition and language translation. However, many characteristics of the way these networks process the information or, more in general, the reason why they work so well are still unclear. In this work we analyze some features of a deep belief network during training across different layers in an unsupervised setting. First of all we study how the plasticity varies across the network’s layers, computing the variation of the architecture’s weights when the dataset to be learned is changed. We observe an increasing behaviour of the plasticity across layers, meaning that the features learned in deep layers are more dataset dependent, instead the shallow ones are more generic. Then we analyze some features of the internal representation (i.e. the probability distribution) of the hidden layers, finding that shallow layers are well described by a pairwise model, while in deep layers, higher order interactions seem to be more present. This could be related with the hierarchical extraction of features performed by the network. Finally we observe that the representations across the layers become close to the hierarchical feature model, that is a theoretical model consistent with the principles of maximal a priori ignorance and of maximal relevance, describing the internal representation of a learning machine.