Learning capabilities of belief propagation based algorithms for sparse binary neural networks

With the growth in size and complexity of modern data sets, the computational and energetic costs of training large, fully connected neural networks, became an important issue. Therefore exploring the learning capabilities of sparse (possibly binarized) architectures, that possess many less degrees of freedom but empirically appear to generalize well, is an important research direction. In this work, the learning performance of belief propagation (BP) based algorithms, applied to simple two layer sparse neural networks with discrete synapses, is analysed. Initially, the framework in which the work is carried on is the so called teacher-student scenario, in which the learning problem corresponds to the inference of the weight values of a teacher network whose architecture is given. In this first part BP provides encouraging results, allowing to perfectly reconstruct the weights of the ‘teacher’ network that generated the training data, using only a small number of data points. Subsequently, the focus shifts to the mismatched setting with a discrepancy between the used architecture and the one of the ‘teacher’ network. The results are analysed in order to assess whether BP-based learning is relevant in this case as well, and if yes, to which extent of mismatch this is possible.