Training Gaussian Restricted Boltzmann machines using Expectation Propagation

This thesis investigates a new training technique for Restricted Boltzmann Machines (RBMs), a type of stochastic neural network employed in unsupervised learning. Specifically, the focus of this work is on RBMs with a Gaussian prior distribution for the hidden units, leading to a training technique that depends exclusively on the visible units provided as input. The method relies on Expectation Propagation (EP), a Bayesian inference technique designed to approximate intractable distributions. As a testing ground, we analyze its performance on the MNIST dataset, a large database of handwritten digits commonly used for training and testing machine learning algorithms. First, we begin by detailing the historical background and the foundational concepts of RBMs, followed by a similar exposition for EP. Secondly, we present the mathematical steps employed for implementing the EP formalism to RBMs. Afterward, we train our model on the digits 0 and 1 of the MNIST dataset, leveraging many independent copies of the approximation to improve the efficacy of the algorithm. After identifying the optimal conditions for efficient training and achieving satisfactory results, we employ the exploration of the dataset space achieved at convergence to cluster the dataset. A similar analysis is then performed for the more complex case of digits ranging from 0 to 4. Finally, we investigate the potential of integrating a population dynamics scheme into RBM training, allowing multiple independent copies of the system to evolve and interact with each other, exchanging information to explore the solution space more effectively.