Spectral Analysis of Infinitely Wide Convolutional Neural Networks

Recent works have shown the equivalence between training infinitely wide fully connected neural networks (FCNs) by gradient descent and kernel regression with the Neural Tangent Kernel (NTK). This kernel can also be extended to convolutional neural networks (CNNs), modern architectures that achieve stellar performance in image recognition, and other translational-invariant pattern detection tasks. The resulting Convolutional NTKs have been shown to perform strongly in classification experiments. Still, we lack a quantitative understanding of the generalization capabilities of these models. In this thesis, we introduce a minimal convolutional architecture, and we compute the associated NTK. Following recent works on the statistical mechanics of generalization in kernel methods, we study this kernel's performance in a teacher-student setting, comparing it with the NTK of a two-layer FCN when learning translational-invariant data. Finally, we test our predictions with numerical experiments both on synthetic and real data. Our results show that these kernels cannot compress invariant dimensions and escape the curse of dimensionality. However, the convolutional kernel's eigenfunctions are better aligned with translational-invariant data, effectively lowering the generalization error by a dimensional-dependent prefactor.