Politecnico di Torino (logo)

Spectral Analysis of Infinitely Wide Convolutional Neural Networks

Alessandro Favero

Spectral Analysis of Infinitely Wide Convolutional Neural Networks.

Rel. Alfredo Braunstein, Matthieu Wyart. Politecnico di Torino, Corso di laurea magistrale in Physics Of Complex Systems (Fisica Dei Sistemi Complessi), 2020

PDF (Tesi_di_laurea) - Tesi
Licenza: Creative Commons Attribution Non-commercial No Derivatives.

Download (1MB) | Preview

Recent works have shown the equivalence between training infinitely wide fully connected neural networks (FCNs) by gradient descent and kernel regression with the Neural Tangent Kernel (NTK). This kernel can also be extended to convolutional neural networks (CNNs), modern architectures that achieve stellar performance in image recognition, and other translational-invariant pattern detection tasks. The resulting Convolutional NTKs have been shown to perform strongly in classification experiments. Still, we lack a quantitative understanding of the generalization capabilities of these models. In this thesis, we introduce a minimal convolutional architecture, and we compute the associated NTK. Following recent works on the statistical mechanics of generalization in kernel methods, we study this kernel's performance in a teacher-student setting, comparing it with the NTK of a two-layer FCN when learning translational-invariant data. Finally, we test our predictions with numerical experiments both on synthetic and real data. Our results show that these kernels cannot compress invariant dimensions and escape the curse of dimensionality. However, the convolutional kernel's eigenfunctions are better aligned with translational-invariant data, effectively lowering the generalization error by a dimensional-dependent prefactor.

Relators: Alfredo Braunstein, Matthieu Wyart
Academic year: 2020/21
Publication type: Electronic
Number of Pages: 47
Corso di laurea: Corso di laurea magistrale in Physics Of Complex Systems (Fisica Dei Sistemi Complessi)
Classe di laurea: New organization > Master science > LM-44 - MATHEMATICAL MODELLING FOR ENGINEERING
Ente in cotutela: EPFL École Polytechnique Fédérale de Lausanne (SVIZZERA)
URI: http://webthesis.biblio.polito.it/id/eprint/15963
Modify record (reserved for operators) Modify record (reserved for operators)