polito.it
Politecnico di Torino (logo)

Spectral Analysis of Infinitely Wide Convolutional Neural Networks

Alessandro Favero

Spectral Analysis of Infinitely Wide Convolutional Neural Networks.

Rel. Alfredo Braunstein, Matthieu Wyart. Politecnico di Torino, Corso di laurea magistrale in Physics Of Complex Systems (Fisica Dei Sistemi Complessi), 2020

[img]
Preview
PDF (Tesi_di_laurea) - Tesi
Licenza: Creative Commons Attribution Non-commercial No Derivatives.

Download (1MB) | Preview
Abstract:

Recent works have shown the equivalence between training infinitely wide fully connected neural networks (FCNs) by gradient descent and kernel regression with the Neural Tangent Kernel (NTK). This kernel can also be extended to convolutional neural networks (CNNs), modern architectures that achieve stellar performance in image recognition, and other translational-invariant pattern detection tasks. The resulting Convolutional NTKs have been shown to perform strongly in classification experiments. Still, we lack a quantitative understanding of the generalization capabilities of these models. In this thesis, we introduce a minimal convolutional architecture, and we compute the associated NTK. Following recent works on the statistical mechanics of generalization in kernel methods, we study this kernel's performance in a teacher-student setting, comparing it with the NTK of a two-layer FCN when learning translational-invariant data. Finally, we test our predictions with numerical experiments both on synthetic and real data. Our results show that these kernels cannot compress invariant dimensions and escape the curse of dimensionality. However, the convolutional kernel's eigenfunctions are better aligned with translational-invariant data, effectively lowering the generalization error by a dimensional-dependent prefactor.

Relatori: Alfredo Braunstein, Matthieu Wyart
Anno accademico: 2020/21
Tipo di pubblicazione: Elettronica
Numero di pagine: 47
Soggetti:
Corso di laurea: Corso di laurea magistrale in Physics Of Complex Systems (Fisica Dei Sistemi Complessi)
Classe di laurea: Nuovo ordinamento > Laurea magistrale > LM-44 - MODELLISTICA MATEMATICO-FISICA PER L'INGEGNERIA
Ente in cotutela: EPFL École Polytechnique Fédérale de Lausanne (SVIZZERA)
Aziende collaboratrici: ECOLE POLYTECHNIQUE FEDERALE DE LAUSANNE
URI: http://webthesis.biblio.polito.it/id/eprint/15963
Modifica (riservato agli operatori) Modifica (riservato agli operatori)