Leonardo Tredese
Structured Pruning of Vision Transformers at Training Time.
Rel. Daniele Jahier Pagliari, Alessio Burrello, Matteo Risso, Beatrice Alessandra Motetti. Politecnico di Torino, Corso di laurea magistrale in Data Science And Engineering, 2023
|
Preview |
PDF (Tesi_di_laurea)
- Tesi
Licenza: Creative Commons Attribution Non-commercial No Derivatives. Download (4MB) | Preview |
Abstract
Attention-based transformers have emerged as a powerful paradigm achiev- ing state-of-the-art results on tasks such as natural language processing and computer vision. However, transformers typically present higher computa- tional costs and parameter count compared to convolutional networks. This inefficiency impedes deploying transformers to resource constrained devices such as edge devices. Structured pruning techniques present a promising direction to compress transformers for the edge computing scenario. This thesis investigates pruning techniques to induce structured sparsity in vision transformers, thereby reducing computational requirements while minimizing accuracy degradation. The goal is developing methodologies for efficient vision transformer inference. Structured pruning learns importance scores for individual network com- ponents at training time by solving an optimization problem that tries to maximize task performance while minimizing the number of parameters in the model.
The importance scores are then transformed into binary masks that prune unimportant structures such as specific linear layers output di- mensions or entire attention heads
Relatori
Anno Accademico
Tipo di pubblicazione
Numero di pagine
Corso di laurea
Classe di laurea
URI
![]() |
Modifica (riservato agli operatori) |
