Zero-Cost Proxies in Neural Architecture Search: A Comprehensive Study and Design of a novel hybrid proxy

Automated machine learning (AutoML) is a new, rapidly growing area of machine learning (ML) that aims to automate the creation of machine learning pipelines, including the design, training and deployment of machine learning models, model selection and hyperparameter tuning. Neural Architecture Search (NAS) is a sub-field of AutoML that focuses specifically on automating the design of neural network architectures. NAS has become increasingly popular due to its potential to significantly improve the performance of deep learning models by discovering optimal neural network architectures. However, the search process can be computationally expensive as it requires training and evaluating a large number of architectures. This is where Zero-cost proxies come in. They are low-fidelity approximation techniques that can be used to estimate the performance of a candidate architecture without the need for full training. Zero-cost (ZC) proxies significantly lower the computational cost of the search procedure and can result in a more effective and efficient NAS. In this thesis, AutoML and NAS are exhaustively presented providing the reader with the necessary context for understanding the Zero-cost proxies. Next, existing Zero-cost proxies are exposed and investigated, followed by the research and analysis of the newly proposed Hybrid ZC proxy. The Hybrid Zero-cost proxy is obtained by combining the benefits of both Data-Independent (DI) and Data-Dependent (DD) ZC proxies. Specifically, the Hybrid ZC proxy is created through a weighted sum of LogSynflow, which is the best performing ZC proxy DI, and Nwot , which is the best performing ZC proxy DD. To make a quantitative comparison with the other existing ZC proxy metrics, several tests are conducted to evaluate the performances on various reference datasets. Specifically, the performance of each ZC proxy are measured by calculating two correlation coefficients, namely Kendall-tau and Spearman, between the score obtained by the neural network architecture with the ZC proxy and the accuracy achieved by training and testing the model. The experimental results show that the proposed Hybrid Zero-cost proxy outperforms the LogSynflow and Nwot ZC proxies in terms of Kendall-tau and Spearman correlation values, achieving higher correlation values. Overall, the Hybrid Zero-cost proxy proposed in this Master’s thesis show highly promising performance and outperform its main competitors, LogSynflow and Nwot, in all three tested datasets. These results suggest that the proposed solution could be a crucial factor in enabling the development of faster and more effective Automated machine learning systems. Further research and experimentation are needed to confirm its potential impact, but these initial findings are a strong indication of its possibilities.