A General-Purpose Transferable Predictor for Neural Architecture Search

by   Fred X. Han, et al.

Understanding and modelling the performance of neural architectures is key to Neural Architecture Search (NAS). Performance predictors have seen widespread use in low-cost NAS and achieve high ranking correlations between predicted and ground truth performance in several NAS benchmarks. However, existing predictors are often designed based on network encodings specific to a predefined search space and are therefore not generalizable to other search spaces or new architecture families. In this paper, we propose a general-purpose neural predictor for NAS that can transfer across search spaces, by representing any given candidate Convolutional Neural Network (CNN) with a Computation Graph (CG) that consists of primitive operators. We further combine our CG network representation with Contrastive Learning (CL) and propose a graph representation learning procedure that leverages the structural information of unlabeled architectures from multiple families to train CG embeddings for our performance predictor. Experimental results on NAS-Bench-101, 201 and 301 demonstrate the efficacy of our scheme as we achieve strong positive Spearman Rank Correlation Coefficient (SRCC) on every search space, outperforming several Zero-Cost Proxies, including Synflow and Jacov, which are also generalizable predictors across search spaces. Moreover, when using our proposed general-purpose predictor in an evolutionary neural architecture search algorithm, we can find high-performance architectures on NAS-Bench-101 and find a MobileNetV3 architecture that attains 79.2 accuracy on ImageNet.


page 1

page 2

page 3

page 4


Neural Architecture Search with GBDT

Neural architecture search (NAS) with an accuracy predictor that predict...

Self-supervised Representation Learning for Evolutionary Neural Architecture Search

Recently proposed neural architecture search (NAS) algorithms adopt neur...

GENNAPE: Towards Generalized Neural Architecture Performance Estimators

Predicting neural architecture performance is a challenging task and is ...

AIO-P: Expanding Neural Performance Predictors Beyond Image Classification

Evaluating neural network performance is critical to deep neural network...

DCLP: Neural Architecture Predictor with Curriculum Contrastive Learning

Neural predictors currently show great potential in the performance eval...

Contrastive Neural Architecture Search with Neural Architecture Comparators

One of the key steps in Neural Architecture Search (NAS) is to estimate ...

Arch-Graph: Acyclic Architecture Relation Predictor for Task-Transferable Neural Architecture Search

Neural Architecture Search (NAS) aims to find efficient models for multi...

Please sign up or login with your details

Forgot password? Click here to reset