Canvas: End-to-End Kernel Architecture Search in Neural Networks

by   Chenggang Zhao, et al.

The demands for higher performance and accuracy in neural networks (NNs) never end. Existing tensor compilation and Neural Architecture Search (NAS) techniques orthogonally optimize the two goals but actually share many similarities in their concrete strategies. We exploit such opportunities by combining the two into one and make a case for Kernel Architecture Search (KAS). KAS reviews NAS from a system perspective and zooms into a more fine-grained level to generate neural kernels with both high performance and good accuracy. To demonstrate the potential of KAS, we build an end-to-end framework, Canvas, to find high-quality kernels as convolution replacements. Canvas samples from a rich set of fine-grained primitives to stochastically and iteratively construct new kernels and evaluate them according to user-specified constraints. Canvas supports freely adjustable tensor dimension sizes inside the kernel and uses two levels of solvers to satisfy structural legality and fully utilize model budgets. The evaluation shows that by replacing standard convolutions with generated new kernels in common NNs, Canvas achieves average 1.5x speedups compared to the previous state-of-the-art with acceptable accuracy loss and search efficiency. Canvas verifies the practicability of KAS by rediscovering many manually designed kernels in the past and producing new structures that may inspire future machine learning innovations. For source code and implementation, we open-sourced Canvas at


page 3

page 9

page 11

page 12


AtomNAS: Fine-Grained End-to-End Neural Architecture Search

Designing of search space is a critical problem for neural architecture ...

Efficient Neural Architecture Search with Performance Prediction

Neural networks are powerful models that have a remarkable ability to ex...

Efficient Differentiable Neural Architecture Search with Meta Kernels

The searching procedure of neural architecture search (NAS) is notorious...

Isolate First, Then Share: a New OS Architecture for Datacenter Computing

This paper presents the "isolate first, then share" OS model in which th...

Neural Architecture Search as Program Transformation Exploration

Improving the performance of deep neural networks (DNNs) is important to...

Can GPT-4 Perform Neural Architecture Search?

We investigate the potential of GPT-4~\cite{gpt4} to perform Neural Arch...

Learning Efficient, Explainable and Discriminative Representations for Pulmonary Nodules Classification

Automatic pulmonary nodules classification is significant for early diag...

Please sign up or login with your details

Forgot password? Click here to reset