DIPPM: a Deep Learning Inference Performance Predictive Model using Graph Neural Networks

by   Karthick Panner Selvam, et al.

Deep Learning (DL) has developed to become a corner-stone in many everyday applications that we are now relying on. However, making sure that the DL model uses the underlying hardware efficiently takes a lot of effort. Knowledge about inference characteristics can help to find the right match so that enough resources are given to the model, but not too much. We have developed a DL Inference Performance Predictive Model (DIPPM) that predicts the inference latency, energy, and memory usage of a given input DL model on the NVIDIA A100 GPU. We also devised an algorithm to suggest the appropriate A100 Multi-Instance GPU profile from the output of DIPPM. We developed a methodology to convert DL models expressed in multiple frameworks to a generalized graph structure that is used in DIPPM. It means DIPPM can parse input DL models from various frameworks. Our DIPPM can be used not only helps to find suitable hardware configurations but also helps to perform rapid design-space exploration for the inference performance of a model. We constructed a graph multi-regression dataset consisting of 10,508 different DL models to train and evaluate the performance of DIPPM, and reached a resulting Mean Absolute Percentage Error (MAPE) as low as 1.9


Deep Learning Inference Frameworks Benchmark

Deep learning (DL) has been widely adopted those last years but they are...

Bosch Deep Learning Hardware Benchmark

The widespread use of Deep Learning (DL) applications in science and ind...

A Comparative Measurement Study of Deep Learning as a Service Framework

Big data powered Deep Learning (DL) and its applications have blossomed ...

Comparing the costs of abstraction for DL frameworks

High level abstractions for implementing, training, and testing Deep Lea...

A Survey of Multi-Tenant Deep Learning Inference on GPU

Deep Learning (DL) models have achieved superior performance. Meanwhile,...

OpenDR: An Open Toolkit for Enabling High Performance, Low Footprint Deep Learning for Robotics

Existing Deep Learning (DL) frameworks typically do not provide ready-to...

AutoCl : A Visual Interactive System for Automatic Deep Learning Classifier Recommendation Based on Models Performance

Nowadays, deep learning (DL) models being increasingly applied to variou...

Please sign up or login with your details

Forgot password? Click here to reset