AutoDiCE: Fully Automated Distributed CNN Inference at the Edge

by   Xiaotian Guo, et al.

Deep Learning approaches based on Convolutional Neural Networks (CNNs) are extensively utilized and very successful in a wide range of application areas, including image classification and speech recognition. For the execution of trained CNNs, i.e. model inference, we nowadays witness a shift from the Cloud to the Edge. Unfortunately, deploying and inferring large, compute and memory intensive CNNs on edge devices is challenging because these devices typically have limited power budgets and compute/memory resources. One approach to address this challenge is to leverage all available resources across multiple edge devices to deploy and execute a large CNN by properly partitioning the CNN and running each CNN partition on a separate edge device. Although such distribution, deployment, and execution of large CNNs on multiple edge devices is a desirable and beneficial approach, there currently does not exist a design and programming framework that takes a trained CNN model, together with a CNN partitioning specification, and fully automates the CNN model splitting and deployment on multiple edge devices to facilitate distributed CNN inference at the Edge. Therefore, in this paper, we propose a novel framework, called AutoDiCE, for automated splitting of a CNN model into a set of sub-models and automated code generation for distributed and collaborative execution of these sub-models on multiple, possibly heterogeneous, edge devices, while supporting the exploitation of parallelism among and within the edge devices. Our experimental results show that AutoDiCE can deliver distributed CNN inference with reduced energy consumption and memory usage per edge device, and improved overall system throughput at the same time.


The Effects of Partitioning Strategies on Energy Consumption in Distributed CNN Inference at The Edge

Nowadays, many AI applications utilizing resource-constrained edge devic...

How Secure is Distributed Convolutional Neural Network on IoT Edge Devices?

Convolutional Neural Networks (CNN) has found successful adoption in man...

DistrEdge: Speeding up Convolutional Neural Network Inference on Distributed Edge Devices

As the number of edge devices with computing resources (e.g., embedded G...

Latency-Memory Optimized Splitting of Convolution Neural Networks for Resource Constrained Edge Devices

With the increasing reliance of users on smart devices, bringing essenti...

Performance Prediction for Convolutional Neural Networks in Edge Devices

Running Convolutional Neural Network (CNN) based applications on edge de...

SensiX++: Bringing MLOPs and Multi-tenant Model Serving to Sensory Edge Devices

We present SensiX++ - a multi-tenant runtime for adaptive model executio...

Receptive Field-based Segmentation for Distributed CNN Inference Acceleration in Collaborative Edge Computing

This paper studies inference acceleration using distributed convolutiona...

Please sign up or login with your details

Forgot password? Click here to reset