Kernel-as-a-Service: A Serverless Interface to GPUs

by   Nathan Pemberton, et al.

Serverless computing has made it easier than ever to deploy applications over scalable cloud resources, all the while driving higher utilization for cloud providers. While this technique has worked well for easily divisible resources like CPU and local DRAM, it has struggled to incorporate more expensive and monolithic resources like GPUs or other application accelerators. We cannot simply slap a GPU on a FaaS platform and expect to keep all the benefits serverless promises. We need a more tailored approach if we want to best utilize these critical resources. In this paper we present Kernel-as-a-Service (KaaS), a serverless interface to GPUs. In KaaS, GPUs are first-class citizens that are invoked just like any other serverless function. Rather than mixing host and GPU code as is typically done, KaaS runs graphs of GPU-only code while host code is run on traditional functions. The KaaS system is responsible for managing GPU memory and schedules user kernels across the entire pool of available GPUs rather than relying on static allocations. This approach allows us to more effectively share expensive GPU resources, especially in multitenant environments like the cloud. We add support for KaaS to the Ray distributed computing framework and evaluate it with workloads including a TVM-based deep learning compiler and a BLAS library. Our results show that KaaS is able to drive up to 50x higher throughput and 16x lower latency when GPU resources are contended.


page 1

page 2

page 3

page 4


Effective GPU Sharing Under Compiler Guidance

Modern computing platforms tend to deploy multiple GPUs (2, 4, or more) ...

GPU PaaS Computation Model in Aneka Cloud Computing Environment

Due to the surge in the volume of data generated and rapid advancement i...

The anachronism of whole-GPU accounting

NVIDIA has been making steady progress in increasing the compute perform...

Recent Advances in Overcoming Bottlenecks in Memory Systems and Managing Memory Resources in GPU Systems

This article features extended summaries and retrospectives of some of t...

Expanding IceCube GPU computing into the Clouds

The IceCube collaboration relies on GPU compute for many of its needs, i...

Targeting GPUs with OpenMP Directives on Summit: A Simple and Effective Fortran Experience

We use OpenMP directives to target hardware accelerators (GPUs) on Summi...

The OoO VLIW JIT Compiler for GPU Inference

Current trends in Machine Learning (ML) inference on hardware accelerate...

Please sign up or login with your details

Forgot password? Click here to reset