Benchmarking Contemporary Deep Learning Hardware and Frameworks:A Survey of Qualitative Metrics

by   Wei Dai, et al.

This paper surveys benchmarking principles, machine learning devices including GPUs, FPGAs, and ASICs, and deep learning software frameworks. It also reviews these technologies with respect to benchmarking from the perspectives of a 6-metric approach to frameworks and an 11-metric approach to hardware platforms. Because MLPerf is a benchmark organization working with industry and academia, and offering deep learning benchmarks that evaluate training and inference on deep learning hardware devices, the survey also mentions MLPerf benchmark results, benchmark metrics, datasets, deep learning frameworks and algorithms. We summarize seven benchmarking principles, differential characteristics of mainstream AI devices, and qualitative comparison of deep learning hardware and frameworks.


page 1

page 4


Benchmarking Deep Learning Hardware and Frameworks: Qualitative Metrics

Previous survey papers offer knowledge of deep learning hardware devices...

BENCHIP: Benchmarking Intelligence Processors

The increasing attention on deep learning has tremendously spurred the d...

MLPerf Training Benchmark

Machine learning is experiencing an explosion of software and hardware s...

Codabench: Flexible, Easy-to-Use and Reproducible Benchmarking for Everyone

Obtaining standardized crowdsourced benchmark of computational methods i...

Qualitative Benchmarking of Deep Learning Hardware and Frameworks: Review and Tutorial

Previous survey papers offer knowledge of deep learning hardware devices...

A Modular Benchmarking Infrastructure for High-Performance and Reproducible Deep Learning

We introduce Deep500: the first customizable benchmarking infrastructure...

MLPerf Mobile Inference Benchmark: Why Mobile AI Benchmarking Is Hard and What to Do About It

MLPerf Mobile is the first industry-standard open-source mobile benchmar...

Code Repositories

Please sign up or login with your details

Forgot password? Click here to reset