Beyond Statistical Similarity: Rethinking Metrics for Deep Generative Models in Engineering Design

by   Lyle Regenwetter, et al.

Deep generative models, such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), Diffusion Models, and Transformers, have shown great promise in a variety of applications, including image and speech synthesis, natural language processing, and drug discovery. However, when applied to engineering design problems, evaluating the performance of these models can be challenging, as traditional statistical metrics based on likelihood may not fully capture the requirements of engineering applications. This paper doubles as a review and a practical guide to evaluation metrics for deep generative models (DGMs) in engineering design. We first summarize well-accepted `classic' evaluation metrics for deep generative models grounded in machine learning theory and typical computer science applications. Using case studies, we then highlight why these metrics seldom translate well to design problems but see frequent use due to the lack of established alternatives. Next, we curate a set of design-specific metrics which have been proposed across different research communities and can be used for evaluating deep generative models. These metrics focus on unique requirements in design and engineering, such as constraint satisfaction, functional performance, novelty, and conditioning. We structure our review and discussion as a set of practical selection criteria and usage guidelines. Throughout our discussion, we apply the metrics to models trained on simple 2-dimensional example problems. Finally, to illustrate the selection process and classic usage of the presented metrics, we evaluate three deep generative models on a multifaceted bicycle frame design problem considering performance target achievement, design novelty, and geometric constraints. We publicly release the code for the datasets, models, and metrics used throughout the paper at


page 1

page 2

page 3

page 4


Towards Goal, Feasibility, and Diversity-Oriented Deep Generative Models in Design

Deep Generative Machine Learning Models (DGMs) have been growing in popu...

Deep Generative Models in Engineering Design: A Review

Automated design synthesis has the potential to revolutionize the modern...

Design Target Achievement Index: A Differentiable Metric to Enhance Deep Generative Models in Multi-Objective Inverse Design

Deep Generative Machine Learning Models have been growing in popularity ...

A comparative study of stochastic and deep generative models for multisite precipitation synthesis

Future climate change scenarios are usually hypothesized using simulatio...

Diffusing the Optimal Topology: A Generative Optimization Approach

Topology Optimization seeks to find the best design that satisfies a set...

Towards Computational Architecture of Liberty: A Comprehensive Survey on Deep Learning for Generating Virtual Architecture in the Metaverse

3D shape generation techniques utilizing deep learning are increasing at...

Deep Generative Models on 3D Representations: A Survey

Generative models, as an important family of statistical modeling, targe...

Please sign up or login with your details

Forgot password? Click here to reset