Efficient and Robust Mixed-Integer Optimization Methods for Training Binarized Deep Neural Networks

by   Jannis Kurtz, et al.

Compared to classical deep neural networks its binarized versions can be useful for applications on resource-limited devices due to their reduction in memory consumption and computational demands. In this work we study deep neural networks with binary activation functions and continuous or integer weights (BDNN). We show that the BDNN can be reformulated as a mixed-integer linear program with bounded weight space which can be solved to global optimality by classical mixed-integer programming solvers. Additionally, a local search heuristic is presented to calculate locally optimal networks. Furthermore to improve efficiency we present an iterative data-splitting heuristic which iteratively splits the training set into smaller subsets by using the k-mean method. Afterwards all data points in a given subset are forced to follow the same activation pattern, which leads to a much smaller number of integer variables in the mixed-integer programming formulation and therefore to computational improvements. Finally for the first time a robust model is presented which enforces robustness of the BDNN during training. All methods are tested on random and real datasets and our results indicate that all models can often compete with or even outperform classical DNNs on small network architectures confirming the viability for applications having restricted memory or computing power.


page 1

page 2

page 3

page 4


An Integer Programming Approach to Deep Neural Networks with Binary Activation Functions

We study deep neural networks with binary activation functions (BDNN), i...

Identifying Critical Neurons in ANN Architectures using Mixed Integer Programming

We introduce a novel approach to optimize the architecture of deep neura...

Federated K-Means Clustering via Dual Decomposition-based Distributed Optimization

The use of distributed optimization in machine learning can be motivated...

XEngine: Optimal Tensor Rematerialization for Neural Networks in Heterogeneous Environments

Memory efficiency is crucial in training deep learning networks on resou...

On Training Neural Networks with Mixed Integer Programming

Recent work has shown potential in using Mixed Integer Programming (MIP)...

Deep Learning-powered Iterative Combinatorial Auctions

In this paper, we study the design of deep learning-powered iterative co...

A distribution-free mixed-integer optimization approach to hierarchical modelling of clustered and longitudinal data

We create a mixed-integer optimization (MIO) approach for doing cluster-...

Please sign up or login with your details

Forgot password? Click here to reset