Towards Explainable Bit Error Tolerance of Resistive RAM-Based Binarized Neural Networks

by   Sebastian Buschjäger, et al.

Non-volatile memory, such as resistive RAM (RRAM), is an emerging energy-efficient storage, especially for low-power machine learning models on the edge. It is reported, however, that the bit error rate of RRAMs can be up to 3.3 cases. Binary neural networks (BNNs), a resource efficient variant of neural networks (NNs), can tolerate a certain percentage of errors without a loss in accuracy and demand lower resources in computation and storage. The bit error tolerance (BET) in BNNs can be achieved by flipping the weight signs during training, as proposed by Hirtzlin et al., but their method has a significant drawback, especially for fully connected neural networks (FCNN): The FCNNs overfit to the error rate used in training, which leads to low accuracy under lower error rates. In addition, the underlying principles of BET are not investigated. In this work, we improve the training for BET of BNNs and aim to explain this property. We propose straight-through gradient approximation to improve the weight-sign-flip training, by which BNNs adapt less to the bit error rates. To explain the achieved robustness, we define a metric that aims to measure BET without fault injection. We evaluate the metric and find that it correlates with accuracy over error rate for all FCNNs tested. Finally, we explore the influence of a novel regularizer that optimizes with respect to this metric, with the aim of providing a configurable trade-off in accuracy and BET.


Bit Error Tolerance Metrics for Binarized Neural Networks

To reduce the resource demand of neural network (NN) inference systems, ...

Training wide residual networks for deployment using a single bit for each weight

For fast and energy-efficient deployment of trained deep neural networks...

SparkXD: A Framework for Resilient and Energy-Efficient Spiking Neural Network Inference using Approximate DRAM

Spiking Neural Networks (SNNs) have the potential for achieving low ener...

Characterizing a Neutron-Induced Fault Model for Deep Neural Networks

The reliability evaluation of Deep Neural Networks (DNNs) executed on Gr...

Live Recovery of Bit Corruptions in Datacenter Storage Systems

Due to its high performance and decreasing cost per bit, flash is becomi...

Gradient-based Bit Encoding Optimization for Noise-Robust Binary Memristive Crossbar

Binary memristive crossbars have gained huge attention as an energy-effi...

Please sign up or login with your details

Forgot password? Click here to reset