Rethinking skip connection model as a learnable Markov chain

by   Dengsheng Chen, et al.

Over past few years afterward the birth of ResNet, skip connection has become the defacto standard for the design of modern architectures due to its widespread adoption, easy optimization and proven performance. Prior work has explained the effectiveness of the skip connection mechanism from different perspectives. In this work, we deep dive into the model's behaviors with skip connections which can be formulated as a learnable Markov chain. An efficient Markov chain is preferred as it always maps the input data to the target domain in a better way. However, while a model is explained as a Markov chain, it is not guaranteed to be optimized following an efficient Markov chain by existing SGD-based optimizers which are prone to get trapped in local optimal points. In order to towards a more efficient Markov chain, we propose a simple routine of penal connection to make any residual-like model become a learnable Markov chain. Aside from that, the penal connection can also be viewed as a particular model regularization and can be easily implemented with one line of code in the most popular deep learning frameworks [Source code: <>]. The encouraging experimental results in multi-modal translation and image recognition empirically confirm our conjecture of the learnable Markov chain view and demonstrate the superiority of the proposed penal connection.


page 1

page 2

page 3

page 4


A new belief Markov chain model and its application in inventory prediction

Markov chain model is widely applied in many fields, especially the fiel...

Markov Chain Lifting and Distributed ADMM

The time to converge to the steady state of a finite Markov chain can be...

A Markov Chain Model for the Cure Rate of Non-Performing Loans

A Markov-chain model is developed for the purpose estimation of the cure...

Stochastic Gradient Descent under Markovian Sampling Schemes

We study a variation of vanilla stochastic gradient descent where the op...

Generative Flow Networks: a Markov Chain Perspective

While Markov chain Monte Carlo methods (MCMC) provide a general framewor...

Linking Microscopic and Macroscopic Models for Evolution: Markov Chain Network Training and Conservation Law Approximations

In this paper, a general framework for the analysis of a connection betw...

UPANets: Learning from the Universal Pixel Attention Networks

Among image classification, skip and densely-connection-based networks h...

Please sign up or login with your details

Forgot password? Click here to reset