Deep Equilibrium Optical Flow Estimation

by   Shaojie Bai, et al.

Many recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms by encouraging iterative refinements toward a stable flow estimation. However, these RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. They can converge poorly and thereby suffer from performance degradation. To combat these drawbacks, we propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer (using any black-box solver), and differentiates through this fixed point analytically (thus requiring O(1) training memory). This implicit-depth approach is not predicated on any specific model, and thus can be applied to a wide range of SOTA flow estimation model designs. The use of these DEQ flow estimators allows us to compute the flow faster using, e.g., fixed-point reuse and inexact gradients, consumes 4∼6× times less training memory than the recurrent counterpart, and achieves better results with the same computation budget. In addition, we propose a novel, sparse fixed-point correction scheme to stabilize our DEQ flow estimators, which addresses a longstanding challenge for DEQ models in general. We test our approach in various realistic settings and show that it improves SOTA methods on Sintel and KITTI datasets with substantially better computational and memory efficiency.


page 1

page 8

page 15

page 16


Stabilizing Equilibrium Models by Jacobian Regularization

Deep equilibrium networks (DEQs) are a new class of models that eschews ...

Deep Equilibrium Models

We present a new approach to modeling sequential data: the deep equilibr...

SAMFlow: Eliminating Any Fragmentation in Optical Flow with Segment Anything Model

Optical flow estimation aims to find the 2D dense motion field between t...

Speed estimation evaluation on the KITTI benchmark based on motion and monocular depth information

In this technical report we investigate speed estimation of the ego-vehi...

Feature extraction of machine learning and phase transition point of Ising model

We study the features extracted by the Restricted Boltzmann Machine (RBM...

Deep Equilibrium Object Detection

Query-based object detectors directly decode image features into object ...

Deep Equilibrium Models for Video Snapshot Compressive Imaging

The ability of snapshot compressive imaging (SCI) systems to efficiently...

Please sign up or login with your details

Forgot password? Click here to reset