An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems

by   Feng Bao, et al.

We propose an ensemble score filter (EnSF) for solving high-dimensional nonlinear filtering problems with superior accuracy. A major drawback of existing filtering methods, e.g., particle filters or ensemble Kalman filters, is the low accuracy in handling high-dimensional and highly nonlinear problems. EnSF attacks this challenge by exploiting the score-based diffusion model, defined in a pseudo-temporal domain, to characterizing the evolution of the filtering density. EnSF stores the information of the recursively updated filtering density function in the score function, in stead of storing the information in a set of finite Monte Carlo samples (used in particle filters and ensemble Kalman filters). Unlike existing diffusion models that train neural networks to approximate the score function, we develop a training-free score estimation that uses mini-batch-based Monte Carlo estimator to directly approximate the score function at any pseudo-spatial-temporal location, which provides sufficient accuracy in solving high-dimensional nonlinear problems as well as saves tremendous amount of time spent on training neural networks. Another essential aspect of EnSF is its analytical update step, gradually incorporating data information into the score function, which is crucial in mitigating the degeneracy issue faced when dealing with very high-dimensional nonlinear filtering problems. High-dimensional Lorenz systems are used to demonstrate the performance of our method. EnSF provides surprisingly impressive performance in reliably tracking extremely high-dimensional Lorenz systems (up to 1,000,000 dimension) with highly nonlinear observation processes, which is a well-known challenging problem for existing filtering methods.


page 1

page 2

page 3

page 4


Machine learning-based conditional mean filter: a generalization of the ensemble Kalman filter for nonlinear data assimilation

Filtering is a data assimilation technique that performs the sequential ...

Algorithms for high-dimensional non-linear filtering and smoothing problems

Several numerical tools designed to overcome the challenges of smoothing...

Multilevel Ensemble Kalman-Bucy Filters

In this article we consider the linear filtering problem in continuous-t...

Nonlinear Kalman Filtering with Divergence Minimization

We consider the nonlinear Kalman filtering problem using Kullback-Leible...

Island filters for partially observed spatiotemporal systems

Statistical inference for high-dimensional partially observed, nonlinear...

A Class of Spatial Filtering Problems with Unknown Spatial Observations

We consider a class of high-dimensional spatial filtering problems, wher...

A General Class of Score-Driven Smoothers

Motivated by the observation that score-driven models can be viewed as a...

Please sign up or login with your details

Forgot password? Click here to reset