Efficient data assimilation for spatiotemporal chaos: A local ensemble transform Kalman filter

Abstract Data assimilation is an iterative approach to the problem of estimating the state of a dynamical system using both current and past observations of the system together with a model for the system’s time evolution. Rather than solving the problem from scratch each time new observations become available, one uses the model to “forecast” the current state, using a prior state estimate (which incorporates information from past data) as the initial condition, then uses current data to correct the prior forecast to a current state estimate. This Bayesian approach is most effective when the uncertainty in both the observations and in the state estimate, as it evolves over time, are accurately quantified. In this article, we describe a practical method for data assimilation in large, spatiotemporally chaotic systems. The method is a type of “ensemble Kalman filter”, in which the state estimate and its approximate uncertainty are represented at any given time by an ensemble of system states. We discuss both the mathematical basis of this approach and its implementation; our primary emphasis is on ease of use and computational speed rather than improving accuracy over previously published approaches to ensemble Kalman filtering. We include some numerical results demonstrating the efficiency and accuracy of our implementation for assimilating real atmospheric data with the global forecast model used by the US National Weather Service.

• Publication year

  2005

• Venue

  Physica D: Nonlinear Phenomena

• Publication date

  2005-11-28

• Fields of study

  Physics, Computer Science

• Identifiers
  DOI 10.1016/J.PHYSD.2006.11.008arXiv physics/0511236
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Assimilation

  2019influential reference
• A comparative study of 4D-VAR and a 4D Ensemble Kalman Filter: perfect model simulations with Lorenz-96

  2007cited by this paper
• Four-dimensional local ensemble transform Kalman filter: numerical experiments with a global circulation model

  2007cited by this paper
• Local Ensemble Transform Kalman Filtering with an AGCM at a T159/L48 Resolution

  2007cited by this paper
• Variational Assimilation of Meteorological Observations With the Adjoint Vorticity Equation. I: Theory

  2007cited by this paper
• A non-Gaussian Ensemble Filter for Assimilating Infrequent Noisy Observations

  2007cited by this paper
• Data Assimilation: The Ensemble Kalman Filter

  2006cited by this paper
• Local Ensemble Transform Kalman Filter : An Efficient Scheme for Assimilating Atmospheric Data

  2006cited by this paper
• Predictability of Weather and Climate: Predictability – a problem partly solved

  2006cited by this paper
• Local ensemble Kalman filtering in the presence of model bias

  2006cited by this paper
• ENSEMBLE KALMAN FILTER EXPERIMENTS WITH A PRIMITIVE-EQUATION GLOBAL MODEL

  2005cited by this paper
• Mechanisms for the Development of Locally Low-Dimensional Atmospheric Dynamics

  2005cited by this paper
• ENSEMBLE KALMAN FILTER EXPERIMENTS WITH A PRIMITIVE-EQUATION GLOBAL MODEL

  2005cited by this paper
• Assessing a local ensemble Kalman filter: perfect model experiments with the National Centers for Environmental Prediction global model

  2005cited by this paper
• Maximum Likelihood Ensemble Filter: Theoretical Aspects

  2005cited by this paper
• Atmospheric Data Assimilation with an Ensemble Kalman Filter: Results with Real Observations

  2005cited by this paper
• Which Is Better, an Ensemble of Positive–Negative Pairs or a Centered Spherical Simplex Ensemble?

  2004cited by this paper
• Four-dimensional ensemble Kalman filtering

  2004cited by this paper
• Impacts of Initial Estimate and Observation Availability on Convective-Scale Data Assimilation with an Ensemble Kalman Filter

  2004cited by this paper
• The potential of the ensemble Kalman filter for NWP—a comparison with 4D‐Var

  2003cited by this paper
• The Ensemble Kalman Filter: theoretical formulation and practical implementation

  2003cited by this paper
• Ensemble Data Assimilation without Perturbed Observations

  2002influential reference
• Exploiting Local Low Dimensionality of the Atmospheric Dynamics for Efficient Ensemble Kalman Filtering

  2002cited by this paper
• A Partitioned Kalman Filter and Smoother

  2002influential reference
• A New Approach to Linear Filtering and Prediction Problems

  2002cited by this paper
• Atmospheric Modeling, Data Assimilation and Predictability

  2002cited by this paper
• Distance-dependent filtering of background error covariance estimates in an ensemble Kalman filter

  2001influential reference
• An Ensemble Adjustment Kalman Filter for Data Assimilation

  2001influential reference
• A New Approach to Linear Filtering and Prediction Problems

  2001influential reference
• Sequential Monte Carlo Methods in Practice

  2001cited by this paper
• A Sequential Ensemble Kalman Filter for Atmospheric Data Assimilation

  2001influential reference
• Local low dimensionality of atmospheric dynamics.

  2001cited by this paper
• Editors: Sequential Monte Carlo Methods in Practice

  2001cited by this paper
• A Hybrid Ensemble Kalman Filter-3D Variational Analysis Scheme

  2000cited by this paper
• An ensemble Kalman smoother for nonlinear dynamics

  2000cited by this paper
• q 2002 American Meteorological Society

  2000cited by this paper
• Data Assimilation into a Primitive-Equation Model with a Parallel Ensemble Kalman Filter

  2000cited by this paper
• A Monte Carlo Implementation of the Nonlinear Filtering Problem to Produce Ensemble Assimilations and Forecasts

  1999cited by this paper
• Analysis Scheme in the Ensemble Kalman Filter

  1998cited by this paper
• Optimal sites for supplementary weather observations: simulation with a small model

  1998cited by this paper
• Data Assimilation Using an Ensemble Kalman Filter Technique

  1998influential reference
• Unified Notation for Data Assimilation : Operational, Sequential and Variational

  1997cited by this paper
• Sequential data assimilation with a nonlinear quasi‐geostrophic model using Monte Carlo methods to forecast error statistics

  1994cited by this paper
• The National Meteorological Center's spectral-statistical interpolation analysis system

  1992cited by this paper
• Data assimilation in meteorology and oceanography

  1991cited by this paper
• Variational algorithms for analysis and assimilation of meteorological observations: theoretical aspects

  1986cited by this paper
• A Global Three-Dimensional Multivariate Statistical Interpolation Scheme

  1981cited by this paper
• WHICH IS BETTER?

  1964cited by this paper
• New Results in Linear Filtering and Prediction Theory

  1961cited by this paper

• STUDY ON ESTIMATION OF TURBULENT FLOW STRUCTURES BELOW WATER SURFACE BASED ON WATER SURFACE DATA BY SEQUENTIAL DATA ASSIMILATION

  2026cites this paper
• Ensemble-based snow depth data assimilation for a multi-layer snow scheme over the European Arctic

  2026cites this paper
• An adjoint-free method of optimizing initial state and wind forcing in a regional configuration of the Beaufort Sea spectral wave model

  2026cites this paper
• Global assimilation of NRL MODIS aerosol optical thickness and its impact on aerosol direct radiative effect over a full year

  2026cites this paper
• Data assimilation for nonlinear systems with a localized weighted ensemble transform Kalman filter

  2026cites this paper
• Operational numerical weather prediction with ICON on GPUs (version 2024.10)

  2026cites this paper
• A Score-based Diffusion Model Approach for Adaptive Learning of Stochastic Partial Differential Equation Solutions

  2026cites this paper
• Evaluation of GFSv16 for Near‐Real‐Time Data Impact Studies During the Atmospheric River Reconnaissance Program 2022

  2026cites this paper
• Robust adaptive meshing, mesh density functions, and nonlocal observations for ensemble based data assimilation

  2026cites this paper
• Localization in the mapping particle filter

  2026cites this paper
• Evaluate the impact of power-law scattering amplitude fitting on dual-polarization radar data assimilation – summertime cases study

  2026cites this paper
• CTRIP‐HyDAS: A Global‐Scale Data Assimilation Framework for SWOT‐Derived Discharge Using Synthetic Observations at High Resolution (1/12°)

  2026influential citation
• Domain-Decomposed Lagrangian Data Assimilation for Drifting Sea-Ice Floe Dynamics

  2026cites this paper
• LEVDA: Latent Ensemble Variational Data Assimilation via Differentiable Dynamics

  2026cites this paper
• Operational all‐sky assimilation of geostationary infrared water‐vapour channels in the regional ICON‐D2 model with an ensemble Kalman filter

  2026cites this paper
• Neural ensemble Kalman filter: Data assimilation for compressible flows with shocks

  2026cites this paper
• Optimal typhoon scenario selection using ensemble clustering and Bayesian estimation of realization probability

  2026cites this paper
• A review of current best practices and future directions in assimilating GRACE/-FO terrestrial water storage data into numerical models

  2026cites this paper
• Robustness of the Ensemble Score Filter to the Type of Assimilated Observation Networks

  2026cites this paper
• Comparison of ensemble assimilation methods in a hydrological model dedicated to agricultural best management practices

  2026cites this paper
• Impact of reduced non-Gaussianity on analysis and forecast accuracy by assimilating every-30 s radar observation with ensemble Kalman filter: idealized experiments of deep convection

  2026cites this paper
• Stabilizing autoregressive forecasts in chaotic systems via multi-rate latent recurrence

  2026cites this paper
• Twin experiments for data assimilation of cavitating flow around a hydrofoil

  2025cites this paper
• Adversarial Transform Particle Filters

  2025cites this paper
• Bridging Artificial Intelligence and Data Assimilation: The Data-driven Ensemble Forecasting System ClimaX-LETKF

  2025influential citation
• Exploring strongly coupled land/atmosphere data assimilation for numerical weather prediction

  2025cites this paper
• Exploring the Potential of Strongly Coupled Lagrangian Data Assimilation in an Ocean–Atmosphere System

  2025cites this paper
• Improving Coastal Circulation Using HF‐Radar Surface Currents

  2025cites this paper
• The Greenhouse gas Emission Monitoring network to Inform Net-zero Initiatives UK (GEMINI-UK): network design, theoretical performance, and initial data

  2025cites this paper
• Comparison of temperature and wind profiles between ground-based remote sensing observations and numerical weather prediction model in complex Alpine topography: the Meiringen campaign

  2025cites this paper
• The greenhouse gas observation mission with Global Observing SATellite for Greenhouse gases and Water cycle (GOSAT-GW): objectives, conceptual framework and scientific contributions

  2025cites this paper
• Local conditioning in posterior sampling methods with example cases in subsurface inversion

  2025cites this paper
• Optimising the design of an observational testbed using observing system simulation experiments: A case study in the southeastern Arabian Sea

  2025cites this paper
• Improving the ensemble square root filter (EnSRF) in the Community Inversion Framework: a case study with ICON-ART 2024.01

  2025cites this paper
• Exploring Ultra Rapid Data Assimilation Based on Ensemble Transform Kalman Filter with the Lorenz 96 Model

  2025cites this paper
• Enhancement of large-eddy simulations for the prediction of an intake flow rig using sequential data assimilation

  2025influential citation
• Data assimilation with extremum Monte Carlo methods

  2025cites this paper
• Bottom–up approach for mitigating extreme events with limited intervention options: a case study with Lorenz 96 model

  2025cites this paper
• Efficient parameter estimation from a planar interface in directional solidification via AMR-accelerated phase-field data assimilation

  2025cites this paper
• Improving convection‐permitting all‐sky infrared radiance ensemble data assimilation through mitigating deleterious non‐Gaussian artifacts

  2025cites this paper
• Data assimilation of state-indirect observations for constraining detonation wave dynamics

  2025cites this paper
• ADAF: An Artificial Intelligence Data Assimilation Framework for Weather Forecasting

  2025cites this paper
• Reparameterizing 4DVAR with neural fields

  2025cites this paper
• Copernicus Atmosphere Monitoring Service – Regional Air Quality Production System v1.0

  2025cites this paper
• Data-driven Exploration of Tropical Cyclone's Controllability

  2025cites this paper
• Trends and seasonality of 2019–2023 global methane emissions inferred from a localized ensemble transform Kalman filter (CHEEREIO v1.3.1) applied to TROPOMI satellite observations

  2025influential citation
• Paleoclimate data assimilation: Principles and prospects

  2025cites this paper
• Quantifying CO emissions from boreal wildfires by assimilating TROPOMI and TCCON observations

  2025influential citation
• Multi-level data assimilation for ocean forecasting using the shallow-water equations

  2025cites this paper
• Improved weather predictions through data assimilation for GFDL SHiELD

  2025cites this paper
• Incremental Analysis Updates in a Convective‐Scale Ensemble Kalman Filter Using Minute‐by‐Minute Phased Array Radar Observations

  2025cites this paper
• State updating of the Xin'anjiang model: joint assimilating streamflow and multi-source soil moisture data via the asynchronous ensemble Kalman filter with enhanced error models

  2025cites this paper
• On process-oriented conditional targeted covariance inflation (TCI) for 3D-volume radar data assimilation

  2025influential citation
• Increasing Atmospheric Surface Spread in an Ensemble Model Using Land Cover Fraction Perturbations

  2025cites this paper
• From initiation of convective storms to their impact — the Swabian MOSES 2023 campaign in southwestern Germany

  2025cites this paper
• A Python interface to the Fortran-based Parallel Data Assimilation Framework: pyPDAF v1.0.2

  2025cites this paper
• The Ensemble Kalman Update is an Empirical Matheron Update

  2025cites this paper
• An innovative approach to ZDR data assimilation using an ensemble Kalman filter: a proof-of-concept study

  2025cites this paper
• Identifying drivers of surface ozone bias in global chemical reanalysis with explainable machine learning

  2025cites this paper
• Data-driven phase-field analysis of static recrystallization in an aluminum alloy

  2025cites this paper
• Ensemble Kalman filter in latent space using a variational autoencoder pair

  2025cites this paper
• Flow-Dependent Large-Scale Blending for Limited-Area Ensemble Data Assimilation

  2025cites this paper
• Accelerating River Discharge in High Mountain Asia

  2025cites this paper
• A Feasibility Study on Improving Tropical‐Cyclone Inner‐Core Structure Based on Data Assimilation of Inner‐Core Atmospheric Motion Vectors

  2025cites this paper
• Satellite-Based Emission Inversion for Air Pollutants and Greenhouse Gases: A Review

  2025cites this paper
• Overview and Prospect of Data Assimilation in Numerical Weather Prediction

  2025cites this paper
• CNN-based Surface Temperature Forecasts with Ensemble Numerical Weather Prediction over Medium-range Forecast Periods

  2025cites this paper
• Assimilation of Raman lidar profiles in an operational, convective‐scale numerical weather prediction model

  2025cites this paper
• How Does Assimilating SMAP Soil Moisture Improve Characterization of the Terrestrial Water Cycle in an Integrated Land Surface‐Subsurface Model?

  2025cites this paper
• Stacking Ensemble Learning and SHAP-Based Insights for Urban Air Quality Forecasting: Evidence from Shenyang and Global Implications

  2025cites this paper
• All-sky AMSU-A radiance data assimilation using the gain-form of Local Ensemble Transform Kalman filter within MPAS-JEDI-2.1.0: implementation, tuning, and evaluation

  2025cites this paper
• Improving Indian Ocean analysis using ROMS with sea level anomaly assimilation

  2025cites this paper
• A wave data assimilation system for the North Sea based on Ensemble Kalman Filtering and the potential of satellite altimetry

  2025cites this paper
• Can ensemble‐based parameter estimation aid parameterization design?

  2025cites this paper
• Assimilation of Argo Profile Measurements and Sea Surface Temperature Data by the NEMO Ocean Circulation Model Using an Ensemble Kalman Filter and Three-Dimensional Variational Analysis

  2025cites this paper
• Grain for Green Project dominates greening in afforested areas rather than that in grass revegetation areas of the Loess Plateau, China—using Deep Crossing LSTM Age network

  2025cites this paper
• The Modular and Integrated Data Assimilation System at Environment and Climate Change Canada (MIDAS v3.9.1)

  2025influential citation
• Data assimilation using a global Girsanov nudged particle filter

  2025cites this paper
• Achievements in atmospheric sciences by the large-ensemble and high-resolution forecasting studies using the supercomputer Fugaku

  2025cites this paper
• Preserving nonlinear constraints in variational flow filtering data assimilation

  2025cites this paper
• The Atmosphere, Ocean and Sea Ice Coupled Model for Medium-range Weather Prediction

  2025cites this paper
• Impact of Assimilating GPS Precipitable Water Vapor on Simulations of Two North American Monsoon Convective Events Using Observing System Simulation Experiments

  2025cites this paper
• Learning Enhanced Ensemble Filters

  2025cites this paper
• TEDA: A lightweight Python framework for educational data assimilation

  2025cites this paper
• Ensemble-based model predictive control using data assimilation techniques

  2025cites this paper
• Predicting ice supersaturation for contrail avoidance: ensemble forecasting using ICON with two-moment ice microphysics

  2025cites this paper
• AMR-accelerated phase-field data assimilation for dendrite solidification

  2025cites this paper
• Model Training, Data Assimilation, and Forecast Experiments with a Hybrid Atmospheric Model that Incorporates Machine Learning

  2025cites this paper
• A Bayesian Characterization of Ensemble Kalman Updates

  2025influential citation
• Assessing the relative impacts of satellite ozone and its precursor observations to improve global tropospheric ozone analysis using multiple chemical reanalysis systems

  2025cites this paper
• Deterministic and ensemble forecasts of the Kuroshio south of Japan

  2025cites this paper
• IEnSF: Iterative Ensemble Score Filter for Reducing Error in Posterior Score Estimation in Nonlinear Data Assimilation

  2025cites this paper
• Assessing the Influence of Observations in Ensemble‐Based Data Assimilation Systems

  2025influential citation
• Using data assimilation tools to dissect GraphDOP

  2025cites this paper
• Observation error estimation in climate proxies with data assimilation and innovation statistics

  2025cites this paper
• Quantifying uncertainties in satellite NO2 superobservations for data assimilation and model evaluation

  2025cites this paper
• Error-correction across gauged and ungauged locations: A data assimilation-inspired approach to post-processing river discharge forecasts

  2025cites this paper
• The limits of knowing: An equilibrium between the observational gain and chaotic loss of information in data assimilation

  2025cites this paper
• Tracking daily NOx emissions from an urban agglomeration based on TROPOMI NO2 and a local ensemble transform Kalman filter

  2025cites this paper
• Appa: Bending Weather Dynamics with Latent Diffusion Models for Global Data Assimilation

  2025cites this paper