Latent data assimilation with non‐explicit observation operator in hydrology

Natural hazards can cause significant damage to human life and property. Among them, floods are one of the most severe and frequent natural disasters, making flood prediction crucial. River discharge is an essential factor in causing floods, so accurate and fast prediction of river discharge is crucial for flood mitigation. Data assimilation (DA) as a method of combining different sources of data (e.g., state field and observations) has the ability to estimate the possible states of river discharge. However, DA on high‐dimensional data such as river discharge can be computationally expensive. Furthermore, when the DA process lacks explicit mappings from the state field to the observations, DA cannot be conducted effectively. In this work, we design a latent neural mapping (LNM) in the form of a neural network (NN) as the observation operator and integrate this within a three‐dimensional variational data assimilation (3D‐Var) framework. By operating within latent space, the resulting approach helps mitigate computational costs and allows us to run DA within seconds despite the high‐dimensional data. In addition, several alternative NNs are employed to build mapping functions, which map data from the state space to the observation space (and vice versa), and benchmarked against the latent‐space‐based LNM approach. We test the LNM with real river discharge data from the UK and Ireland. The National River Flow Archive (NRFA) dataset provides the observations, and the data provided by a surrogate model from the European Flood Awareness System (EFAS) dataset serves as the state field. LNM outperforms the alternative methods in terms of accuracy and efficiency. The LNM developed can be applied to areas other than hydrology to integrate data efficiently with models.

• Publication year

  2025

• Venue

  Quarterly Journal of the Royal Meteorological Society

• Publication date

  2025-08-27

• Fields of study

  Not labeled

• Identifiers
  DOI 10.1002/qj.5009
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Simultaneous Inference of Sea Ice State and Surface Emissivity Model Using Machine Learning and Data Assimilation

  2024cited by this paper
• Efficient deep data assimilation with sparse observations and time-varying sensors

  2023cited by this paper
• 3D‐Var data assimilation using a variational autoencoder

  2023cited by this paper
• Calibrated river-level estimation from river cameras using convolutional neural networks

  2023cited by this paper
• Learning skillful medium-range global weather forecasting

  2023cited by this paper
• Co-design and co-production of flood forecast products: Summary of a hybrid workshop

  2023cited by this paper
• High temporal resolution urban flood prediction using attention-based LSTM models

  2023cited by this paper
• Fast Prediction of Urban Flooding Water Depth Based on CNN−LSTM

  2023cited by this paper
• A machine learning approach to the observation operator for satellite radiance data assimilation

  2023cited by this paper
• Machine Learning With Data Assimilation and Uncertainty Quantification for Dynamical Systems: A Review

  2023cited by this paper
• Improving LSTM hydrological modeling with spatiotemporal deep learning and multi-task learning: a case study of three mountainous areas on the Tibetan Plateau

  2023cited by this paper
• Deep Learning Augmented Data Assimilation: Reconstructing Missing Information With Convolutional Autoencoders

  2022cited by this paper
• NeuralHydrology - A Python library for Deep Learning research in hydrology

  2022cited by this paper
• Generalised Latent Assimilation in Heterogeneous Reduced Spaces with Machine Learning Surrogate Models

  2022cited by this paper
• The future of Earth system prediction: Advances in model-data fusion

  2022cited by this paper
• Prediction of Flow Based on a CNN-LSTM Combined Deep Learning Approach

  2022cited by this paper
• Multi-dimensional hydrological–hydraulic model with variational data assimilation for river networks and floodplains

  2022cited by this paper
• Optimal and fast field reconstruction with reduced basis and limited observations: Application to reactor core online monitoring

  2021cited by this paper
• Evaluating the impact of post-processing medium-range ensemble streamflow forecasts from the European Flood Awareness System

  2021cited by this paper
• Data Learning: Integrating Data Assimilation and Machine Learning

  2021cited by this paper
• EMO-5: A high-resolution multi-variable gridded meteorological data set for Europe

  2021cited by this paper
• Hydrological Concept Formation inside Long Short-Term Memory (LSTM) networks

  2021cited by this paper
• Deep learning for automated river-level monitoring through river-camera images: an approach based on water segmentation and transfer learning

  2021cited by this paper
• Integrating Recurrent Neural Networks With Data Assimilation for Scalable Data‐Driven State Estimation

  2021cited by this paper
• Causes, impacts and patterns of disastrous river floods

  2021cited by this paper
• Ensemble streamflow data assimilation using WRF-Hydro and DART: novel localization and inflation techniques applied to Hurricane Florence flooding

  2021cited by this paper
• A novel ensemble-based conceptual-data-driven approach for improved streamflow simulations

  2021cited by this paper
• The Role of Factors Affecting Flood Hazard Zoning Using Analytical Hierarchy Process: A Review

  2021cited by this paper
• Observation data compression for variational assimilation of dynamical systems

  2021cited by this paper
• A Neural Network-Based Observation Operator for Coupled Ocean-Acoustic Variational Data Assimilation

  2021cited by this paper
• Adversarial autoencoders and adversarial LSTM for improved forecasts of urban air pollution simulations

  2021cited by this paper
• Deep denoising autoencoder for seismic random noise attenuation

  2020cited by this paper
• Recent Advances in Real-Time Pluvial Flash Flood Forecasting

  2020cited by this paper
• An inverse-distance-based fitting term for 3D-Var data assimilation in nuclear core simulation

  2020cited by this paper
• Streamflow Prediction Using Deep Learning Neural Network: Case Study of Yangtze River

  2020cited by this paper
• Recent developments in fast and scalable inverse modeling and data assimilation methods in hydrology

  2020cited by this paper
• Hybrid Neural Network - Variational Data Assimilation algorithm to infer river discharges from SWOT-like data

  2020cited by this paper
• Error covariance tuning in variational data assimilation: application to an operating hydrological model

  2020cited by this paper
• Integrating field observations and process-based modeling to predict watershed water quality under environmental perturbations

  2020cited by this paper
• Data Assimilation in the Latent Space of a Neural Network

  2020cited by this paper
• Efficient Data Assimilation in High-Dimensional Hydrologic Modeling through Optimal Spatial Clustering

  2019cited by this paper
• Enhanced flood forecasting through ensemble data assimilation and joint state-parameter estimation

  2019cited by this paper
• Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model

  2019cited by this paper
• Optimal reduced space for Variational Data Assimilation

  2019cited by this paper
• PyTorch: An Imperative Style, High-Performance Deep Learning Library

  2019influential reference
• DEVDAN: Deep Evolving Denoising Autoencoder

  2019cited by this paper
• Bayesian Assimilation of Multiscale Precipitation Data and Sparse Ground Gauge Observations in Mountainous Areas

  2018cited by this paper
• On the representation error in data assimilation

  2018cited by this paper
• Flood vulnerability, risk, and social disadvantage: current and future patterns in the UK

  2018cited by this paper
• Rainfall–runoff modelling using Long Short-Term Memory (LSTM) networks

  2018cited by this paper
• Local SGD Converges Fast and Communicates Little

  2018cited by this paper
• Decoupled Weight Decay Regularization

  2017cited by this paper
• Data assimilation in the geosciences: An overview of methods, issues, and perspectives

  2017cited by this paper
• Earth System Modeling 2.0: A Blueprint for Models That Learn From Observations and Targeted High‐Resolution Simulations

  2017cited by this paper
• High-magnitude flooding across Britain since AD 1750

  2017cited by this paper
• A review of operational methods of variational and ensemble‐variational data assimilation

  2017cited by this paper
• Integrating hydrological modelling, data assimilation and cloud computing for real-time management of water resources

  2017cited by this paper
• Tracking the model: Data assimilation by artificial neural network

  2016cited by this paper
• Data Assimilation: Methods, Algorithms, and Applications

  2016cited by this paper
• Medical Image Denoising Using Convolutional Denoising Autoencoders

  2016cited by this paper
• Hybridizing Bayesian and variational data assimilation for high-resolution hydrologic forecasting

  2016cited by this paper
• Data assimilation framework: Linking an open data assimilation library (OpenDA) to a widely adopted model interface (OpenMI)

  2014cited by this paper
• Improved Neural Network Model and Its Application in Hydrological Simulation

  2014cited by this paper
• Reduced Order Models and Data Assimilation for Hydrological Applications

  2013cited by this paper
• Representation Learning: A Review and New Perspectives

  2012cited by this paper
• Contractive Auto-Encoders: Explicit Invariance During Feature Extraction

  2011cited by this paper
• Identification of geographic runoff sources in a data sparse region: hydrological processes and the limitations of tracer‐based approaches

  2010cited by this paper
• Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion

  2010cited by this paper
• A review of forecast error covariance statistics in atmospheric variational data assimilation. I: Characteristics and measurements of forecast error covariances

  2008cited by this paper
• Supporting Online Material for Reducing the Dimensionality of Data with Neural Networks

  2006cited by this paper
• The Ensemble Kalman Filter: theoretical formulation and practical implementation

  2003cited by this paper
• The ECMWF implementation of three‐dimensional variational assimilation (3D‐Var). I: Formulation

  1998cited by this paper
• Gradient-based learning applied to document recognition

  1998influential reference
• ARMA Models and the Box–Jenkins Methodology

  1997cited by this paper
• Long Short-Term Memory

  1997cited by this paper
• Learning representations by back-propagating errors

  1986cited by this paper
• River flow forecasting through conceptual models part I — A discussion of principles☆

  1970influential reference
• Distributed under Creative Commons Cc-by 4.0 Scikit-image: Image Processing in Python

  year unknowncited by this paper

• No citing papers are available for this paper.