Toward automating post processing of aquatic sensor data

Sensors measuring environmental phenomena at high frequency commonly report anomalies related to fouling, sensor drift and calibration, and datalogging and transmission issues. Suitability of data for analyses and decision making often depends on manual review and adjustment of data. Machine learning techniques have potential to automate identification and correction of anomalies, streamlining the quality control process. We explored approaches for automating anomaly detection and correction of aquatic sensor data for implementation in a Python package (PyHydroQC). We applied both classical and deep learning time series regression models that estimate values, identify anomalies based on dynamic thresholds, and offer correction estimates. Techniques were developed and performance assessed using data reviewed, corrected, and labeled by technicians in an aquatic monitoring use case. Auto-Regressive Integrated Moving Average (ARIMA) consistently performed best, and aggregating results from multiple models improved detection. PyHydroQC includes custom functions and a workflow for anomaly detection and correction.

• Publication year

  2021

• Venue

  Environmental Modelling & Software

• Publication date

  2021-07-23

• Fields of study

  Computer Science, Environmental Science

• Identifiers
  DOI 10.31223/x5z62x
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Anomaly Detection Based on Convolutional Recurrent Autoencoder for IoT Time Series

  2022cited by this paper
• Active learning for anomaly detection in environmental data

  2020cited by this paper
• Anomaly Detection for IoT Time-Series Data: A Survey

  2020cited by this paper
• Comparison of stochastic and machine learning methods for multi-step ahead forecasting of hydrological processes

  2019cited by this paper
• A Feature‐Based Procedure for Detecting Technical Outliers in Water‐Quality Data From In Situ Sensors

  2019cited by this paper
• Learning Ensembles of Anomaly Detectors on Synthetic Data

  2019cited by this paper
• Outlier Detection in Non-stationary Data Streams

  2019cited by this paper
• Time Series FeatuRe Extraction on basis of Scalable Hypothesis tests (tsfresh - A Python package)

  2018cited by this paper
• Environmental Data Science

  2018cited by this paper
• Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding

  2018cited by this paper
• Forecasting at Scale

  2018cited by this paper
• Conference on Intelligent Computation in Manufacturing Engineering , 18-20 July 2018 , Gulf of Naples , Italy Anomaly Detection in Discrete Manufacturing Using Self-Learning Approaches

  2018cited by this paper
• A framework for automated anomaly detection in high frequency water-quality data from in situ sensors.

  2018cited by this paper
• Assessing subjectivity in environmental sensor data post processing via a controlled experiment

  2018cited by this paper
• Introduction to Data

  2017influential reference
• Designing and Implementing a Network for Sensing Water Quality and Hydrology across Mountain to Urban Transitions

  2017cited by this paper
• Multivariate time series anomaly detection: A framework of Hidden Markov Models

  2017cited by this paper
• Unsupervised real-time anomaly detection for streaming data

  2017cited by this paper
• A survey on pre-processing techniques: Relevant issues in the context of environmental data mining

  2016cited by this paper
• LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection

  2016cited by this paper
• Emerging Tools for Continuous Nutrient Monitoring Networks: Sensors Advancing Science and Water Resources Protection

  2016cited by this paper
• A user-driven case-based reasoning tool for infilling missing values in daily mean river flow records

  2016cited by this paper
• Sensors in the Stream: The High-Frequency Wave of the Present.

  2016cited by this paper
• Open source software for visualization and quality control of continuous hydrologic and water quality sensor data

  2015cited by this paper
• A data management and publication workflow for a large-scale, heterogeneous sensor network

  2015cited by this paper
• LSTM: A Search Space Odyssey

  2015cited by this paper
• Guidelines and Standard Procedures for Continuous Water-Quality Monitors: Station Operation, Record Computation, and Data Reporting

  2014cited by this paper
• Quantity is Nothing without Quality: Automated QA/QC for Streaming Environmental Sensor Data

  2013cited by this paper
• Automated Weather Sensor Quality Control

  2012cited by this paper
• Automated quality control methods for sensor data: a novel observatory approach

  2012cited by this paper
• Scikit-learn: Machine Learning in Python

  2011cited by this paper
• Environmental sensor anomaly detection using learning machines

  2011cited by this paper
• REAL-TIME QUALITY CONTROL (QC) PROCESSING, NOTIFICATION, AND VISUALIZATION SERVICES, SUPPORTING DATA MANAGEMENT OF THE INTELLIGENT RIVER©

  2010cited by this paper
• Statsmodels: Econometric and Statistical Modeling with Python

  2010cited by this paper
• Quality Assurance Procedures for Mesoscale Meteorological Data

  2010cited by this paper
• Anomaly detection in streaming environmental sensor data: A data-driven modeling approach

  2010cited by this paper
• Anomaly detection: A survey

  2009cited by this paper
• Dynamic , Rule-based Quality Control Framework for Real-time Sensor Data

  2008cited by this paper
• A relational model for environmental and water resources data

  2008cited by this paper
• Probabilistic Models for Anomaly Detection in Remote Sensor Data Streams

  2007cited by this paper
• Real‐time Bayesian anomaly detection in streaming environmental data

  2007cited by this paper
• AUTOMATED FAULT DETECTION FOR IN-SITU ENVIRONMENTAL SENSORS

  2006cited by this paper
• Environmental Sensor Networks: A revolution in the earth system science?

  2006cited by this paper
• A method for automatic validation of long time series of data in urban hydrology.

  2002cited by this paper
• A quality-control method for physical and chemical monitoring data. Application to dissolved oxygen levels in the river Loire (France)

  2001cited by this paper

• Linking water quality assessment to source apportionment with machine learning-assisted WQI, PMF, and SOM: A case study of the Jinma River basin

  2026cites this paper
• A critical review of statistical, signal processing and machine learning methods for continuous and high-frequency water quality data improvement

  2026cites this paper
• Multidecadal trends in CO2 evasion and aquatic metabolism in a large temperate river

  2025cites this paper
• Interpretable Quality Control of Sparsely Distributed Environmental Sensor Networks Using Graph Neural Networks

  2025cites this paper
• Insights on nitrate transport in a shallow, sandy aquifer at various temporal and spatial scales

  2025cites this paper
• Hydrologic information systems: An introductory overview

  2025cites this paper
• Best practice in high-frequency water quality monitoring for improved management and assessment; a novel decision workflow

  2025influential citation
• Unsupervised Anomaly Detection in Spatio‐Temporal Stream Network Sensor Data

  2024cites this paper
• MitH: A framework for Mitigating Hygroscopicity in low-cost PM sensors

  2024cites this paper
• Bayesian design for sampling anomalous spatio-temporal data

  2024cites this paper
• SentemQC - A novel and cost-efficient method for quality assurance and quality control of high-resolution frequency sensor data in fresh waters

  2024cites this paper
• System for automated Quality Control (SaQC) to enable traceable and reproducible data streams in environmental science

  2023cites this paper
• Machine Learning-Assisted, Process-Based Quality Control for Detecting Compromised Environmental Sensors.

  2023cites this paper
• Value and limitations of machine learning in high-frequency nutrient data for gap-filling, forecasting, and transport process interpretation

  2023cites this paper
• Advances in Catchment Science, Hydrochemistry, and Aquatic Ecology Enabled by High-Frequency Water Quality Measurements

  2023cites this paper
• Reconstructing Missing and Anomalous Data Collected from High-Frequency In-Situ Sensors in Fresh Waters

  2021cites this paper
• Multidecadal trends in CO 2 evasion and aquatic metabolism in a large 1 temperate river 2

  year unknowncites this paper