Scarcity of Labels in Non-Stationary Data Streams: A Survey

In a dynamic stream there is an assumption that the underlying process generating the stream is non-stationary and that concepts within the stream will drift and change as the stream progresses. Concepts learned by a classification model are prone to change and non-adaptive models are likely to deteriorate and become ineffective over time. The challenge of recognising and reacting to change in a stream is compounded by the scarcity of labels problem. This refers to the very realistic situation in which the true class label of an incoming point is not immediately available (or might never be available) or in situations where manually annotating data points are prohibitively expensive. In a high-velocity stream, it is perhaps impossible to manually label every incoming point and pursue a fully supervised approach. In this article, we formally describe the types of change, which can occur in a data-stream and then catalogue the methods for dealing with change when there is limited access to labels. We present an overview of the most influential ideas in the field along with recent advancements and we highlight trends, research gaps, and future research directions.

• Publication year

  2022

• Venue

  ACM Computing Surveys

• Publication date

  2022-01-21

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/3494832
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Algorithmic Fairness

  2023cited by this paper
• Can Active Learning Preemptively Mitigate Fairness Issues?

  2021cited by this paper
• RECORD: Resource Constrained Semi-Supervised Learning under Distribution Shift

  2020cited by this paper
• Can I Trust My Fairness Metric? Assessing Fairness with Unlabeled Data and Bayesian Inference

  2020cited by this paper
• Fairness in Machine Learning: A Survey

  2020cited by this paper
• Fairness in Semi-Supervised Learning: Unlabeled Data Help to Reduce Discrimination

  2020cited by this paper
• Fairness Constraints in Semi-supervised Learning

  2020cited by this paper
• Ethical principles in machine learning and artificial intelligence: cases from the field and possible ways forward

  2020cited by this paper
• Improved Long Short-Term Memory based anomaly detection with concept drift adaptive method for supporting IoT services

  2020cited by this paper
• Online RBM: Growing Restricted Boltzmann Machine on the fly for unsupervised representation

  2020cited by this paper
• Self-adaption neighborhood density clustering method for mixed data stream with concept drift

  2020cited by this paper
• Fair Active Learning

  2020cited by this paper
• SOINN+, a Self-Organizing Incremental Neural Network for Unsupervised Learning from Noisy Data Streams

  2020cited by this paper
• MVStream: Multiview Data Stream Clustering

  2020cited by this paper
• Hidden Risks of Machine Learning Applied to Healthcare: Unintended Feedback Loops Between Models and Future Data Causing Model Degradation

  2020cited by this paper
• No Free Lunch Theorem for concept drift detection in streaming data classification: A review

  2019cited by this paper
• A density-based competitive data stream clustering network with self-adaptive distance metric

  2019cited by this paper
• Autoencoders Covering Space as a Life-Long Classifier

  2019cited by this paper
• Leveraging Semi-Supervised Learning for Fairness using Neural Networks

  2019cited by this paper
• Classifier Selection for Highly Imbalanced Data Streams with Minority Driven Ensemble

  2019cited by this paper
• DEVDAN: Deep Evolving Denoising Autoencoder

  2019cited by this paper
• A buffer-based online clustering for evolving data stream

  2019cited by this paper
• Incremental semi-supervised learning on streaming data

  2019cited by this paper
• Generalizing from a Few Examples

  2019cited by this paper
• Proportionally Fair Clustering

  2019cited by this paper
• Ant Colony Stream Clustering: A Fast Density Clustering Algorithm for Dynamic Data Streams

  2019influential reference
• A Survey on Bias and Fairness in Machine Learning

  2019cited by this paper
• Synchronization-based clustering on evolving data stream

  2019cited by this paper
• AMANDA: Semi-supervised density-based adaptive model for non-stationary data with extreme verification latency

  2019cited by this paper
• Dynamic Feature Selection for Clustering High Dimensional Data Streams

  2019cited by this paper
• Assessing cognitive mental workload via EEG signals and an ensemble deep learning classifier based on denoising autoencoders

  2019cited by this paper
• Scalable Fair Clustering

  2019cited by this paper
• Optimizing Data Stream Representation: An Extensive Survey on Stream Clustering Algorithms

  2019cited by this paper
• Uncovering and Mitigating Algorithmic Bias through Learned Latent Structure

  2019cited by this paper
• Outlier and anomaly pattern detection on data streams

  2018cited by this paper
• Threaded ensembles of autoencoders for stream learning

  2018cited by this paper
• Fairness Through Computationally-Bounded Awareness

  2018cited by this paper
• Combining active learning with concept drift detection for data stream mining

  2018cited by this paper
• A Bi-Criteria Active Learning Algorithm for Dynamic Data Streams

  2018cited by this paper
• Online Active Learning in Data Stream Regression Using Uncertainty Sampling Based on Evolving Generalized Fuzzy Models

  2018cited by this paper
• Anomaly detection for medical images based on a one-class classification

  2018cited by this paper
• Analytical investigation of autoencoder-based methods for unsupervised anomaly detection in building energy data

  2018cited by this paper
• Autonomous Deep Learning: Continual Learning Approach for Dynamic Environments

  2018cited by this paper
• Active learning for classifying data streams with unknown number of classes

  2018cited by this paper
• Fair Clustering Through Fairlets

  2018cited by this paper
• Evolving Spiking Neural Networks for online learning over drifting data streams

  2018cited by this paper
• Evaluating Stream Classifiers with Delayed Labels Information

  2018cited by this paper
• FGCH: a fast and grid based clustering algorithm for hybrid data stream

  2018cited by this paper
• Invariant Representations without Adversarial Training

  2018cited by this paper
• Online anomaly detection with concept drift adaptation using recurrent neural networks

  2018cited by this paper
• Fully online clustering of evolving data streams into arbitrarily shaped clusters

  2017cited by this paper
• Ensemble learning for data stream analysis: A survey

  2017cited by this paper
• On applying the Restricted Boltzmann Machine to active concept drift detection

  2017cited by this paper
• Software defect prediction using stacked denoising autoencoders and two-stage ensemble learning

  2017cited by this paper
• Runaway Feedback Loops in Predictive Policing

  2017cited by this paper
• Encoder Based Lifelong Learning

  2017cited by this paper
• On-line active learning: A new paradigm to improve practical useability of data stream modeling methods

  2017influential reference
• Online Deep Learning: Learning Deep Neural Networks on the Fly

  2017cited by this paper
• LEVELIW: Learning extreme verification latency with importance weighting

  2017cited by this paper
• Adaptive random forests for evolving data stream classification

  2017cited by this paper
• Unsupervised deep embedding for novel class detection over data stream

  2017cited by this paper
• SAND: Semi-Supervised Adaptive Novel Class Detection and Classification over Data Stream

  2016influential reference
• An Adaptive Density Data Stream Clustering Algorithm

  2016cited by this paper
• MuDi-Stream: A multi density clustering algorithm for evolving data stream

  2016cited by this paper
• Discussion and review on evolving data streams and concept drift adapting

  2016cited by this paper
• An adaptive streaming active learning strategy based on instance weighting

  2016cited by this paper
• Learning under extreme verification latency quickly: FAST COMPOSE

  2016influential reference
• Recognizing input space and target concept drifts in data streams with scarcely labeled and unlabelled instances

  2016cited by this paper
• Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments

  2016cited by this paper
• A new Growing Neural Gas for clustering data streams

  2016cited by this paper
• Overcoming catastrophic forgetting in neural networks

  2016cited by this paper
• Matching Networks for One Shot Learning

  2016influential reference
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders

  2016cited by this paper
• Using sensitive personal data may be necessary for avoiding discrimination in data-driven decision models

  2016cited by this paper
• Deep learning in partially-labeled data streams

  2015cited by this paper
• Integrating new classes on the fly in evolving fuzzy classifier designs and their application in visual inspection

  2015cited by this paper
• Learning in Nonstationary Environments: A Survey

  2015influential reference
• A survey on data stream clustering and classification

  2015cited by this paper
• Recurrent Classifier Based on an Incremental Metacognitive-Based Scaffolding Algorithm

  2015cited by this paper
• Classification of Evolving Data Streams with Infinitely Delayed Labels

  2015influential reference
• Data Stream Classification Guided by Clustering on Nonstationary Environments and Extreme Verification Latency

  2015influential reference
• An ensemble of cluster-based classifiers for semi-supervised classification of non-stationary data streams

  2015cited by this paper
• Data stream clustering algorithms: A review

  2015cited by this paper
• An embarrassingly simple approach to zero-shot learning

  2015influential reference
• The Mechanical Turk

  2015cited by this paper
• The Variational Fair Autoencoder

  2015cited by this paper
• Machine Learning in Medicine

  2015cited by this paper
• Intelligence for Embedded Systems

  2014cited by this paper
• Certifying and Removing Disparate Impact

  2014cited by this paper
• Computational Intelligence Challenges and Applications on Large-Scale Astronomical Time Series Databases

  2014cited by this paper
• COMPOSE: A Semisupervised Learning Framework for Initially Labeled Nonstationary Streaming Data

  2014influential reference
• Active Learning With Drifting Streaming Data

  2014influential reference
• Core support extraction for learning from initially labeled nonstationary environments using COMPOSE

  2014cited by this paper
• DBSCAN: Past, present and future

  2014cited by this paper
• Mining Textual Stream with Partial Labeled Instances Using Ensemble Framework

  2014cited by this paper
• A survey on concept drift adaptation

  2014cited by this paper
• Clustering Based Active Learning for Evolving Data Streams

  2013cited by this paper
• A Survey on Multi-view Learning

  2013cited by this paper
• Learning to Detect Patterns of Crime

  2013cited by this paper
• Enhancing density-based clustering: Parameter reduction and outlier detection

  2013cited by this paper

• SLEADE: Disagreement-Based Semi-Supervised Learning for Sparsely Labeled Evolving Data Streams

  2026cites this paper
• Region-based deep metric learning for tackling class overlap in online semi-supervised data stream classification

  2026cites this paper
• A Feature Evolution Aware Classification Framework for Streaming Data using Dynamic Autoencoder and Ensembled Learning

  2025cites this paper
• Self-Organizing Recurrent Stochastic Configuration Networks for Nonstationary Data Modeling

  2025cites this paper
• A survey of streaming data anomaly detection in network security

  2025cites this paper
• Edge Intelligence with Spiking Neural Networks

  2025cites this paper
• Latency-Aware Online Continual Learning for Non-Stationary Data Streams

  2025cites this paper
• Transfer learning method based on cluster for drifting data stream classification

  2025cites this paper
• An Auto-Tuning Goal-Oriented Dynamic Feature Selection for Non-Stationary Message Stream Classification Using a Multi-Agent Approach

  2025cites this paper
• A Semi-Supervised Active Learning Neural Network for Data Streams With Concept Drift

  2025cites this paper
• Analysis of Descriptors of Concept Drift and Their Impacts

  2025cites this paper
• Online deep learning’s role in conquering the challenges of streaming data: a survey

  2025cites this paper
• Decentralized federated learning meets Physics-Informed Neural Networks

  2025cites this paper
• Adaptive Dynamic Thresholds for Unsupervised Joint Anomaly Detection and Trend Prediction

  2025cites this paper
• Flexible Windowing for Correlation-Aware Ranking in Anomalous Environments

  2025cites this paper
• A novel transfer learning method leveraging clustering and local variance for drifting data streams classification

  2025cites this paper
• Online transfer learning framework for label scarcity in evolving data streams

  2025cites this paper
• An Automated Approach to Collecting and Labeling Time Series Data for Event Detection Using Elastic Node Hardware

  2024cites this paper
• A reliable adaptive prototype-based learning for evolving data streams with limited labels

  2024cites this paper
• SALAD: A split active learning based unsupervised network data stream anomaly detection method using autoencoders

  2024cites this paper
• Solar Panel Segmentation: Self-Supervised Learning Solutions for Imperfect Datasets

  2024cites this paper
• A Comprehensive Review of Machine Learning Advances on Data Change: A Cross-Field Perspective

  2024cites this paper
• A systematic review on detection and adaptation of concept drift in streaming data using machine learning techniques

  2024cites this paper
• A Systematic Literature Review of Novelty Detection in Data Streams: Challenges and Opportunities

  2024cites this paper
• Active Learning for Data Quality Control: A Survey

  2024cites this paper
• Network security AIOps for online stream data monitoring

  2024cites this paper
• Concept drift adaptation with scarce labels: A novel approach based on diffusion and adversarial learning

  2024cites this paper
• AOCBLS: A novel active and online learning system for ECG arrhythmia classification with less labeled samples

  2024cites this paper
• Self-Organizing Recurrent Stochastic Configuration Networks for Nonstationary Data Modelling

  2024cites this paper
• Resilience to the Flowing Unknown: An Open Set Recognition Framework for Data Streams

  2024cites this paper
• Combining self-labeling and demand based active learning for non-stationary data streams

  2023cites this paper
• An Evolving Population Approach to Data-Stream Classification with Extreme Verification Latency

  2023cites this paper
• Managing the unknown: a survey on Open Set Recognition and tangential areas

  2023cites this paper
• Scaling-up Memristor Monte Carlo with magnetic domain-wall physics

  2023cites this paper
• Semi-supervised One-pass Learning under Distribution Shift

  2023cites this paper
• Dynamic budget allocation for sparsely labeled drifting data streams

  2023cites this paper
• Integrating a Rule-Based Approach to Malware Detection with an LSTM-Based Feature Selection Technique

  2023cites this paper
• Platform and Hardware Independent Reliable Signal Processing

  2023cites this paper
• Semi-supervised federated learning on evolving data streams

  2023cites this paper
• Stream-based Active Learning with Verification Latency in Non-stationary Environments

  2022cites this paper
• Supervised Learning from Data Streams: An Overview and Update

  2022cites this paper
• Environmental benefits of China's entertainment and media market: implications for sustainability

  2022cites this paper
• Information resources estimation for accurate distribution-based concept drift detection

  2022cites this paper