Benchmarking Unsupervised Outlier Detection with Realistic Synthetic Data

Benchmarking unsupervised outlier detection is difficult. Outliers are rare, and existing benchmark data contains outliers with various and unknown characteristics. Fully synthetic data usually consists of outliers and regular instances with clear characteristics and thus allows for a more meaningful evaluation of detection methods in principle. Nonetheless, there have only been few attempts to include synthetic data in benchmarks for outlier detection. This might be due to the imprecise notion of outliers or to the difficulty to arrive at a good coverage of different domains with synthetic data. In this work, we propose a generic process for the generation of datasets for such benchmarking. The core idea is to reconstruct regular instances from existing real-world benchmark data while generating outliers so that they exhibit insightful characteristics. We propose and describe a generic process for the benchmarking of unsupervised outlier detection, as sketched so far. We then describe three instantiations of this generic process that generate outliers with specific characteristics, like local outliers. To validate our process, we perform a benchmark with state-of-the-art detection methods and carry out experiments to study the quality of data reconstructed in this way. Next to showcasing the workflow, this confirms the usefulness of our proposed process. In particular, our process yields regular instances close to the ones from real data. Summing up, we propose and validate a new and practical process for the benchmarking of unsupervised outlier detection.

• Publication year

  2020

• Venue

  ACM Transactions on Knowledge Discovery from Data

• Publication date

  2020-04-15

• Fields of study

  Mathematics, Computer Science

• Identifiers
  DOI 10.1145/3441453arXiv 2004.06947
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Generative Adversarial Networks

  2021cited by this paper
• Out-of-distribution Detection in Classifiers via Generation

  2019cited by this paper
• Method evaluation, parameterization, and result validation in unsupervised data mining: A critical survey

  2019cited by this paper
• MDCGen: Multidimensional Dataset Generator for Clustering

  2019influential reference
• SynC: A Unified Framework for Generating Synthetic Population with Gaussian Copula

  2019cited by this paper
• Is this Real?: Generating Synthetic Data that Looks Real

  2019cited by this paper
• Learning Vine Copula Models For Synthetic Data Generation

  2018influential reference
• Statistical Data Generation Using Sample Data

  2018cited by this paper
• A comparative evaluation of outlier detection algorithms: Experiments and analyses

  2018influential reference
• Hyperparameter selection of one-class support vector machine by self-adaptive data shifting

  2018cited by this paper
• Realistic Synthetic Data Generation: The ATEN Framework

  2018cited by this paper
• There and back again: Outlier detection between statistical reasoning and data mining algorithms

  2018cited by this paper
• A generic approach to nonparametric function estimation with mixed data

  2017cited by this paper
• Hiding outliers in high-dimensional data spaces

  2017cited by this paper
• batchtools: Tools for R to work on batch systems

  2017influential reference
• Fault Detection and Diagnosis for Nonlinear and Non-Gaussian Processes Based on Copula Subspace Division

  2017cited by this paper
• Feature projection k-NN classifier model for imbalanced and incomplete medical data

  2016influential reference
• mclust 5: Clustering, Classification and Density Estimation Using Gaussian Finite Mixture Models

  2016cited by this paper
• On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study

  2016influential reference
• A Comparative Evaluation of Unsupervised Anomaly Detection Algorithms for Multivariate Data

  2016influential reference
• The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets

  2015cited by this paper
• A Meta-Analysis of the Anomaly Detection Problem

  2015influential reference
• Truncation of vine copulas using fit indices

  2015cited by this paper
• Generative Adversarial Nets

  2014cited by this paper
• Systematic construction of anomaly detection benchmarks from real data

  2013influential reference
• An n-Spheres Based Synthetic Data Generator for Supervised Classification

  2013cited by this paper
• MixSim: An R Package for Simulating Data to Study Performance of Clustering Algorithms

  2012influential reference
• A weighted k-nearest neighbor density estimate for geometric inference

  2011influential reference
• Synthetic Generation of High-Dimensional Datasets

  2011cited by this paper
• A Framework for Generating Data to Simulate Application Scoring

  2011cited by this paper
• A Bayes-true data generator for evaluation of supervised and unsupervised learning methods

  2011influential reference
• Simulating Data to Study Performance of Finite Mixture Modeling and Clustering Algorithms

  2010influential reference
• Pair-copula constructions of multiple dependence

  2009influential reference
• Isolation Forest

  2008cited by this paper
• A Synthetic Data Generator for Clustering and Outlier Analysis

  2006influential reference
• Generation of Random Clusters with Specified Degree of Separation

  2006influential reference
• OCLUS: An Analytic Method for Generating Clusters with Known Overlap

  2005influential reference
• A Classification Framework for Anomaly Detection

  2005cited by this paper
• The Elements of Statistical Learning: Data Mining, Inference, and Prediction

  2004influential reference
• Using statistical image models for objective evaluation of spot detection in two‐dimensional gels

  2003cited by this paper
• Resampling approach for anomaly detection in multispectral images

  2003cited by this paper
• Synthesizing test data for fraud detection systems

  2003cited by this paper
• Uniform Object Generation for Optimizing One-class Classifiers

  2002cited by this paper
• Fast Outlier Detection in High Dimensional Spaces

  2002cited by this paper
• Efficient algorithms for mining outliers from large data sets

  2000cited by this paper
• LOF: identifying density-based local outliers

  2000influential reference
• A Method for Generating Simulated Plasmodes and Artificial Test Clusters with User-Defined Shape, Size, and Orientation.

  1999cited by this paper
• DataGen: a generator of datasets for evaluation of classification algorithms

  1998influential reference
• Estimating the Bayes error rate through classifier combining

  1996cited by this paper
• A plant-wide industrial process control problem

  1993cited by this paper
• An algorithm for generating artificial test clusters

  1985influential reference
• A NEW MEASURE OF RANK CORRELATION

  1938influential reference

• Why the Counterintuitive Phenomenon of Likelihood Rarely Appears in Tabular Anomaly Detection with Deep Generative Models?

  2026cites this paper
• Machine learning in modern database systems: Techniques, architectures, and deployment challenges

  2026cites this paper
• MacrOData: New Benchmarks of Thousands of Datasets for Tabular Outlier Detection

  2026cites this paper
• Unsupervised Detection of Anomalous Commits in Software Repositories

  2025cites this paper
• A robust unsupervised method for outlier set detection

  2025cites this paper
• Real-Time Decorrelation-Based Anomaly Detection for Multivariate Time Series

  2025cites this paper
• Investigating Mask-aware Prototype Learning for Tabular Anomaly Detection

  2025cites this paper
• Sensor Fusion Enhances Anomaly Detection in a Flood Forecasting System

  2025cites this paper
• Ensuring real-time data integrity in Smart Building Applications: A systematic end-to-end comprehensive pipeline evaluated in numerous real-life cases

  2025cites this paper
• Real-Time Anomaly Detection with Synthetic Anomaly Monitoring (SAM)

  2025cites this paper
• TAD-Bench: A Comprehensive Benchmark for Embedding-Based Text Anomaly Detection

  2025cites this paper
• Survey and Benchmark of Anomaly Detection in Business Processes

  2025cites this paper
• Fuzzy Granule Density-Based Outlier Detection With Multi-Scale Granular Balls

  2025cites this paper
• Advancing rock mass classification using machine learning approach

  2025cites this paper
• Fall detection system using tabular GAN for data augmentation with integration of isolation forest model

  2025cites this paper
• Integrating granular computing with density estimation for anomaly detection in high-dimensional heterogeneous data

  2024cites this paper
• A Closer Look at AUROC and AUPRC under Class Imbalance

  2024cites this paper
• Using Autonomous Outlier Detection Methods for Thermophysical Property Data

  2024cites this paper
• Benchmarking Anomaly Detection Algorithms: Deep Learning and Beyond

  2024cites this paper
• Learning-Based Link Anomaly Detection in Continuous-Time Dynamic Graphs

  2024cites this paper
• Masked Memory Network for Semi-Supervised Anomaly Detection in Internet of Things

  2024cites this paper
• Hyperspectral Unmixing Under Endmember Variability: A Variational Inference Framework

  2024cites this paper
• Adjacent Image Augmentation and Its Framework for Self-Supervised Learning in Anomaly Detection

  2024cites this paper
• Unsupervised Parameter-free Outlier Detection using HDBSCAN* Outlier Profiles

  2024cites this paper
• What do anomaly scores actually mean? Dynamic characteristics beyond accuracy

  2024cites this paper
• ADOD: Adaptive Density Outlier Detection

  2024cites this paper
• Anomaly detection in streaming data: A comparison and evaluation study

  2023cites this paper
• Understanding the limitations of self-supervised learning for tabular anomaly detection

  2023cites this paper
• ADGym: Design Choices for Deep Anomaly Detection

  2023cites this paper
• Procedure code overutilization detection from healthcare claims using unsupervised deep learning methods

  2023cites this paper
• Benchmarking Anomaly Detection Algorithms using Hyperparameter Optimization

  2023cites this paper
• Anomaly Detection with Score Distribution Discrimination

  2023cites this paper
• Unleashing the Potential of Unsupervised Deep Outlier Detection through Automated Training Stopping

  2023cites this paper
• AD-MERCS: Modeling Normality and Abnormality in Unsupervised Anomaly Detection

  2023cites this paper
• Outlier detection based on extreme value theory and applications

  2023cites this paper
• From Explanation to Action: An End-to-End Human-in-the-loop Framework for Anomaly Reasoning and Management

  2023cites this paper
• Unsupervised anomaly detection algorithms on real-world data: how many do we need?

  2023influential citation
• Unsupervised outlier detection using neural network-based mixtures of probabilistic principal component analyzers for building chiller fault diagnosis

  2022cites this paper
• ADBench: Anomaly Detection Benchmark

  2022influential citation
• Benchmarking Node Outlier Detection on Graphs

  2022cites this paper
• Sparx: Distributed Outlier Detection at Scale

  2022cites this paper
• BOND: Benchmarking Unsupervised Outlier Node Detection on Static Attributed Graphs

  2022cites this paper
• No Pattern, No Recognition: a Survey about Reproducibility and Distortion Issues of Text Clustering and Topic Modeling

  2022influential citation
• Simulated Workbench Design for Characterisation and Selection of Appropriate Outlier Ensemble Algorithm

  2022influential citation
• Toward Unsupervised Outlier Model Selection

  2022cites this paper
• C-AllOut: Catching & Calling Outliers by Type

  2021cites this paper
• The Effect of Hyperparameter Tuning on the Comparative Evaluation of Unsupervised Anomaly Detection Methods

  2021cites this paper