Deep Learning for Unsupervised Insider Threat Detection in Structured Cybersecurity Data Streams

Analysis of an organization's computer network activity is a key component of early detection and mitigation of insider threat, a growing concern for many organizations. Raw system logs are a prototypical example of streaming data that can quickly scale beyond the cognitive power of a human analyst. As a prospective filter for the human analyst, we present an online unsupervised deep learning approach to detect anomalous network activity from system logs in real time. Our models decompose anomaly scores into the contributions of individual user behavior features for increased interpretability to aid analysts reviewing potential cases of insider threat. Using the CERT Insider Threat Dataset v6.2 and threat detection recall as our performance metric, our novel deep and recurrent neural network models outperform Principal Component Analysis, Support Vector Machine and Isolation Forest based anomaly detection baselines. For our best model, the events labeled as insider threat activity in our dataset had an average anomaly score in the 95.53 percentile, demonstrating our approach's potential to greatly reduce analyst workloads.

• Publication year

  2017

• Venue

  AAAI Workshops

• Publication date

  2017-10-02

• Fields of study

  Mathematics, Computer Science

• Identifiers
  arXiv 1710.00811
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection

  2016cited by this paper
• AI^2: Training a Big Data Machine to Defend

  2016cited by this paper
• Anomaly detection in ECG time signals via deep long short-term memory networks

  2015cited by this paper
• Non-linear prediction with LSTM recurrent neural networks for acoustic novelty detection

  2015cited by this paper
• A novel approach for automatic acoustic novelty detection using a denoising autoencoder with bidirectional LSTM neural networks

  2015cited by this paper
• Supervised and Unsupervised methods to detect Insider Threat from Enterprise Social and Online Activity Data

  2015cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Generating Test Data for Insider Threat Detectors

  2014cited by this paper
• Bridging the Gap: A Pragmatic Approach to Generating Insider Threat Data

  2013cited by this paper
• Anomaly Detection for Discrete Sequences: A Survey

  2012cited by this paper
• Probabilistic reasoning for streaming anomaly detection

  2012cited by this paper
• Scikit-learn: Machine Learning in Python

  2011cited by this paper
• Intelligence-Driven Computer Network Defense Informed by Analysis of Adversary Campaigns and Intrusion Kill Chains

  2010cited by this paper
• Outside the Closed World: On Using Machine Learning for Network Intrusion Detection

  2010cited by this paper
• Isolation Forest

  2008cited by this paper
• Novelty detection: a review - part 2: : neural network based approaches

  2003cited by this paper
• A Novel Anomaly Detection Scheme Based on Principal Component Classifier

  2003cited by this paper
• Estimating the Support of a High-Dimensional Distribution

  2001cited by this paper
• Intrusion Detection with Neural Networks

  1997cited by this paper
• Long Short-Term Memory

  1997cited by this paper
• Mixture Density Networks

  1994cited by this paper
• A neural network component for an intrusion detection system

  1992cited by this paper

• A Review of User Account Anomaly Detection and Insider Threat Detection Techniques

  2026cites this paper
• Securing cloud data with machine learning: trends, gaps, and performance metrics

  2026cites this paper
• Digital Forensic Readiness to Mitigate Insider Threats in the SaaS Cloud Environment

  2026influential citation
• Mask-Based Window-Level Insider Threat Detection for Campaign Discovery

  2026cites this paper
• DPI-ITD: A Dual-Perspective Information-Driven Framework for Insider Threat Detection in IoT Systems

  2025cites this paper
• DDCC: Synergizing Denoising Diffusion Probabilistic Models and Curriculum-Based Complexity Control for Insider Threat Detection

  2025cites this paper
• Unsupervised anomaly detection on cybersecurity data streams: a case with BETH dataset

  2025cites this paper
• Adaptive AI-Driven Cybersecurity Framework for Threat Detection and Response in Networks

  2025cites this paper
• Machine Learning-Based Network Monitoring for Cybersecurity Threat Detection

  2025cites this paper
• AI-Powered Cybersecurity Framework: From Reactive Defense to Proactive, Resilient, and Autonomous Protection

  2025cites this paper
• A systematic review on insider threat detection using natural language processing

  2025influential citation
• Behavior‐based Confidence Scoring to Support Access Management in Zero Trust Systems

  2025influential citation
• Measuring the Human Risk Factor: Developing Predictive Models for Insider Threat Detection Using Behavioral Analytics

  2025influential citation
• Leveraging large language models for SQL behavior-based database intrusion detection

  2025cites this paper
• Insight-LLM: LLM-enhanced Multi-view Fusion in Insider Threat Detection

  2025cites this paper
• AI-Driven Cybersecurity Threat Detection: Building Resilient Defense Systems Using Predictive Analytics

  2025influential citation
• Real-Time Detection of Insider Threats Using Behavioral Analytics and Deep Evidential Clustering

  2025cites this paper
• Detection of computer attacks based on sonification of network traffic 252-263

  2025influential citation
• Contrastive Learning with Knowledge-Enhanced Prompts for Insider Threat Detection

  2025cites this paper
• Enhancing Cybersecurity: Classification and Prediction of Significant Cyber Incidents with an Optimized Seq2seq Model

  2025cites this paper
• Towards Robust SSH Anomaly Detection: A Deep Learning Comparative Analysis

  2025cites this paper
• Insider Threat Detection Using GCN and Bi-LSTM with Explicit and Implicit Graph Representations

  2025cites this paper
• Insider Threat Pattern Detection Using Deep Learning to Evaluate Cyber Value at Risk (CVaR)

  2025cites this paper
• Large Language Model-driven Framework for Wireless Network Security Underlying uRLLC

  2025cites this paper
• Endpoint Security Agent: A Comprehensive Approach to Real-Time System Monitoring and Threat Detection

  2025influential citation
• Enhancing System Security Monitoring with AI: Integrating the Anomaly Transformer and Prometheus for Real-Time Anomaly Detection

  2025cites this paper
• Enhancing Cybersecurity Against Side-Channel Attacks: an Improved Cnn-Based Threat Detection Model Using Nadam Optimizer

  2025cites this paper
• Optimizing IoT network security: machine learning approaches for smart village intrusion detection

  2025cites this paper
• A Survey of Open-Source Tools for Transmission and Distribution Systems Research

  2025cites this paper
• Leveraging Deep Learning for Access Control and Data Security in Cloud Environments

  2024cites this paper
• LAN: Learning Adaptive Neighbors for Real-Time Insider Threat Detection

  2024influential citation
• A Comparative Analysis of Deep Learning Based Techniques for Cyber Security

  2024cites this paper
• A Lower-Cost Insider Threat Detection Framework Based on User Cluster and Behavior Feature Analysis

  2024cites this paper
• Reconsider Time Series Analysis for Insider Threat Detection

  2024cites this paper
• Comparative evaluation of data imbalance addressing techniques for CNN-based insider threat detection

  2024cites this paper
• Machine learning approaches to detect, prevent and mitigate malicious insider threats: State-of-the-art review

  2024cites this paper
• Sentiment classification for insider threat identification using metaheuristic optimized machine learning classifiers

  2024cites this paper
• TabSec: A Collaborative Framework for Novel Insider Threat Detection

  2024cites this paper
• AD-NEv++ - The multi-architecture neuroevolution-based multivariate anomaly detection framework

  2024cites this paper
• Detecting Lateral Movement in Advanced Persistent Threats Based on Remote Desktop Protocol

  2024cites this paper
• A multivariate time series anomaly detection method with Multi-Grain Dynamic Receptive Field

  2024cites this paper
• Comparative Analysis of Anomaly Detection Approaches in Firewall Logs: Integrating Light-Weight Synthesis of Security Logs and Artificially Generated Attack Detection

  2024cites this paper
• Machine learning based Cyber Attack detection on Internet Traffic

  2024cites this paper
• Anomaly Detection in Smart Environments: A Comprehensive Survey

  2024cites this paper
• conDENSE: Conditional Density Estimation for Time Series Anomaly Detection

  2024cites this paper
• Machine learning in identity and access management systems: Survey and deep dive

  2024cites this paper
• Navigating Cybersecurity Training: A Comprehensive Review

  2024cites this paper
• User Behavior Threat Detection Based on Adaptive Sliding Window GAN

  2024cites this paper
• Towards efficient deep autoencoders for multivariate time series anomaly detection

  2024cites this paper
• Rethinking Graph Backdoor Attacks: A Distribution-Preserving Perspective

  2024cites this paper
• User Behavior Analysis to Detect Insider Threat by Using Machine Learning Algorithms

  2024cites this paper
• Survey on Predictive Algorithms to Detect Insider Threat on a Network Using Different Combination of Machine Learning Algorithms

  2024cites this paper
• MITREtrieval: Retrieving MITRE Techniques From Unstructured Threat Reports by Fusion of Deep Learning and Ontology

  2024cites this paper
• FedITD: A Federated Parameter-Efficient Tuning With Pre-Trained Large Language Models and Transfer Learning Framework for Insider Threat Detection

  2024cites this paper
• BRITD: behavior rhythm insider threat detection with time awareness and user adaptation

  2024influential citation
• Optimising Insider Threat Prediction: Exploring BiLSTM Networks and Sequential Features

  2024cites this paper
• Insider Threat Detection and its Behavior with Excessive Access Privileges

  2024cites this paper
• Threat detection based on multi-scale spatiotemporal feature fusion of user behavior

  2023cites this paper
• Temporal feature aggregation with attention for insider threat detection from activity logs

  2023cites this paper
• Aerospace Integrated Networks Innovation for Empowering 6G: A Survey and Future Challenges

  2023cites this paper
• Deep reinforcement learning for data-efficient weakly supervised business process anomaly detection

  2023cites this paper
• Modeling an intrusion detection using recurrent neural networks

  2023cites this paper
• Insider Threat Detection Based On Heterogeneous Graph Neural Network

  2023cites this paper
• Dealing with extremly Unbalanced Data and Detecting Insider Threats with Deep Neural Networks

  2023cites this paper
• A High Accuracy and Adaptive Anomaly Detection Model With Dual-Domain Graph Convolutional Network for Insider Threat Detection

  2023cites this paper
• A Hash-based Multidimensional Graph Neural Network Approach for Zero Trust Oriented Access Control Security

  2023cites this paper
• Examining the Impact of Machine Learning on Streamlining Data Analysis in Real-Time

  2023cites this paper
• Machine Learning based Identity and Access Management Systems (ML I&AM): a Taxonomy

  2023cites this paper
• Hierarchical Classification Using Ensemble of Feed-Forward Networks for Insider Threat Detection from Activity Logs

  2023cites this paper
• Robust Anomaly-Based Insider Threat Detection Using Graph Neural Network

  2023cites this paper
• A Systematic Review on Machine Learning Insider Threat Detection Models, Datasets and Evaluation Metrics

  2023cites this paper
• APTer: Towards the Investigation of APT Attribution

  2023cites this paper
• Deep Learning-Enabled Heterogeneous Transfer Learning for Improved Network Attack Detection in Internal Networks

  2023cites this paper
• Insider threat detection using supervised machine learning algorithms

  2023cites this paper
• A Data-driven Approach for Risk Exposure Analysis in Enterprise Security

  2023cites this paper
• Optimal Repair Strategy Against Advanced Persistent Threats Under Time-Varying Networks

  2023cites this paper
• UAG: User Action Graph Based on System Logs for Insider Threat Detection

  2023cites this paper
• ThreatLand: Extracting Intelligence from Audit Logs via NLP methods

  2023cites this paper
• Insider Threat Detection: Using Classification Models

  2023cites this paper
• Design And Analysis Of Insider Threat Detection And Prediction System Using Machine Learning Techniques

  2023cites this paper
• Applying One-Class Algorithms for Data Stream-Based Insider Threat Detection

  2023influential citation
• Rare Category Analysis for Complex Data: A Review

  2023cites this paper
• CAS - Attention based ISO/IEC 15408–2 Compliant Continuous Audit System for Insider Threat Detection

  2023cites this paper
• AD-NEv: A Scalable Multilevel Neuroevolution Framework for Multivariate Anomaly Detection

  2023cites this paper
• Comparing Autoencoder and Isolation Forest in Network Anomaly Detection

  2023cites this paper
• Identifying Insider Cyber Threats Using Behaviour Modelling and Analysis

  2023cites this paper
• DTITD: An Intelligent Insider Threat Detection Framework Based on Digital Twin and Self-Attention Based Deep Learning Models

  2023cites this paper
• Continuous User Trust Assessment Based on Emphasized Contextual Differentiation Behavior Analysis

  2023cites this paper
• An Unsupervised Approach For Detecting Lateral Movement Logins Based On Knowledge Graph

  2022cites this paper
• Multi-source data fusion for insider threat detection using residual networks

  2022cites this paper
• An Analysis of Insider Attack Detection Using Machine Learning Algorithms

  2022cites this paper
• Insider Threat Detection Using Machine Learning Approach

  2022cites this paper
• Insider Threat Detection Using Generative Adversarial Graph Attention Networks

  2022cites this paper
• Potential, concepts, and key advances for a ubiquitous adaptive indigenous microengineering and nanoengineering in 6G network

  2022cites this paper
• MADDC: Multi-Scale Anomaly Detection, Diagnosis and Correction for Discrete Event Logs

  2022cites this paper
• Online Location-based Detection of False Data Injection Attacks in Smart Grid Using Deep Learning

  2022cites this paper
• DA-GAN: Domain Adaptation for Generative Adversarial Networks-assisted Cyber Threat Detection

  2022cites this paper
• Enhancing false negative and positive rates for efficient insider threat detection

  2022cites this paper
• RAP-Net: A Resource Access Pattern Network for Insider Threat Detection

  2022cites this paper
• SeqA-ITD: User Behavior Sequence Augmentation for Insider Threat Detection at Multiple Time Granularities

  2022cites this paper