Joint Attention in Autonomous Driving (JAAD)

In this paper we present a novel dataset for a critical aspect of autonomous driving, the joint attention that must occur between drivers and of pedestrians, cyclists or other drivers. This dataset is produced with the intention of demonstrating the behavioral variability of traffic participants. We also show how visual complexity of the behaviors and scene understanding is affected by various factors such as different weather conditions, geographical locations, traffic and demographics of the people involved. The ground truth data conveys information regarding the location of participants (bounding boxes), the physical conditions (e.g. lighting and speed) and the behavior of the parties involved.

• Publication year

  2016

• Venue

  arXiv.org

• Publication date

  2016-09-15

• Fields of study

  Computer Science, Engineering, Psychology

• Identifiers
  arXiv 1609.04741
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Pedestrian

  2018cited by this paper
• Autonomous Driving

  2018cited by this paper
• Driving to Safety

  2016cited by this paper
• A pedestrian's smile and drivers' behavior: When a smile increases careful driving.

  2016cited by this paper
• Analysis of the Influence of Pedestrians’ eye Contact on Drivers’ Comfort Boundary During the Crossing Conflict

  2016cited by this paper
• BORIS: a free, versatile open‐source event‐logging software for video/audio coding and live observations

  2016cited by this paper
• Autonomous Vehicle Implementation Predictions: Implications for Transport Planning

  2015cited by this paper
• Show me your moves! Conveying navigation intention of a mobile robot to humans

  2015cited by this paper
• Are There Deep Reasons Underlying the Pathologies of Today's Deep Learning Algorithms?

  2015cited by this paper
• Google’s Driverless Cars Run into Problem: Cars with Drivers

  2015cited by this paper
• Estimating Driver Unawareness of Pedestrian Based On Visual Behaviors and Driving Behaviors

  2014cited by this paper
• Attentional Mechanisms for Socially Interactive Robots–A Survey

  2014cited by this paper
• Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

  2014cited by this paper
• Estimating driver awareness of pedestrians from driving behavior based on a probabilistic model

  2013cited by this paper
• Intriguing properties of neural networks

  2013cited by this paper
• Pedestrian Detection: An Evaluation of the State of the Art

  2012cited by this paper
• Semantic structure from motion with points, regions, and objects

  2012cited by this paper
• Two-Granularity Tracking: Mediating Trajectory and Detection Graphs for Tracking under Occlusions

  2012cited by this paper
• Development of joint attention and social referencing

  2011cited by this paper
• Pedestrian detection: A benchmark

  2009cited by this paper
• Driver/Pedestrian Understanding and Behavior at Marked and Unmarked Crosswalks

  2007cited by this paper
• Stanley: The robot that won the DARPA Grand Challenge

  2006cited by this paper
• Active Learning of Joint Attention

  2006cited by this paper
• The challenges of joint attention

  2006cited by this paper
• Car-to-Car Communication

  2006cited by this paper
• Learning to Interpret Pointing Gestures: Experiments with Four-Legged Autonomous Robots

  2005cited by this paper
• Probabilistic Gaze Imitation and Saliency Learning in a Robotic Head

  2005cited by this paper
• Modeling of Driving Behavior When Approaching an Intersection Based on Measured Behavioral Data on An Actual Road

  2003cited by this paper
• The 100 Car Naturalistic Driving Study

  2002cited by this paper
• Combining embodied models and empirical research for understanding the development of shared attention

  2002cited by this paper
• Learning and communication via imitation: an autonomous robot perspective

  2001cited by this paper
• Robovie: an interactive humanoid robot

  2001cited by this paper
• Imitation and mechanisms of joint attention: a developmental structure for building social skills on a humanoid robot

  1999cited by this paper
• How to build robots that make friends and influence people

  1999cited by this paper

• Review and Perspectives on Pedestrian Trajectory Prediction for Safe Transportation

  2026cites this paper
• VIT-Ped: Visionary Intention Transformer for Pedestrian Behavior Analysis

  2026cites this paper
• Predicting Vulnerable Road User Behavior With Transformer-Based Gumbel Distribution Networks

  2025influential citation
• V-PTP-IC: End-to-End Joint Modeling of Dynamic Scenes and Social Interactions for Pedestrian Trajectory Prediction from Vehicle-Mounted Cameras

  2025cites this paper
• Pedestrian Group Activity Recognition for Autonomous Vehicles and Robots: A Survey and Perspectives

  2025cites this paper
• Visionary Co-Driver: Enhancing Driver Perception of Potential Risks With LLM and HUD

  2025cites this paper
• Spatiotemporal Interactive Pedestrian Intention Prediction With Illumination Transformation

  2025influential citation
• Is Your VLM for Autonomous Driving Safety-Ready? A Comprehensive Benchmark for Evaluating External and In-Cabin Risks

  2025cites this paper
• Large Foundation Models for Trajectory Prediction in Autonomous Driving: A Comprehensive Survey

  2025cites this paper
• Timewise Intentions and Time-Varying Distribution Network for Pedestrian Trajectory Prediction

  2025cites this paper
• DEV: A Driver-Environment-Vehicle Closed-Loop Framework for Risk-Aware Adaptive Automation of Driving

  2025cites this paper
• MMHU: A Massive-Scale Multimodal Benchmark for Human Behavior Understanding

  2025cites this paper
• Unsupervised pedestrian intention estimation through deep neural embeddings and spatio-temporal graph convolutional networks

  2025cites this paper
• Evaluation of Spatio-Temporal Small Object Detection in Real-World Adverse Weather Conditions

  2025cites this paper
• AHFS-PTP: an adaptive hybrid feature selection method for pedestrian trajectory prediction

  2025cites this paper
• Roadside Sensor Systems for Vulnerable Road User Protection: A Review of Methods and Applications

  2025cites this paper
• Fine-Grained Evaluation of Large Vision-Language Models in Autonomous Driving

  2025cites this paper
• Pedestrian Vision Language Model for Intentions Prediction

  2025cites this paper
• MSLSTM-PID: Multi-Stream LSTM for Pedestrian Intention Detection

  2025cites this paper
• Target-Oriented Environmental Context Modeling for Pedestrian Risky Behavior Detection

  2025cites this paper
• CNN-BiLSTM-Based Vehicle Intention Recognition From the Ego-Vehicle Perspective

  2025cites this paper
• EMT: A Visual Multi-Task Benchmark Dataset for Autonomous Driving in the Arab Gulf Region

  2025cites this paper
• Interact, Plan, and Go: Transformers With Social Intentions for Trajectory Prediction

  2025cites this paper
• A Multi-Stage Goal-Driven Network for Pedestrian Trajectory Prediction

  2024cites this paper
• Padik - Pedestrian-Driver Interaction Dataset in Heterogeneous, Unstructured Traffic Environments

  2024cites this paper
• A Comprehensive Review on Traffic Datasets and Simulators for Autonomous Vehicles

  2024cites this paper
• ROAD-Waymo: Action Awareness at Scale for Autonomous Driving

  2024cites this paper
• Human Action Anticipation: A Survey

  2024cites this paper
• MindReaD: Enhancing Pedestrian-Vehicle Interaction with Micro-Level Reasoning Data Annotation

  2024cites this paper
• Interpretable Prediction of Pedestrian Crossing Intention: Fusion of Human Skeletal Information in Natural Driving Scenarios

  2024cites this paper
• Context-aware Multi-task Learning for Pedestrian Intent and Trajectory Prediction

  2024cites this paper
• AugTrEP: Scene and Occlusion-Aware Pedestrian Crossing Intention Prediction

  2024cites this paper
• A Survey on Datasets for the Decision Making of Autonomous Vehicles

  2024influential citation
• Multi-Modal Sensor Framework with Learnable Uncertainty Estimator for Pedestrian Trajectory Prediction

  2023cites this paper
• Protecting Vulnerable Road Users: Semantic Video Analysis for Accident Prediction

  2023cites this paper
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services

  2023cites this paper
• Spatiotemporal-Linear: Towards Universal Multivariate Time Series Forecasting

  2023cites this paper
• Pedestrian and Passenger Interaction with Autonomous Vehicles: Field Study in a Crosswalk Scenario

  2023cites this paper
• Prediction of Social Dynamic Agents and Long-Tailed Learning Challenges: A Survey

  2023cites this paper
• The IMPTC Dataset: An Infrastructural Multi-Person Trajectory and Context Dataset

  2023influential citation
• Involvement of Deep Learning for Vision Sensor-Based Autonomous Driving Control: A Review

  2023cites this paper
• A Review of the Impact of Rain on Camera-Based Perception in Automated Driving Systems

  2023cites this paper
• From Video to Hybrid Simulator: Exploring Affective Responses toward Non-Verbal Pedestrian Crossing Actions Using Camera and Physiological Sensors

  2023influential citation
• Context-Aware Pedestrian Trajectory Prediction with Multimodal Transformer

  2023cites this paper
• A Survey on Datasets for Decision-making of Autonomous Vehicle

  2023influential citation
• SDR-GAIN: A High Real-Time Occluded Pedestrian Pose Completion Method for Autonomous Driving

  2023cites this paper
• Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypoints

  2023cites this paper
• Addressing uncertainty in the safety assurance of machine-learning

  2023cites this paper
• Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?

  2023cites this paper
• Vulnerable Road Users and Connected Autonomous Vehicles Interaction: A Survey

  2022cites this paper
• The effect of drivers’ eye contact on pedestrians’ perceived safety

  2022cites this paper
• “If You Could See Me through My Eyes”: Predicting Pedestrian Perception

  2022cites this paper
• Pedestrian Stop and Go Forecasting with Hybrid Feature Fusion

  2022cites this paper
• Unequal allocation of overt and covert attention in Multiple Object Tracking

  2022cites this paper
• Vision-based Intention and Trajectory Prediction in Autonomous Vehicles: A Survey

  2022cites this paper
• Investigating Affective Responses toward In-Video Pedestrian Crossing Actions using Camera and Physiological Sensors

  2022cites this paper
• Learning Trajectory-Conditioned Relations to Predict Pedestrian Crossing Behavior

  2022influential citation
• Comparing autoencoder-based approaches for anomaly detection in highway driving scenario images

  2022cites this paper
• Modification and Experimental Validation of a Logistic Regression Vehicle-Pedestrian Model

  2022cites this paper
• Discovering and Understanding Algorithmic Biases in Autonomous Pedestrian Trajectory Predictions

  2022cites this paper
• Online Adaptive Temporal

  2022cites this paper
• Intelligent Pedestrian Intention Prediction Framework

  2022cites this paper
• Multimodal multi-pedestrian path prediction for autonomous cars

  2021cites this paper
• Monocular pedestrian orientation recognition based on Capsule Network for a Novel Collision Warning System

  2021cites this paper
• Behavior Prediction of Traffic Actors for Intelligent Vehicle Using Artificial Intelligence Techniques: A Review

  2021cites this paper
• Is attention to bounding boxes all you need for pedestrian action prediction?

  2021cites this paper
• Lane Intrusion Behaviors Dataset: Action Recognition in Real-world Highway Scenarios for Self-driving

  2021cites this paper
• Multi-Agent Variational Occlusion Inference Using People as Sensors

  2021cites this paper
• Review of Video Predictive Understanding: Early Action Recognition and Future Action Prediction

  2021influential citation
• Coupling Intent and Action for Pedestrian Crossing Behavior Prediction

  2021influential citation
• Generation of Pedestrian Crossing Scenarios Using Ped-Cross Generative Adversarial Network

  2021influential citation
• A context-aware pedestrian trajectory prediction framework for automated vehicles

  2021cites this paper
• GPRAR: Graph Convolutional Network based Pose Reconstruction and Action Recognition for Human Trajectory Prediction

  2021cites this paper
• Stepwise Goal-Driven Networks for Trajectory Prediction

  2021influential citation
• Holistic LSTM for Pedestrian Trajectory Prediction

  2021cites this paper
• PSI: A Pedestrian Behavior Dataset for Socially Intelligent Autonomous Car

  2021cites this paper
• BiTraP: Bi-Directional Pedestrian Trajectory Prediction With Multi-Modal Goal Estimation

  2020cites this paper
• Pedestrian Intention Prediction for Autonomous Driving Using a Multiple Stakeholder Perspective Model

  2020cites this paper
• Considering Reliability of Deep Learning Function to Boost Data Suitability and Anomaly Detection

  2020cites this paper
• Pedestrian Intention Prediction: A Multi-task Perspective

  2020cites this paper
• RNN-based Pedestrian Crossing Prediction using Activity and Pose-related Features

  2020influential citation
• Bosch Deep Learning Hardware Benchmark

  2020cites this paper
• Recognition and 3D Localization of Pedestrian Actions from Monocular Video

  2020cites this paper
• VRUNet: Multi-Task Learning Model for Intent Prediction of Vulnerable Road Users

  2020influential citation
• Classification of a Pedestrian's Behaviour Using Dual Deep Neural Networks

  2020cites this paper
• FuSSI-Net: Fusion of Spatio-temporal Skeletons for Intention Prediction Network

  2020influential citation
• Unsupervised Pedestrian Pose Prediction: A Deep Predictive Coding Network-Based Approach for Autonomous Vehicle Perception

  2020cites this paper
• Impact of Video Compression on the Performance of Object Detection Algorithms in Automotive Applications

  2020influential citation
• Pedestrian Activity Recognition Using 2-D Pose Estimation for Autonomous Vehicles

  2020cites this paper
• Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction

  2020influential citation
• JRMOT: A Real-Time 3D Multi-Object Tracker and a New Large-Scale Dataset

  2020cites this paper
• M2P3: multimodal multi-pedestrian path prediction by self-driving cars with egocentric vision

  2020cites this paper
• Classifying Pedestrian Actions In Advance Using Predicted Video Of Urban Driving Scenes

  2019cites this paper
• Context Model for Pedestrian Intention Prediction Using Factored Latent-Dynamic Conditional Random Fields

  2019influential citation
• Test Your Self-Driving Algorithm: An Overview of Publicly Available Driving Datasets and Virtual Testing Environments

  2019cites this paper
• Multi-Task Cross-Modality Deep Learning for Pedestrian Risk Estimation. (Apprentissage profond multi-tâches inter-modalités pour l'estimation des risques pour les piétons défendu)

  2019cites this paper
• Understanding pedestrian behaviour with pose estimation and recurrent networks

  2019cites this paper
• User Acceptance in Autonomous Vehicles: The Evolution of the End User

  2019cites this paper
• Generation of Pedestrian Pose Structures using Generative Adversarial Networks

  2019cites this paper
• Pedestrian Graph: Pedestrian Crossing Prediction Based on 2D Pose Estimation and Graph Convolutional Networks

  2019cites this paper