MAAD: A Model and Dataset for "Attended Awareness" in Driving

We propose a computational model to estimate a person’s attended awareness of their environment. We define "attended awareness" to be those parts of a potentially dynamic scene which a person has attended to in recent history and which they are still likely to be physically aware of. Our model takes as input scene information in the form of a video and noisy gaze estimates, and outputs visual saliency, a refined gaze estimate and an estimate of the person’s attended awareness. In order to test our model, we capture a new dataset with a high-precision gaze tracker including 24.5 hours of gaze sequences from 23 subjects attending to videos of driving scenes. The dataset also contains third-party annotations of the subjects’ attended awareness based on observations of their scan path. Our results show that our model is able to reasonably estimate attended awareness in a controlled setting, and in the future could potentially be extended to real egocentric driving data to help enable more effective ahead-of-time warnings in safety systems and thereby augment driver performance. We also demonstrate our model’s effectiveness on the tasks of saliency, gaze calibration and denoising, using both our dataset and an existing saliency dataset. We make our model and dataset available at https://github.com/ToyotaResearchInstitute/att-aware/.

• Publication year

  2021

• Venue

  2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)

• Publication date

  2021-10-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1109/ICCVW54120.2021.00382arXiv 2110.08610
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

  2020cited by this paper
• How Do Drivers Allocate Their Potential Attention? Driving Fixation Prediction via Convolutional Neural Networks

  2020cited by this paper
• When I Look into Your Eyes: A Survey on Computer Vision Contributions for Human Gaze Estimation and Tracking

  2020cited by this paper
• Classifying Eye-Tracking Data Using Saliency Maps

  2020cited by this paper
• STAViS: Spatio-Temporal AudioVisual Saliency Network

  2020cited by this paper
• A Deep Network Solution for Attention and Aesthetics Aware Photo Cropping

  2019cited by this paper
• Gaze-based Intention Anticipation over Driving Manoeuvres in Semi-Autonomous Vehicles

  2019cited by this paper
• Gaze360: Physically Unconstrained Gaze Estimation in the Wild

  2019cited by this paper
• TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection

  2019cited by this paper
• DADA: A Large-scale Benchmark and Model for Driver Attention Prediction in Accidental Scenarios

  2019cited by this paper
• MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learning

  2019cited by this paper
• TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 1 ModDrop

  2018cited by this paper
• Panoptic Segmentation

  2018cited by this paper
• Performance Evaluation Strategies for Eye Gaze Estimation Systems with Quantitative Metrics and Visualizations

  2018influential reference
• Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification

  2017cited by this paper
• More than the Useful Field: Considering peripheral vision in driving.

  2017cited by this paper
• Predicting the Driver's Focus of Attention: The DR(eye)VE Project

  2017influential reference
• Predicting Driver Attention in Critical Situations

  2017cited by this paper
• SalGAN: Visual Saliency Prediction with Adversarial Networks

  2017cited by this paper
• Shallow and Deep Convolutional Networks for Saliency Prediction

  2016cited by this paper
• A Deeper Look at Saliency: Feature Contrast, Semantics, and Beyond

  2016cited by this paper
• A deep multi-level network for saliency prediction

  2016cited by this paper
• Predicting Human Eye Fixations via an LSTM-Based Saliency Attentive Model

  2016cited by this paper
• Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning

  2016cited by this paper
• Recurrent Mixture Density Network for Spatiotemporal Visual Attention

  2016cited by this paper
• The Eyes Are the Windows to the Mind

  2016cited by this paper
• Consistent Video Saliency Using Local Gradient Flow Optimization and Global Refinement

  2015cited by this paper
• Salient Object Detection: A Benchmark

  2015cited by this paper
• Delving into egocentric actions

  2015cited by this paper
• U-Net: Convolutional Networks for Biomedical Image Segmentation

  2015cited by this paper
• SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks

  2015cited by this paper
• Real-world spatial regularities affect visual working memory for objects

  2015cited by this paper
• Adam: A Method for Stochastic Optimization

  2014cited by this paper
• Updating visual-spatial working memory during object movement.

  2014cited by this paper
• The Driver Distraction Detection Algorithm AttenD

  2013cited by this paper
• Dynamic simulation and prediction of drivers' attention distribution

  2013cited by this paper
• Actions in the Eye: Dynamic Gaze Datasets and Learnt Saliency Models for Visual Recognition

  2013cited by this paper
• State-of-the-Art in Visual Attention Modeling

  2013cited by this paper
• A probabilistic model of visual working memory: Incorporating higher order regularities into working memory capacity estimates.

  2013cited by this paper
• Probabilistic learning of task-specific visual attention

  2012cited by this paper
• A computational model of the car driver interfaced with a simulation platform for future Virtual Human Centred Design applications: COSMO-SIVIC

  2012cited by this paper
• Depth Matters: Influence of Depth Cues on Visual Saliency

  2012cited by this paper
• Some hazards are more attractive than others: drivers of varying experience respond differently to different types of hazard.

  2012cited by this paper
• Eye guidance in natural vision: reinterpreting salience.

  2011cited by this paper
• Visual working memory as visual attention sustained internally over time

  2011cited by this paper
• Measuring the Objectness of Image Windows

  2011cited by this paper
• Calibration-free gaze sensing using saliency maps

  2010cited by this paper
• Loss of visual working memory within seconds: the combined use of refreshable and non-refreshable features.

  2010cited by this paper
• Yarbus, eye movements, and vision

  2010cited by this paper
• Torchvision the machine-vision package of torch

  2010cited by this paper
• On the Roles of Eye Gaze and Head Dynamics in Predicting Driver's Intent to Change Lanes

  2009cited by this paper
• Learning to predict where humans look

  2009cited by this paper
• The role of visual working memory in attentive tracking of unique objects.

  2009cited by this paper
• A Fast Approximation of the Bilateral Filter Using a Signal Processing Approach

  2006cited by this paper
• Active Vision: The Psychology of Looking and Seeing

  2006cited by this paper
• Effects of visual and cognitive load in real and simulated motorway driving

  2005cited by this paper
• A Computational Model of Attention/Situation Awareness

  2002cited by this paper
• Binding in short-term visual memory.

  2002cited by this paper
• Mean Shift: A Robust Approach Toward Feature Space Analysis

  2002cited by this paper
• Storage of features, conjunctions and objects in visual working memory.

  2001cited by this paper
• Characterising the Visual Buffer: Real-World Evidence for Overwriting Early in Each Fixation

  2001cited by this paper
• An integrated model of eye movements and visual encoding

  2001cited by this paper
• A Model of Saliency-Based Visual Attention for Rapid Scene Analysis

  1998cited by this paper
• Toward a Theory of Situation Awareness in Dynamic Systems

  1995cited by this paper
• THE ATTENTIONAL DEMAND OF AUTOMOBILE DRIVING

  1967cited by this paper
• Eye Movements and Vision

  1967cited by this paper

• Estimating cognitive biases with attention-aware inverse planning

  2025cites this paper
• SAFe-Copilot: Unified Shared Autonomy Framework

  2025cites this paper
• Top-Down Attention-Based Mechanisms for Interpretable Autonomous Driving

  2025cites this paper
• AEGIS: Human Attention-based Explainable Guidance for Intelligent Vehicle Systems

  2025cites this paper
• Causal-Entity Reflected Egocentric Traffic Accident Video Synthesis

  2025cites this paper
• SimCoachCorpus: A naturalistic dataset with language and trajectories for embodied teaching

  2025cites this paper
• Data Limitations for Modeling Top-Down Effects on Drivers’ Attention

  2024cites this paper
• Enhancing End-to-End Autonomous Driving Systems Through Synchronized Human Behavior Data

  2024cites this paper
• Context-Aware Driver Attention Estimation Using Multi-Hierarchy Saliency Fusion With Gaze Tracking

  2024cites this paper
• Attention-Based Interrelation Modeling for Explainable Automated Driving

  2023cites this paper
• Understanding and Modeling the Effects of Task and Context on Drivers’ Gaze Allocation

  2023cites this paper
• Leveraging Driver Field-of-View for Multimodal Ego-Trajectory Prediction

  2023cites this paper
• Attention for Vision-Based Assistive and Automated Driving: A Review of Algorithms and Datasets

  2022cites this paper
• DriveRNN: Predicting Drivers' Attention with Deep Recurrent Networks

  2022cites this paper
• Causal-Entity Reflected Egocentric Traffic Accident Video Synthesis

  year unknowncites this paper