Pedestrian intention prediction: A convolutional bottom-up multi-task approach

Abstract The ability to predict pedestrian behaviour is crucial for road safety, traffic management systems, Advanced Driver Assistance Systems (ADAS), and more broadly autonomous vehicles. We present a vision-based system that simultaneously locates where pedestrians are in the scene, estimates their body pose and predicts their intention to cross the road. Given a single image, our proposed neural network is designed using a bottom-up approach and thus runs at nearly constant time without relying on a pedestrian detector. Our method jointly detects human body poses and predicts their intention in a multitask framework. Experimental results show that the proposed model outperforms the precision scores of the state-of-the-art for the task of intention prediction by approximately 20% while running in real-time (5 fps). The source code is publicly available so that it can be easily integrated into an ADAS or into any traffic light management systems.

• Publication year

  2021

• Venue

  Transportation Research Part C-emerging Technologies

• Publication date

  2021-09-01

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.1016/J.TRC.2021.103259
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Estimating Pedestrian Crossing States Based on Single 2D Body Pose

  2020cited by this paper
• Perceiving Humans: From Monocular 3D Localization to Social Distancing

  2020cited by this paper
• Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction

  2020cited by this paper
• An efficient human action recognition framework with pose-based spatiotemporal features

  2020cited by this paper
• Convolutional Relational Machine for Group Activity Recognition

  2019cited by this paper
• Classifying Pedestrian Actions In Advance Using Predicted Video Of Urban Driving Scenes

  2019cited by this paper
• Inferring Pedestrian Motions at Urban Crosswalks

  2019cited by this paper
• PifPaf: Composite Fields for Human Pose Estimation

  2019cited by this paper
• Real-time Intent Prediction of Pedestrians for Autonomous Ground Vehicles via Spatio-Temporal DenseNet

  2019cited by this paper
• Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories With GANs

  2019cited by this paper
• Probabilistic Prediction of Pedestrian Crossing Intention Using Roadside LiDAR Data

  2019cited by this paper
• Pedestrian Path, Pose, and Intention Prediction Through Gaussian Process Dynamical Models and Pedestrian Activity Recognition

  2019cited by this paper
• MonoLoco: Monocular 3D Pedestrian Localization and Uncertainty Estimation

  2019cited by this paper
• Looking Ahead: Anticipating Pedestrians Crossing with Future Frames Prediction

  2019cited by this paper
• Dynamic Task Prioritization for Multitask Learning

  2018cited by this paper
• Learning to Forecast Pedestrian Intention from Pose Dynamics

  2018cited by this paper
• Set-Based Prediction of Pedestrians in Urban Environments Considering Formalized Traffic Rules

  2018cited by this paper
• Learning to Predict Pedestrian Intention via Variational Tracking Networks

  2018cited by this paper
• Pedestrian trajectory prediction via the Social‐Grid LSTM model

  2018cited by this paper
• 2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning

  2018cited by this paper
• A Computationally Efficient Model for Pedestrian Motion Prediction

  2018cited by this paper
• PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model

  2018cited by this paper
• Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks

  2018cited by this paper
• OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

  2018cited by this paper
• SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints

  2018cited by this paper
• Is the Pedestrian going to Cross? Answering by 2D Pose Estimation

  2018cited by this paper
• Detecting Intentions of Vulnerable Road Users Based on Collective Intelligence

  2018cited by this paper
• Long-Term Recurrent Predictive Model for Intent Prediction of Pedestrians via Inverse Reinforcement Learning

  2018cited by this paper
• Jointly Learning Energy Expenditures and Activities Using Egocentric Multimodal Signals

  2017cited by this paper
• Pedestrian Prediction by Planning Using Deep Neural Networks

  2017cited by this paper
• Intent prediction of vulnerable road users from motion trajectories using stacked LSTM network

  2017cited by this paper
• Trajectory prediction of traffic agents at urban intersections through learned interactions

  2017cited by this paper
• DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents

  2017cited by this paper
• Automated Curriculum Learning for Neural Networks

  2017cited by this paper
• Context-Aware Trajectory Prediction

  2017cited by this paper
• Are They Going to Cross? A Benchmark Dataset and Baseline for Pedestrian Crosswalk Behavior

  2017cited by this paper
• Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection

  2017cited by this paper
• Mask R-CNN

  2017cited by this paper
• Early Detection of Sudden Pedestrian Crossing for Safe Driving During Summer Nights

  2017cited by this paper
• Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

  2017cited by this paper
• Pedestrian intention recognition by means of a Hidden Markov Model and body language

  2017cited by this paper
• Context based pedestrian intention prediction using factored latent dynamic conditional random fields

  2017cited by this paper
• Learning to Predict Human Behavior in Crowded Scenes

  2017cited by this paper
• On-Board Detection of Pedestrian Intentions

  2017cited by this paper
• Pedestrian Movement Direction Recognition Using Convolutional Neural Networks

  2017cited by this paper
• RMPE: Regional Multi-person Pose Estimation

  2016cited by this paper
• NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

  2016cited by this paper
• Social LSTM: Human Trajectory Prediction in Crowded Spaces

  2016cited by this paper
• UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory

  2016cited by this paper
• Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network

  2016cited by this paper
• Stereo-Vision-Based Pedestrian's Intention Detection in a Moving Vehicle

  2015cited by this paper
• DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation

  2015cited by this paper
• A Controlled Interactive Multiple Model Filter for Combined Pedestrian Intention Recognition and Path Prediction

  2015cited by this paper
• Trajectory analysis and prediction for improved pedestrian safety: Integrated framework and evaluations

  2015cited by this paper
• Human Action Recognition Using Factorized Spatio-Temporal Convolutional Networks

  2015cited by this paper
• Pedestrian Intention and Pose Prediction through Dynamical Models and Behaviour Classification

  2015cited by this paper
• A Probabilistic Model for the Estimation of Pedestrian Crossing Behavior at Signalized Intersections

  2015cited by this paper
• Probability estimation for pedestrian crossing intention at signalized crosswalks

  2015cited by this paper
• Robust real-time pedestrians detection in urban environments with low-resolution cameras

  2014cited by this paper
• Pedestrian crossing prediction using multiple context-based models

  2014cited by this paper
• Pedestrian path prediction based on body language and action classification

  2014cited by this paper
• Striving for Simplicity: The All Convolutional Net

  2014cited by this paper
• Pedestrian detection aided by deep learning semantic tasks

  2014cited by this paper
• Large-Scale Video Classification with Convolutional Neural Networks

  2014cited by this paper
• Pedestrian path prediction using body language traits

  2014cited by this paper
• Context-Based Pedestrian Path Prediction

  2014cited by this paper
• Kinects and human kinetics: a new approach for studying pedestrian behavior

  2014cited by this paper
• Will the Pedestrian Cross? A Study on Pedestrian Path Prediction

  2014cited by this paper
• An Approach to Pose-Based Action Recognition

  2013cited by this paper
• Pedestrian Detection: An Evaluation of the State of the Art

  2012cited by this paper
• Curriculum learning

  2009cited by this paper
• Pedestrian crossing detection based on evidential fusion of video-sensors

  2009cited by this paper
• Histograms of oriented gradients for human detection

  2005cited by this paper
• Multitask Learning

  1997cited by this paper
• Long Short-Term Memory

  1997cited by this paper

• Multimodal understanding with GPT-4o to enhance generalizable pedestrian behavior prediction

  2026cites this paper
• Intentions prediction for human–robot collaboration in utility tunnel maintenance

  2025cites this paper
• AttentionDriveNet: Fusion of deep cognitive network with Attention modeling for robust navigation in Self-driving vehicles

  2025cites this paper
• Pedestrian Crossing Intention Prediction with Multi-Kernel Graph Convolution and Attention Pooling

  2025cites this paper
• A Systematic Literature Review of Pedestrian Safety in Urban Transport Systems

  2025cites this paper
• A Survey on Deep Multi-Task Learning in Connected Autonomous Vehicles

  2025cites this paper
• Social-Pose: Enhancing Trajectory Prediction With Human Body Pose

  2025cites this paper
• Driver temporal segmentation of pedestrian crossing intentions during negotiations

  2025cites this paper
• Pedestrian Intention Prediction via Vision-Language Foundation Models

  2025cites this paper
• Navigating Uncertainty: Advanced Techniques in Pedestrian Intention Prediction for Autonomous Vehicles—A Comprehensive Review

  2025cites this paper
• A review on the estimation of vulnerable road user behavior for automated vehicles

  2025cites this paper
• Pedestrian Crossing Direction Prediction at Intersections for Pedestrian Safety

  2025cites this paper
• Minimizing Human Labeling in Training Deep Models for Pedestrian Intention Prediction

  2025cites this paper
• VRUCrossSafe for crossing intention prediction of vulnerable road users for improving safe crossing at intersections

  2025cites this paper
• Handling Pedestrian Uncertainty in Coordinating Autonomous Vehicles at Signal-Free Intersections

  2025cites this paper
• WatchoutPed: A dataset and model for Vulnerable Pedestrian Anticipation in surveillance videos

  2025cites this paper
• Interaction-Aware Model Predictive Decision-Making for Socially-Compliant Autonomous Driving in Mixed Urban Traffic Scenarios

  2025cites this paper
• Pedestrian Facial Attention Detection Using Deep Fusion and Multi-Modal Fusion Classifier

  2025cites this paper
• Predicting Pedestrian Crossing Intention in Autonomous Vehicles: A Review

  2025cites this paper
• Modeling interpretable social interactions for pedestrian trajectory

  2024cites this paper
• Pedestrian Crossing Intention Prediction From Surveillance Videos for Over-the-Horizon Safety Warning

  2024cites this paper
• Faster Pedestrian Crossing Intention Prediction Based on Efficient Fusion of Diverse Intention Influencing Factors

  2024cites this paper
• Vehicle speed prediction using a convolutional neural network combined with a gated recurrent unit with attention

  2024cites this paper
• PIP-Net: Pedestrian Intention Prediction in the Wild

  2024cites this paper
• Evade or rush? Investigating jaywalkers' sequential crossing decisions at mid-blocks without crossing facilities

  2024cites this paper
• Multi-Pedestrian Tracking and Map-Based Intention Estimation for Autonomous Driving Scenario

  2024cites this paper
• PedAST-GCN: Fast Pedestrian Crossing Intention Prediction Using Spatial–Temporal Attention Graph Convolution Networks

  2024cites this paper
• Vulnerable Road User Detection and Safety Enhancement: A Comprehensive Survey

  2024cites this paper
• Diving Deeper Into Pedestrian Behavior Understanding: Intention Estimation, Action Prediction, and Event Risk Assessment

  2024cites this paper
• Automatic Vehicle Trajectory Behavior Classification Based on Unmanned Aerial Vehicle-Derived Trajectories Using Machine Learning Techniques

  2024cites this paper
• Multi-modal transformer with language modality distillation for early pedestrian action anticipation

  2024cites this paper
• Enhancing Availability of Autonomous Shuttle Services: A Conceptual Approach toward Challenges and Opportunities

  2024cites this paper
• Real-Time Pedestrian Crossing Anticipation Based on an Action–Interaction Dual-Branch Network

  2024cites this paper
• Real-time Object Detection in Autonomous Vehicles with YOLO

  2024cites this paper
• Pedestrian Intention Detection for Autonomous Driving: A Novel Object Detection Approach

  2024cites this paper
• Pedestrian Crossing Intention Prediction in the Wild: A Survey

  2024cites this paper
• A Sequential Pattern Mining Approach for Situation-Aware Human Activity Projection

  2024cites this paper
• Video action recognition collaborative learning with dynamics via PSO-ConvNet Transformer

  2023influential citation
• Recognizing Real-World Intentions using A Multimodal Deep Learning Approach with Spatial-Temporal Graph Convolutional Networks

  2023cites this paper
• Synthetic Data Generation Framework, Dataset, and Efficient Deep Model for Pedestrian Intention Prediction

  2023cites this paper
• GPT-4V Takes the Wheel: Promises and Challenges for Pedestrian Behavior Prediction

  2023cites this paper
• Predicting Human Activity – State of the Art

  2023cites this paper
• Enhancing Object Detection in Self-Driving Cars Using a Hybrid Approach

  2023cites this paper
• Local and Global Contextual Features Fusion for Pedestrian Intention Prediction

  2023cites this paper
• Multi-scale pedestrian intent prediction using 3D joint information as spatio-temporal representation

  2023cites this paper
• Deep hybrid model with satellite imagery: how to combine demand modeling and computer vision for behavior analysis?

  2023cites this paper
• Intelligent multimodal pedestrian detection using hybrid metaheuristic optimization with deep learning model

  2023cites this paper
• Pedestrian Crossing Intention Prediction Model Considering Social Interaction between Multi-Pedestrians and Multi-Vehicles

  2023cites this paper
• CIPF: Crossing Intention Prediction Network based on Feature Fusion Modules for Improving Pedestrian Safety

  2023influential citation
• DeepStep: Direct Detection of Walking Pedestrian From Motion by a Vehicle Camera

  2023cites this paper
• Visual Extensions and Anomaly Detection in the Pneuma Experiment with a Swarm of Drones

  2023cites this paper
• PIT: Progressive Interaction Transformer for Pedestrian Crossing Intention Prediction

  2023cites this paper
• Traffic Light and Uncertainty Aware Pedestrian Crossing Intention Prediction for Automated Vehicles

  2023cites this paper
• Dual Quaternion Rotational and Translational Equivariance in 3D Rigid Motion Modelling

  2023cites this paper
• Pedestrian and vehicle behaviour prediction in autonomous vehicle system - A review

  2023cites this paper
• Prediction of jaywalker-vehicle conflicts based on encoder-decoder framework utilizing multi-source data.

  2023cites this paper
• GPT-4V Takes the Wheel: Evaluating Promise and Challenges for Pedestrian Behavior Prediction

  2023cites this paper
• Feature Selection and Multi-task Learning for Pedestrian Crossing Prediction

  2022cites this paper
• Novel trajectory prediction algorithm using a full dataset: comparison and ablation studies

  2022cites this paper
• Behavioral Intention Prediction in Driving Scenes: A Survey

  2022cites this paper
• Vulnerable Road Users and Connected Autonomous Vehicles Interaction: A Survey

  2022cites this paper
• Generic Sensor Model Usecase Exemplified by Pedestrian Crossing

  2022cites this paper
• WatchPed: Pedestrian Crossing Intention Prediction Using Embedded Sensors of Smartwatch

  2022cites this paper
• Pedestrian Intention Prediction for Autonomous Vehicles: A Comprehensive Survey

  2022cites this paper
• Pedestrian Stop and Go Forecasting with Hybrid Feature Fusion

  2022cites this paper
• PIT: Progressive Interaction Transformer for Pedestrian Intention Prediction

  year unknowncites this paper