Human-assisted motion annotation

Obtaining ground-truth motion for arbitrary, real-world video sequences is a challenging but important task for both algorithm evaluation and model design. Existing ground-truth databases are either synthetic, such as the Yosemite sequence, or limited to indoor, experimental setups, such as the database developed by Baker et al (2007). We propose a human-in-loop methodology to create a ground-truth motion database for the videos taken with ordinary cameras in both indoor and outdoor scenes, using the fact that human beings are experts at segmenting objects and inspecting the match between two frames. We designed an interactive computer vision system to allow a user to efficiently annotate motion. Our methodology is cross-validated by showing that human annotated motion is repeatable, consistent across annotators, and close to the ground truth obtained by Baker et al (2007). Using our system, we collected and annotated 10 indoor and outdoor real-world videos to form a ground-truth motion database. The source code, annotation tool and database is online for public evaluation and benchmarking.

• Publication year

  2008

• Venue

  2008 IEEE Conference on Computer Vision and Pattern Recognition

• Publication date

  2008-06-01

• Fields of study

  Computer Science, Engineering

• Identifiers
  DOI 10.1109/CVPR.2008.4587845
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• LabelMe: A Database and Web-Based Tool for Image Annotation

  2008cited by this paper
• A Database and Evaluation Methodology for Optical Flow

  2007influential reference
• Particle Video: Long-Range Motion Estimation Using Point Trajectories

  2006cited by this paper
• Analysis of Contour Motions

  2006cited by this paper
• Motion magnification

  2005cited by this paper
• On the Spatial Statistics of Optical Flow

  2005influential reference
• Video object cut and paste

  2005cited by this paper
• A Study of the Yosemite Sequence Used as a Test Sequence for Estimation of Optical Flow

  2005cited by this paper
• Interactive video cutout

  2005cited by this paper
• High Accuracy Optical Flow Estimation Based on a Theory for Warping

  2004influential reference
• Keyframe-based tracking for rotoscoping and animation

  2004cited by this paper
• What went where

  2003cited by this paper
• An accurate scene-based traffic model for MPEG video stream

  2003cited by this paper
• Video matting of complex scenes

  2002cited by this paper
• Symmetrical Dense Optical Flow Estimation with Occlusions Detection

  2002cited by this paper
• A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics

  2001cited by this paper
• An integrated Bayesian approach to layer extraction from image sequences

  1999cited by this paper
• CONDENSATION—Conditional Density Propagation for Visual Tracking

  1998cited by this paper
• The Robust Estimation of Multiple Motions: Parametric and Piecewise-Smooth Flow Fields

  1996cited by this paper
• Perceptually Organized Em: a Framework for Motion Segmentation That Combines Information about Form and Motion

  1995cited by this paper
• Representing moving images with layers

  1994cited by this paper
• Good features to track

  1994cited by this paper
• Layered representations for image coding

  1991cited by this paper
• Fast surface interpolation using hierarchical basis functions

  1989cited by this paper
• An Iterative Image Registration Technique with an Application to Stereo Vision

  1981influential reference
• Determining Optical Flow

  1981cited by this paper
• Image processing

  1971cited by this paper
• Universität Des Saarlandes Fachrichtung 6.1 – Mathematik Lucas/kanade Meets Horn/schunck: Combining Local and Global Optic Flow Methods Lucas/kanade Meets Horn/schunck: Combining Local and Global Optic Flow Methods

  year unknowninfluential reference

• R-OFFlow: An Optical Flow Motion Estimation Method Guided by Offset Fields for Self-Compensation of Ladder Jitter in Offshore Wind Turbine Boarding

  2025cites this paper
• Observation-based quantification of aerosol transport using optical flow: A satellite perspective to characterize interregional transport of atmospheric pollution

  2024cites this paper
• Adverse Weather Optical Flow: Cumulative Homogeneous-Heterogeneous Adaptation

  2024cites this paper
• Development of Web3D education platform suitable for schoolchild

  2023cites this paper
• Optical Flow Estimation in Dense Foggy Scenes With Domain-Adaptive Networks

  2023cites this paper
• Fusion of Appearance and Motion Features for Daily Activity Recognition from Egocentric Perspective

  2023cites this paper
• Optical Flow Estimation in 360$^\circ$ Videos: Dataset, Model and Application

  2023cites this paper
• GyroFlow+: Gyroscope-Guided Unsupervised Deep Homography and Optical Flow Learning

  2023influential citation
• Motion-region annotation for complex videos via label propagation across occluders

  2022influential citation
• Prediction of motion sickness degree of stereoscopic panoramic videos based on content perception and binocular characteristics

  2022cites this paper
• Benchmarking equivariance for Deep Learning based optical flow estimators

  2022cites this paper
• Unsupervised Learning of Auroral Optical Flow for Recognition of Poleward Moving Auroral Forms

  2022cites this paper
• Learning Omnidirectional Flow in 360-degree Video via Siamese Representation

  2022cites this paper
• RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos

  2022cites this paper
• Real-Time Efficient FPGA Implementation of the Multi-Scale Lucas-Kanade and Horn-Schunck Optical Flow Algorithms for a 4K Video Stream

  2022influential citation
• Robust Dynamic Background Modeling for Foreground Estimation

  2022cites this paper
• OIFlow: Occlusion-Inpainting Optical Flow Estimation by Unsupervised Learning

  2021cites this paper
• AI-Empowered Persuasive Video Generation: A Survey

  2021cites this paper
• Bayesian Cellular Automata Fusion Model Based on Dual-Stream Strategy for Video Anomaly Action Detection

  2021cites this paper
• GyroFlow: Gyroscope-Guided Unsupervised Optical Flow Learning

  2021cites this paper
• Crowd flow estimation from calibrated cameras

  2020cites this paper
• Analysing a Vein Liveness Detection Scheme

  2020cites this paper
• A New 3D Video Database for Stereoscopic Video Comfortable Prediction

  2020cites this paper
• FleXeen: Visually Manipulating Perceived Fabric Bending Stiffness in Spatial Augmented Reality

  2020cites this paper
• Gabor Feature-Based LogDemons With Inertial Constraint for Nonrigid Image Registration

  2020cites this paper
• Distributed algorithms for scalable proximity operator computation and application to video denoising

  2020cites this paper
• Optical Flow in Dense Foggy Scenes Using Semi-Supervised Learning

  2020cites this paper
• Spatial-Temporal Gaussian Scale Mixture Modeling for Foreground Estimation

  2020cites this paper
• Wireless Mobile Communication and Healthcare: 8th EAI International Conference, MobiHealth 2019, Dublin, Ireland, November 14-15, 2019, Proceedings

  2020cites this paper
• Turning video into traffic data - an application to urban intersection analysis using transfer learning

  2019cites this paper
• A new technique based on 3D convolutional neural networks and filtering optical flow maps for action classification in infrared video

  2019cites this paper
• Multi-view region-adaptive multi-temporal DMM and RGB action recognition

  2019cites this paper
• CNN with spatio-temporal information for fast suspicious object detection and recognition in THz security images

  2019cites this paper
• DDFlow: Learning Optical Flow with Unlabeled Data Distillation

  2019cites this paper
• An alternating proximal approach for blind video deconvolution

  2019cites this paper
• A Tiny Diagnostic Dataset and Diverse Modules for Learning-Based Optical Flow Estimation

  2019cites this paper
• Kernel null-space-based abnormal event detection using hybrid motion information

  2019cites this paper
• Biologically-Inspired Computational Neural Mechanism for Human Action/activity Recognition: A Review

  2019cites this paper
• Labeling of Activity Recognition Datasets: Detection of Misbehaving Users

  2019cites this paper
• Multispectral video acquisition using spectral sweep camera.

  2019cites this paper
• An Unsupervised Method to Extract Video Object via Complexity Awareness and Object Local Parts

  2018cites this paper
• Learning Video-Story Composition via Recurrent Neural Network

  2018cites this paper
• Exploring Semi-Supervised Methods for Labeling Support in Multimodal Datasets

  2018cites this paper
• Optical Flow Dataset and Benchmark for Visual Crowd Analysis

  2018cites this paper
• Smart Annotation Tool for Multi-sensor Gait-based Daily Activity Data

  2018cites this paper
• Extending Layered Models to 3D Motion

  2018cites this paper
• Temporal Interpolation as an Unsupervised Pretraining Task for Optical Flow Estimation

  2018cites this paper
• Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation

  2018cites this paper
• DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency

  2018cites this paper
• Robust Optical Flow in Rainy Scenes

  2018cites this paper
• Patch-based frame interpolation for old films via the guidance of motion paths

  2018influential citation
• Video Shadow Removal Using Spatio‐temporal Illumination Transfer

  2017cites this paper
• A collection of challenging motion segmentation benchmark datasets

  2017cites this paper
• Video Object Segmentation via Global Consistency Aware Query Strategy

  2017influential citation
• Appearance and motion information based human activity recognition

  2017cites this paper
• Variational method for joint optical flow estimation and edge-aware image restoration

  2017influential citation
• Synthesizing Obama

  2017influential citation
• PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume

  2017cites this paper
• Dual Block-Coordinate Forward-Backward Algorithm with Application to Deconvolution and Deinterlacing of Video Sequences

  2017cites this paper
• Restoration super-resolution of image sequences: application to TV archive documents. (Restauration super-résolution de séquences d'images: applications aux documents d'archives TV)

  2017cites this paper
• A Novel Artificial Organic Controller with Hermite Optical Flow Feedback for Mobile Robot Navigation

  2016cites this paper
• The HCI Benchmark Suite: Stereo and Flow Ground Truth with Uncertainties for Urban Autonomous Driving

  2016cites this paper
• Saliency-based abnormal event detection in crowded scenes

  2016cites this paper
• Combining motion and appearance cues for anomaly detection

  2016cites this paper
• Video-Story Composition via Plot Analysis

  2016cites this paper
• Parallel computing of texture-optimisation-based flow visualisation on CUDA

  2016cites this paper
• Geometric bounding box interpolation: an alternative for efficient video annotation

  2016cites this paper
• New Applications of Artificial Intelligence

  2016cites this paper
• Crowd Scene Analysis in Video Surveillance

  2016cites this paper
• Local part model for action recognition

  2016cites this paper
• An evaluation of optical flow algorithms for crowd analytics in surveillance system

  2016cites this paper
• A Novel Framework for Anomaly Detection in Video Surveillance Using Multi-feature Extraction

  2016cites this paper
• RGB-D Scene Flow via Grouping Rigid Motions

  2016cites this paper
• Mobile Robot with Movement Detection Controlled by a Real-Time Optical Flow Hermite Transform

  2016cites this paper
• Scale-Adaptive Video Understanding

  2016influential citation
• Video anomaly detection based on locality sensitive hashing filters

  2016cites this paper
• Video Background Completion Using Motion-Guided Pixel Assignment Optimization

  2016influential citation
• On Motion Estimation Problems in Computer Vision

  2016influential citation
• Road Detection through Supervised Classification

  2016cites this paper
• Introduction to Dense Optical Flow

  2016cites this paper
• Sparse variational regularization for visual motion estimation

  2016cites this paper
• Fast two-cycle curve evolution with narrow perception of background for object tracking and contour refinement

  2016cites this paper
• Contour Tracking with a Spatio-Temporal Intensity Moment

  2016cites this paper
• Shape-Constrained Sparse and Low-Rank Decomposition for Auroral Substorm Detection

  2016influential citation
• Finger vein liveness detection using motion magnification

  2015cites this paper
• Fast Two-Cycle level set tracking with narrow perception of background

  2015cites this paper
• Synthetic sequences and ground-truth flow field generation for algorithm validation

  2015cites this paper
• LIBSVX: A Supervoxel Library and Benchmark for Early Video Processing

  2015influential citation
• Perception, Information Acquisition, and Prediction in Visual Tasks

  2015cites this paper
• A semiautomatic method for pedestrian ground truth generation in thermal infrared on-board videos

  2015influential citation
• Local analysis of confidence measures for Optical Flow quality evaluation

  2015cites this paper
• Learning similarity metrics for dynamic scene segmentation

  2015cites this paper
• Spatio-temporal Video Parsing for Abnormality Detection

  2015cites this paper
• Advances in video motion analysis research for mature and emerging application areas

  2015cites this paper
• Analysis and modeling pf passive stereo and time-of-flight imaging

  2015cites this paper
• Compass Rose: A Rotational Robust Signature for Optical Flow Computation

  2014cites this paper
• Local Part Model for Action Recognition in Realistic Videos

  2014cites this paper
• A hybrid alternating proximal method for blind video restoration

  2014cites this paper
• Light field reconstruction robust to signal dependent noise

  2014cites this paper
• Nonparametric Clustering with Distance Dependent Hierarchies

  2014cites this paper