High-quality passive facial performance capture using anchor frames

We present a new technique for passive and markerless facial performance capture based on anchor frames. Our method starts with high resolution per-frame geometry acquisition using state-of-the-art stereo reconstruction, and proceeds to establish a single triangle mesh that is propagated through the entire performance. Leveraging the fact that facial performances often contain repetitive subsequences, we identify anchor frames as those which contain similar facial expressions to a manually chosen reference expression. Anchor frames are automatically computed over one or even multiple performances. We introduce a robust image-space tracking method that computes pixel matches directly from the reference frame to all anchor frames, and thereby to the remaining frames in the sequence via sequential matching. This allows us to propagate one reconstructed frame to an entire sequence in parallel, in contrast to previous sequential methods. Our anchored reconstruction approach also limits tracker drift and robustly handles occlusions and motion blur. The parallel tracking and mesh propagation offer low computation times. Our technique will even automatically match anchor frames across different sequences captured on different occasions, propagating a single mesh to all performances.

• Publication year

  2011

• Venue

  ACM Transactions on Graphics

• Publication date

  2011-07-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1145/1964921.1964970
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• Comprehensive Facial Performance Capture

  2011cited by this paper
• High resolution passive facial performance capture

  2010influential reference
• High-quality single-shot capture of facial geometry

  2010influential reference
• Globally Consistent Space‐Time Reconstruction

  2010cited by this paper
• Temporal upsampling of performance geometry using photometric alignment

  2010cited by this paper
• Self-calibrating a real-time monocular 3 d facial capture system

  2010cited by this paper
• Efficient reconstruction of nonrigid shape and motion from real-time 3D scanner data

  2009cited by this paper
• Dense 3D motion capture for human faces

  2009cited by this paper
• The Digital Emily project: photoreal facial modeling and animation

  2009cited by this paper
• Markerless garment capture

  2008cited by this paper
• Facial performance synthesis using deformation-driven polynomial displacement maps

  2008cited by this paper
• Unwrap mosaics: a new representation for video editing

  2008cited by this paper
• Space-time surface reconstruction using incompressible flow

  2008cited by this paper
• Mesh massage

  2008cited by this paper
• Rapid Acquisition of Specular and Diffuse Normal Maps from Polarized Spherical Gradient Illumination

  2007cited by this paper
• Multi-scale capture of facial geometry and motion

  2007cited by this paper
• Mirror MoCap: Automatic and efficient capture of dense 3D facial motion parameters from video

  2005cited by this paper
• Distinctive Image Features from Scale-Invariant Keypoints

  2004cited by this paper
• Extracting Animated Meshes with Adaptive Motion Estimation

  2004cited by this paper
• High Resolution Acquisition, Learning and Transfer of Dynamic 3‐D Facial Expressions

  2004cited by this paper
• Spacetime faces: high resolution capture for modeling and animation

  2004cited by this paper
• Cross-parameterization and compatible remeshing of 3D models

  2004cited by this paper
• Reanimating Faces in Images and Video

  2003cited by this paper
• Resynthesizing facial animation through 3D model-based tracking

  1999cited by this paper
• Modeling, tracking and interactive animation of faces and heads//using input from video

  1996cited by this paper
• The integration of optical flow and deformable models with applications to human face shape and motion estimation

  1996cited by this paper
• 3-D Motion Estimation in Model-Based Facial Image Coding

  1993cited by this paper
• Performance-driven facial animation

  1990cited by this paper
• Facial action coding system: a technique for the measurement of facial movement

  1978cited by this paper

• Single-Shot Facial Capture using Polarized RGB Sinusoidal Illumination

  2025cites this paper
• Facial Microscopic Structures Synthesis from a Single Unconstrained Image

  2025cites this paper
• Generative Head-Mounted Camera Captures for Photorealistic Avatars

  2025cites this paper
• YingVideo-MV: Music-Driven Multi-Stage Video Generation

  2025cites this paper
• Audio Driven Real-Time Facial Animation for Social Telepresence

  2025cites this paper
• Topo4D++: Realistic Physically Based 4D Head Capture With Topology-Preserving Gaussian Splatting and Expression Priors

  2025cites this paper
• PhonemeNet: A Transformer Pipeline for Text-Driven Facial Animation

  2025cites this paper
• Pixels2Points: Fusing 2D and 3D Features for Facial Skin Segmentation

  2025cites this paper
• Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Volumetric Performance Captures

  2025cites this paper
• Detail Enhanced Gaussian Splatting for Large-Scale Volumetric Capture

  2025cites this paper
• Convex Solutions to SfT and NRSfM Under Algebraic Deformation Models

  2025cites this paper
• Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Multi-View Performance Captures

  2025cites this paper
• ViSkin: Physics-Based Simulation of Virtual Skin on Personalized Avatars

  2025cites this paper
• IMUSIC: IMU-based Facial Expression Capture

  2024cites this paper
• Learning Topology Uniformed Face Mesh by Volume Rendering for Multi-view Reconstruction

  2024cites this paper
• Towards Geometric-Photometric Joint Alignment for Facial Mesh Registration

  2024cites this paper
• The impact of differences in facial features between real speakers and 3D face models on synthesized lip motions

  2024cites this paper
• Evaluation of a Structured Light Scanner for 3D Facial Imaging: A Comparative Study with Direct Anthropometry

  2024cites this paper
• Efficient 3D Implicit Head Avatar With Mesh-Anchored Hash Table Blendshapes

  2024cites this paper
• GenCA: A Text-conditioned Generative Model for Realistic and Drivable Codec Avatars

  2024cites this paper
• Universal Facial Encoding of Codec Avatars from VR Headsets

  2024cites this paper
• High Quality Human Image Animation using Regional Supervision and Motion Blur Condition

  2024cites this paper
• SPARK: Self-supervised Personalized Real-time Monocular Face Capture

  2024cites this paper
• Topo4D: Topology-Preserving Gaussian Splatting for High-Fidelity 4D Head Capture

  2024cites this paper
• VRMM: A Volumetric Relightable Morphable Head Model

  2024cites this paper
• High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse Rendering

  2024cites this paper
• Formulating facial mesh tracking as a differentiable optimization problem: a backpropagation-based solution

  2024cites this paper
• Strand-accurate multi-view facial hair reconstruction and tracking

  2024cites this paper
• Coherent 3D Portrait Video Reconstruction via Triplane Fusion

  2024cites this paper
• Polarimetric BSSRDF Acquisition of Dynamic Faces

  2024cites this paper
• Enhancing the Authenticity of Rendered Portraits with Identity-Consistent Transfer Learning

  2023cites this paper
• NTO3D: Neural Target Object 3D Reconstruction with Segment Anything

  2023cites this paper
• Learning Audio-Driven Viseme Dynamics for 3D Face Animation

  2023cites this paper
• A Novel 3D Face Reconstruction Model from a Multi-Image 2D Set

  2023cites this paper
• Towards Practical Capture of High-Fidelity Relightable Avatars

  2023influential citation
• ASM: Adaptive Skinning Model for High-Quality 3D Face Modeling

  2023cites this paper
• MEGANE: Morphable Eyeglass and Avatar Network

  2023cites this paper
• Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos

  2023cites this paper
• Accurate and Interpretable Solution of the Inverse Rig for Realistic Blendshape Models with Quadratic Corrective Terms

  2023cites this paper
• GAN-Avatar: Controllable Personalized GAN-based Human Head Avatar

  2023cites this paper
• 4D Facial Expression Diffusion Model

  2023cites this paper
• 3D Face Reconstruction and Gaze Tracking in the HMD for Virtual Interaction

  2023cites this paper
• XAGen: 3D Expressive Human Avatars Generation

  2023cites this paper
• A Sketch-Based Interface for Facial Animation in Immersive Virtual Reality

  2023cites this paper
• A Selective Expression Manipulation With Parametric 3D Facial Model

  2023cites this paper
• FLARE: Fast Learning of Animatable and Relightable Mesh Avatars

  2023cites this paper
• Anatomically Constrained Implicit Face Models

  2023cites this paper
• DisCO: Portrait Distortion Correction with Perspective-Aware 3D GANs

  2023cites this paper
• Face to Face Augmented Reality - Broadening the Horizons of AR Communication

  2023cites this paper
• Instant Multi-View Head Capture through Learnable Registration

  2023cites this paper
• Relightable Gaussian Codec Avatars

  2023cites this paper
• Laplacian face blending

  2022cites this paper
• 3D face reconstruction with dense landmarks

  2022cites this paper
• S2F2: Self-Supervised High Fidelity Face Reconstruction from Monocular Image

  2022cites this paper
• A Survey on RGB-D Datasets

  2022cites this paper
• Semantically Disentangled Variational Autoencoder for Modeling 3D Facial Details

  2022cites this paper
• Practical and Scalable Desktop-Based High-Quality Facial Capture

  2022cites this paper
• State of the Art in Dense Monocular Non‐Rigid 3D Reconstruction

  2022cites this paper
• How Accurate Is Passive Stereo For 3d Face Reconstruction?

  2022cites this paper
• Differentiable Simulation for Outcome‐Driven Orthognathic Surgery Planning

  2022cites this paper
• Rapid Face Asset Acquisition with Recurrent Feature Alignment

  2022cites this paper
• Monocular Facial Performance Capture Via Deep Expression Matching

  2022cites this paper
• Transferring Facial Expressions to Digital Character

  2022cites this paper
• Local anatomically-constrained facial performance retargeting

  2022cites this paper
• Facial hair tracking for high fidelity performance capture

  2022cites this paper
• Neural Parameterization for Dynamic Human Head Editing

  2022cites this paper
• Authentic volumetric avatars from a phone scan

  2022cites this paper
• A METHOD FOR TRIDIMENSIONAL FACIAL MODEL COMPOSITION

  2022cites this paper
• Leveraging Deepfakes to Close the Domain Gap between Real and Synthetic Images in Facial Capture Pipelines

  2022cites this paper
• Shape Transformers: Topology‐Independent 3D Shape Models Using Transformers

  2022cites this paper
• ABBA Voyage: High Volume Facial Likeness and Performance Pipeline

  2022cites this paper
• On obtaining sparse semantic solutions for inverse problems, control, and neural network training

  2021cites this paper
• Towards High Fidelity Monocular Face Reconstruction with Rich Reflectance using Self-supervised Learning and Ray Tracing

  2021cites this paper
• Rendering with style

  2021cites this paper
• Semi-supervised video-driven facial animation transfer for production

  2021cites this paper
• Generating 3D Facial Expressions with Recurrent Neural Networks

  2021cites this paper
• Face Translation based on Semantic Style Transfer and Rendering from One Single Image

  2021cites this paper
• Deep Insights of Deepfake Technology : A Review

  2021cites this paper
• Simplified facial capture with head mounted cameras

  2021cites this paper
• Single-view facial reflectance inference with a differentiable renderer

  2021cites this paper
• 4D Atlas: Statistical Analysis of the Spatiotemporal Variability in Longitudinal 3D Shape Data

  2021cites this paper
• Plausible 3D Face Wrinkle Generation Using Variational Autoencoders

  2021cites this paper
• High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation

  2021cites this paper
• Facial Modelling and Animation: An Overview of The State-of-The Art

  2021cites this paper
• Realtime and Accurate 3D Eye Gaze Capture with DCNN-Based Iris and Pupil Segmentation

  2021cites this paper
• Practical Face Reconstruction via Differentiable Ray Tracing

  2021cites this paper
• Single-shot high-quality facial geometry and skin appearance capture

  2020cites this paper
• Deep relightable textures

  2020influential citation
• Data-driven extraction and composition of secondary dynamics in facial performance capture

  2020cites this paper
• Accurate Real‐time 3D Gaze Tracking Using a Lightweight Eyeball Calibration

  2020cites this paper
• Monocular Real-Time Volumetric Performance Capture

  2020cites this paper
• Robust Methods for Dense Monocular Non-Rigid 3D Reconstruction and Alignment of Point Clouds

  2020influential citation
• Dense Non-Rigid Structure from Motion: A Manifold Viewpoint

  2020influential citation
• Audiovisual Speech Synthesis using Tacotron2

  2020cites this paper
• Dynamic facial asset and rig generation from a single scan

  2020cites this paper
• Real-time Face Video Swapping From A Single Portrait

  2020cites this paper
• Battery Electric Vehicle adoption in regions without strong policies

  2020cites this paper
• Data-Driven 3D Neck Modeling and Animation

  2020cites this paper
• Learning Formation of Physically-Based Face Attributes

  2020cites this paper
• MoveBox: Democratizing MoCap for the Microsoft Rocketbox Avatar Library

  2020cites this paper