Computer generated three-dimensional (3D) graphics has been widely used for the production of visual contents in the film, TV broadcasting and game industries. One of the key challenges for 3D content production is the creation of a realistic performance of human. The whole process is time-consuming and labour-intensive requiring dozens if not hundreds of highly skilled artists and animators working together over years.

The idea is to provide an automated process for realistic 3D virtual characters production by using Computer Vision and Graphics techniques to capture, reconstruct and reanimate human performance from the real world.
4D Performance Capture
Motion Capture techniques (MoCap) provide a convenient way to capture human skeletal motion in the real world and allow motion retargetting to a virtual character. MoCap has been extensively developed for decades and successfully used for research and in industries. However, skeletal MoCap does not capture the surface movements, e.g. moving wrinkles in clothing and the motion of hair, or surface appearance, e.g. costume and facial expressions. These surface movements and appearance are critical to realism and hard to reproduce afterwards.

4D Performance Capture (4DPC) has been introduced recently to capture both shape, appearance and motion of the human body from multiview video. The outcome of 4DPC is a sequence of reconstructed 3D meshes with detailed surface dynamics and video-quality textures. 4DPC meshes are temporal consistent, i.e. the number of vertices and the connectivities are unchanging, making them compatible with conventional animation pipelines. To achieve temporal consistency across multiple sequences, Global temporal non-rigid surface registration/alignment is introduced.

  • A global non-rigid surface alignment framework is introduced to minimise the deformation required to register all meshes from different input sequences into a temporally consistent structure. The input sequences are first independently temporally aligned and then cross sequences aligned through frames with similar 3D shape.
    [Huang et al. 2011]
  • Further global non-sequential alignment provides a means to construct a Minimum Spanning Tree (MST) to include all unaligned frames based on their 3D shape similarity and performing global alignment across the tree.
    [Budd et al. 2013] [Budd et al. 2011]
Character Animation
By simultaneously capturing motion and appearance, 4DPC offers unique advantages over skeletal MoCap for the creation of realistic 3D virtual character animation. However, there is limited tool support and limited reusable capture data available to enable 4DPC based character animation. Recently, Surface Motion Graphs, Hybrid Skeletal-Surface Motion Graphs and 4D Parametric Motion Graphs are introduced for creating character animation using 4DPC data.

  • Surface Motion Graphs as an analog to Motion Graphs for MoCap representing possible transitions with similar shape and motion between sequences, provide a framework for concatenative human motion synthesis. The system is key-frame based and supports user constraints on movement, position and timing. NOTE: Surface Motion Graphs when first introduced does not require temporal consistent mesh sequences.
    [Huang and Hilton 2009] [Huang et al. 2009]
  • Hybrid Skeletal-Surface Motion Graphs provide a means to drive a high-fidelity 4DPC character animation with readily available skeletal MoCap data extends the range of motion which can be produced and promises to accelerate the uptake of 4DPC for animation production.
    [Huang et al. 2015]
  • 4D Parametric Motion Graphs provide a means to interactive control 4DPC based character animation. High-level motion parameterization is introduced by real-time blending of multiple mesh sequences and textures.
Similarity Measure
To support global temporal alignment for non-rigid surface sequences and motion graphs based character animation, similarity measure between reconstructed 3D meshes with/without texture are extensively studied.

  • 3D Shape Histograms are first introduced to provide an accurate similarity measure for motion/poses of the same person in the same costume. Its temporal variants and a quantitative evaluation comparing different 3D shape descriptors are also provided.
    [Huang et al. 2010] [Huang et al. 2007b] [Huang et al. 2007a]
  • Further extension to 6D Shape-Colour Histograms measure 3D shape similarity as well as appearance similarity to facilitate seamless transitions requirement for motion graph based character animation.
    [Huang et al. 2015] [Huang and Hilton 2009]
Free-viewpoint Video Rendering
To support video-quality realistic character animation, several Free-viewpoint Video Rendering techniques are implemented.

  • View-dependant Rendering: original captured multiview images are projected back to a reconstructed 3D mesh seamlessly blending together with a weighting scheme favouring the most direct view. This maximises the visual realism.
  • Texture Extraction: with temporal consistent 4DPC data, a single texture can be extracted for a whole sequence or per frame. This provides a compatible texture rendering solution to conventional animation pipelines.
Research Projects
REFRAME: Location Performance Capture for Real-Time Enriched Media (TSB)
RE@CT: Immersive production and delivery of interactive 3D content (EU)
Video-based Animation of People (EPSRC)
[Huang PhD Thesis]
Video Special Effects
[Huang MRes Thesis]