Facial Feature Tracking

Why Tracking?

Tracked are useful for recognition of non-verbal communication because the feature’s motion and relative position are transformed into a computer usable form. A tracker predicts the position of a facial feature throughout a continuous video. Other emotion recognition approaches use appearance based methods (e.g. optical flow, edge detection) or model fitting (e.g. AAMs) instead of tracking.

Challenges of Tracking

Spontaneous human communication usually contains fast motion of the head across a wide range of poses. Humans also tend to gesture with their hands and scratch their face, which can obscure part or all of the view. The wide range of expressions and variation in peoples face shapes makes robust facial feature tracking very challenging.

Online vs. Offline Tracking

Tracking uses the appearance of surrounding pixels to predict the position of a feature. The appearance can either be fixed (offline) or attempt to adapt to changes in the feature appearance (online tracking). A common problem is unrestricted adaptation combined with noisy tracking leads to tracking drift. A balance between flexibility and robustness needs to be struck.

We propose a facial feature tracking system that uses head pose to control the online learning rate to improve tracking across various head poses. The head pose is estimated using Levenberg-Marquardt (LM) minimisation using the tracked features as constraints.

Linear Predictor Flock trackers

An alternate approach to using alignment of image patches (Lucas-Kanade, etc) is to use linear predictor (LP) tracking. LP tracking uses image intensity information in a sparse template with a linear mapping to obtain an estimated image offset. The linear mapping is learned off-line based on manually marked training frames. The tracker performs more robustly and accurately when extended by rigid flocks and hierarchical tracking.

This method is used in the NVC automatic recognition system.

Last update: July 2009.