MIMiC: Multimodal Interactive Motion Controller

We introduce a new algorithm for real-time interactive motion control and demonstrate its application to motion captured data, pre-recorded videos and HCI. Firstly, a data set of frames are projected into a lower dimensional space. An appearance model is learnt using a multivariate probability distribution. A novel approach to determining transition points is presented based on k-medoids, whereby appropriate points of intersection in the motion trajectory are derived as cluster centres. These points are used to segment the data into smaller subsequences. A transition matrix combined with a kernel density estimation is used to determine suitable transitions between the subsequences to develop novel motion. To facilitate real-time interactive control, conditional probabilities are used to derive motion given user commands. The user commands can come from any modality including auditory, touch and gesture. The system is also extended to HCI using audio signals of speech in a conversation to trigger non-verbal responses from a synthetic listener in real-time. We demonstrate the flexibility of the model by presenting results ranging from data sets composed of vectorised images, 2D and 3D point representations. Results show real-time interaction and plausible motion generation between different types of movement.

[Image showing synthesis and blending of different types of human articulated motion captured walks]

[Image showing candle flame synthesis]

[Image showing plasma beam synthesis. Arrows indicate direction of synthesis flow]

[Image showing synthesis of nods as a response to audio stimulus]

Dumebi Okwechime
Eng-Jon Ong
Richard Bowden


Real-Time Motion Control Using Pose Space Probability Density Estimation
Okwechime, D. Ong, E-J. Bowden, R.
In IEEE International Workshop on Human Computer Interaction, ICCV09, Kyoto, Japan 2009.
pdf 1.6 MB] [bibtex]

MIMiC: Multimodal Interactive Motion Controller
Okwechime, D. Ong, E-J. Bowden, R.
To appear in IEEE Transactions on Multimedia, 2011
DOI: 10.1109/TMM.2010.2096410 
pdf 3 MB] [bibtex]

TMM2011 Supplementary Material - Includes Audio (3:09 mins)
(Requires Quicktime to view)
[Video 6.8 MB]