Visualisation and Prediction of Conversation Interest through Mined Social Signals

This paper introduces a novel approach to social behaviour recognition governed by the exchange of non-verbal cues between people. We conduct experiments to try and deduce distinct rules that dictate the social dynamics of people in a conversation, and utilise semi-supervised computer vision techniques to extract their social signals such as laughing and nodding. Data mining is used to deduce frequently occurring patterns of social trends between a speaker and listener in both interested and not interested social scenarios. The confidence values from rules are utilised to build a Social Dynamic Model (SDM), that can then be used for classification and visualisation. By visualising the rules generated in the SDM, we can analyse distinct social trends between an interested and not interested listener in a conversation. Results show that these distinctions can be applied generally and used to accurately predict conversational interest.

[(a) Image showing full-body view of recorded video data of three people having a conversation. (b) Image showing close-up face view. (c) Diagram showing the configuration of cameras, microphones (mic) and conversers. We refer to the three people in the conversation as person 1, 2, and 3]

[(a) The skeleton of SDM. Consists of 7 black lines that are attached to a central intersection. Each line represents a different speaker's social signal (b) 9 pentagon rings where each ring represents a listener's social response. The individual rings are coded by colour and size. (c) SDM is made up of the rings superimposed on the skeleton. The points where the rings intersect the skeleton are known as nodes and infer a listener's social response given a speaker's social signal]

[(a) SDM generated from the mined interested listener's confidence values. (b) SDM generated from the mined not interested listener's confidence values]

[Prediction percentage scores using varying frame windows for each person. The performance increases to 90% for a time window of approximately 4.5 minutes]

Dumebi Okwechime
Eng-Jon Ong
Andrew Gilbert
Richard Bowden


Snapshot 2011-02-07 14-52-35
Visualisation and Prediction of Conversation Interest through Mined Social Signals
Okwechime, D. Ong, E-J. Gilbert, A., Bowden, R.
In IEEE International Workshop on Social Behavior Analysis, FG2011, Santa Barbara, CA 2011.
pdf 960 KB] [bibtex]