PRODUCTION OF MULTI-MEDIA CONTENT FOR THREE-DIMENSIONAL ENVIRONMENTS DISTRIBUTED OVER NETWORKS:

Prometheus

Convergence of computer and broadcast technologies provides the potential to change television and film content from a 2D form where a producer defines every moment, to an interactive, immersive 3D form where a viewer can manipulate the scene or plot and view a programme on a 3D display. The objective of the Prometheus project is to demonstrate content creation, transmission and display of 3D broadcast material. Prometheus is a three-year LINK project under the Broadcast Technology Programme funded by the UK DTI and EPSRC, completing August 2002, undertaken by a collaborative team of industrial and academic partners. The Prometheus web pages can be found at the Prometheus Home-Page.

The focus of the University of Surrey is on the creation and animation of virtual actors for 3D broadcast content. Current production methods composite real and virtual scene elements using chroma-key technology in a Virtual Studio. The objective is to extend current techniques to a multi-camera virtual production studio to capture 3D for the rapid creation of photorealistic virtual humans and sensor-free, markerless motion capture for actor animation.

Virtual Actors

Recent research at the University of Surrey had demonstrated the feasibility of low-cost reconstruction of photo-realistic 3D models of people from a set of colour images. This technology was commercialised by the company AvatarMe and used to capture 250,000 people at the Millenium Dome in the UK. The system was awarded a 2001 EU IST Prize. These techniques have been further developed and applied in the calibrated multi-camera virtual studio. The pipeline for the creation of virtual humans is illustrated below.

Studio capture of actor appearance from multiple calibrated cameras

Extraction of 3D shape information from images (6 views)

Matching a generic animated virtual human to the 3D shape

Extracting model colour texture from images (Ongoing Research)

The animated virtual actor, with preliminary colour texture, can be seen in a 3D scene at Dancing in Venice

Markerless Motion Capture

Using image processing and pattern recognition techniques, applied to video from multiple cameras, we aim to measure the movements of an actor in detail. These can be used to re-animate the existing model and view it from any viewpoint, or animate a different character altogether, but with the same movements as the real actor. The two video sequences linked below contain stick-diagrams showing animation information, overlaid on the corresponding video sequences. In the simple arm movement sequence, extraction of the animation parameters is fully automatic. For more complex sequences such as the pirouette, some manual intervention is required. Ideally the whole process should be automated, and this is the topic of ongoing research.

Arm movement (port de bras)
Whole body movement (pirouette)

J.Starck@eim.surrey.ac.uk
J.Mitchelson@eim.surrey.ac.uk