• Character Animation
• Machine Learning for Motion Synthesis
• Parallel and Hybrid Computing for Animation
• Interactive Design for Computer Games
• Computer Simulation for Education and Training

Trauma Room Simulation

This project is a realistic medical simulation in a 3D virtual environment, or a video game for medical training. We worked with Carilion Memorial Hospital at Roanoke, VA and developed this system to train residence doctors how to handle emergency situation in Trauma room. The system includes animation, audio, simulation and camera components. The simulation is data-driven, so that the senior doctor can author different training senarios with the tools we provide.

The system has been deployed to the hospital for training sessions. The future plan has been decided for more medical and educational training.

Demo Video Download: (720x480) (35.5M)

Real-time speech motion synthesis

Data-driven approaches have been successfully used for realistic visual speech synthesis. However, little effort has been devoted to real-time lip-synching for interactive applications. In particular, algorithms that are based on a graph of motions are notorious for their exponential complexity. In this work, we present a greedy graph search algorithm that yields vastly superior performance and allows real-time motion synthesis from a large database of motions. The time complexity of the algorithm is linear with respect to the size of an input utterance. In our experiments, the synthesis time for an input sentence of average length is under a second. The performance is satisfied in an interactive virtual environment, such as Video Games.

Demo Video Download: (640x480) (41.9M)

No One Ever Liked Me! - Monologue created by Speech-driven Motion Synthesis

The motion in this animation was created using our novel techniques for automatical expressive speech motion synthesis. The input of our system is a spoken utterance and a set of emotional tags. Its output is a realistic facial animation that is synched to the input audio and conveys faithfully the specified emotions.

The story is taken from play "The Food Chain" by Nicky Silver. The monologue No One Ever Liked Me! in this animation is an alternate ending coming at the end of a running tirade by Otto, a hugely overweight, insecure, rage-filled, Jewish, out-of-control verbal tornado. Otto's got a gun.

Video Download: (720x540): No One Ever Liked Me! (15.5M)

Interactive Motion Decomposition

We introduce a novel method for editing the style of motion data through motion decomposition. Our method extracts the style of a motion using linear decomposition based on Independent Component Analysis. The extracted style components are applied to other motions through a variety of editing operations. The resulting motions retain their original basic content while exhibiting the style of a different motion.

Supporting Video Download: (640x480) (21.7M)

Facial Animation Editing with Independent Component Analysis (ICA)

We present a new method for editing speech related facial motions. Our method uses an unsupervised learning technique, Independent Component Analysis (ICA), to extract a set of meaningful parameters without any annotation of the data. With ICA, we are able to solve a blind source separation problem and describe the original data as a linear combination of two sources. One source captures content (speech) and the other captures style (emotion). By manipulating the independent components we can edit the motions in intuitive ways.

Supporting Video Download: (480x360)

Data-driven Visual Speech with Emotion Control

We present a set of novel techniques for automatically synthesizing speech-driven expressive facial animation. The input of our system is a spoken utterance and a set of emotional tags. These emotional tags can be specified by a user or extracted from the speech signal using a classifier. Its output is a realistic facial animation that is synched to the input audio and conveys faithfully the specified emotions. This approach relies on a database of high-fidelity ecorded facial motions. This database includes speech-related motions with variations across multiple emotions. Our main contribution is a system that is able to generate expressive speech facial animation with real-time performances.

Demo Videos (640x480)(54.7M) for Synthesis Speech Download.