Facial Motion Retargeting with Input-Output Temporal RBMs

Project Members

Matt Zeiler (New York University)
Graham Taylor (New York University)
Rob Fergus (New York University)
Leonid Sigal (Disney Research Pittsburgh)
Iain Matthews (Disney Research Pittsburgh)



Retargeting with the 3rd-order factored Input-Output Temporal RBM. The top row shows the input (from a source subject). The bottom row shows the predicted output from the proposed model (crosses).


Retargeting facial motion from one actor to another is a complex task. Current methods include: Blendshape Mapping, where target face is expressed as a combination of key shapes with the weight given by source data and Geometric Mapping, where expressions of the target face are modelled as a source face offset with respect to the base target geometry. These methods, however, do not take into account dynamic aspects of the facial motion itself, making it hard, for example, to model speech.

We present a type of Temporal Restricted Boltzmann Machine that defines a probability distribution over an output sequence conditional on an input sequence. It shares the desirable properties of Restricted Boltzmann Machines: efficient exact inference, an exponentially more expressive latent state than HMMs, and the ability to model nonlinear structure and dynamics. We apply our model to facial expression transfer. Results demonstrate improved performance over several baselines in 2D and 3D retargeting


Facial Expression Transfer with Input-Output Temporal Restricted Boltzmann Machines-Thumbnail

Facial Expression Transfer with Input-Output Temporal Restricted Boltzmann Machines
December 12, 2012
Neural Information Processing Systems (NIPS) 2011
Paper File [pdf, 2.32 MB]

Copyright Notice

The documents contained in these directories are included by the contributing authors as a means to ensure timely dissemination of scholarly and technical work on a non-commercial basis. Copyright and all rights therein are maintained by the authors or by other copyright holders, notwithstanding that they have offered their works here electronically. It is understood that all persons copying this information will adhere to the terms and constraints invoked by each author's copyright. These works may not be reposted without the explicit permission of the copyright holder.