Disney Research develops Makeup Lamps, a projector of images and light in real time on the face

Components of this laboratory, in collaboration with technicians from other universities, have presented the details of this prototype that allows the actors not to have to put on makeup thanks to the projection of light on their faces in time, whose challenge is to reduce latency from capture to processing.

El sistema Makeup Lamps, in prototype, It is the first that allows dynamic augmentation in real time on human faces, in this case actors, based on an image and light projector that alters the appearance of the face. A project resulting from the research of Amit Bermano and Anselm Grundhöfer, from the laboratory Disney Research Zurich (Swiss), in collaboration with Markus Billeter, from Chalmers University of Technology (Sweden) y Daisuke Iwai, from Osaka University (Japan).

One of the challenges of this system, that allows actors to be dynamically 'made up' in their performances, has been the latency in each step of this process: from capture to processing, since, as the document published by Disney Research points out, an image is generated according to a specific pose, but is shown in a different facial configuration at the time it is projected, and which have now reduced or almost eliminated.

For this, Makeup Lamps use infrared lighting, a high-speed camera and projector integrated into computer-aligned optics to detect details and facial expressions, which allows synchronizing the reading of the face and the projection of the captured image on it.

The interpretation algorithms of this spatial animation are mapped to a lower dimensional space, and facial movement and deformation are smoothed, performing an estimation of all parameters to predict changes in facial expression with an adaptive Kalman filter.

Subsequently, it is possible to create the desired appearance thanks to the interpolation of offset textures and superimposing different images on the face or replacing it with that of another person. (face2face), depending on time, the expression, etc, no need for markers or tracking sensors.

[youtube]https://youtu.be/Ilgu3aFCphs[/youtube]