A team of U.S. and British researchers have created a software program that uses high-level mathematics to turn facial expressions, such as the scrunch of an eyebrow or a wide-eyed stare, into digital art.
The program, known as “empathic painting,” results in a distinct stylized interpretation of an existing photograph, one that gently changes like flowing waves according to a person’s mood.
It falls into a category of research called non-photorealistic computer graphics, an area that has grown over the last 10 years, said John Collomosse of the computer science department at the University of Bath in England.
“We’re interested in the borderline of computer graphics, the output of images, and computer vision, the input of images, and these two disciplines are converging,” said Collomosse, who holds a doctorate’s degree in artistic rendering, images and video.
Over the course of three months, the team sought to see how an algorithm could interpret a person’s facial movements with minimal end-user interaction.
The use of algorithms to create paintings from photos is nothing new. Commercial products such as Adobe’s Photoshop program has a feature to turn still photographs into digital paintings or charcoal renderings.
Academics have published advanced algorithms, but employing them is difficult for nontechnical users, often involving lots of controls for very low-level aspects of the artistic process, Collomosse said. Those low-level parameters include the angle and length of brush strokes.
End users, however, are more interested in higher-level controls, such as if they want a bright and cheery painting or a dark, gloomy one, Collomosse said. Consequently, few of those advanced algorithms have made their way into consumer products, he said.
“I think that’s why this research is interesting,” Collomosse said. “The empathic painting is trying to look at how we can make these sophisticated algorithms more usable.”
The team, which included Maria Shugrina and Margrit Betke of the computer science department at Boston University, set to work using a simple webcam and off-the-shelf hardware. The researchers used a machine with a Pentium 4 2.8GHz processor and a GeForce 6000 graphics accelerator made by Nvidia.
The customized digital painting starts with a real photograph. The software takes the image and breaks it into segments. In a photo of a boat, the boat’s body may be one segment.
The segment is then is turned into three-dimensional brush strokes. Finer strokes fill in the smaller details.
On the other end, a person’s face is broken in two dimensions, an “x” and “y” axis, where emotions are mapped according to their expression.
Using the webcam, the algorithm “reacts” to changing cues, such as the curling of the mouth or wideness of the eyes, and changes the painting accordingly, adding darker colors for sad expressions or suddenly flooding the scene with color for a smile.
“Users felt engaged with the system, remarking on the intuitive nature of the interface,” the researchers wrote in a paper. “They felt able to easily control the style of the painting to produce their desired results.”
The researchers recognize some limitations. The graphics component relies on a subjective mapping of a person’s emotional state and the parameters of the painting algorithm. Also, differences in culture may affect how a person’s emotional state is perceived.
So far, there’s no marketing plan for the empathic painting algorithm. But with the rise in popularity of digital photography over the last five years, demand is increasing for software systems to manage and manipulate digital content.
“If someone wants to take a photo and turn it into a painting on a wall, it is a viable product because there are products out there that do it,” Collomosse said. “This is another way of going about that.”
A video of empathic painting in action is available here.
Add a comment | More DWT | Get a cool job in Japan! | Follow us on Twitter