Would you like a computer game avatar that looks just like you? Or enhance your features with those of someone incredibly beautiful? If yes, then a bit of maths will come in handy.
Who's this?
(Original image by Pete Souza, CC BY 3.0. Image fusion by Martin Benning, Michael Möller, Raz Z. Nossek, Martin Burger, Daniel Cremers, Guy Gilboa, and Carola-Bibiane Schönlieb.)
At this year's Cambridge Science Festival mathematicians from the Cambridge Image Analysis Group will present software that performs face fusion: if you enter an image of your face and an image of someone else's face, the software will merge the two, resulting in a strange combination of both. The idea behind it is familiar from music, when different tracks of a song are mixed together. A sound can be decomposed into its component frequencies, and you can then adjust those frequencies separately using an equaliser (see this article to find out more). In the case of the images, it's not sound frequencies that are isolated, but structures within an image that display a particular level of detail. For example, the wrinkles on someone's face display a particular level of detail (hopefully a very fine one). If you can isolate structures with that level of detail and adjust them, you can enhance or diminish the wrinkles. Alternatively, you could replace them with the wrinkles you've isolated on someone else's face — that's the basic idea behind our face fusion. And it can all be done mathematically.
At the heart of the face fusion process lies a type of diffusion equation. The most famous diffusion equation is the heat equation, which describes how heat spreads (diffuses) through a material. For example when you hold one end of a metal rod held into a fire the heat will gradually spread towards your hand. When it comes to image analysis and image processing, we can use diffusion equations, not to spread heat around, but to spread information from the various pixels that make up the image. The diffusion equation used here is more complicated than the ordinary heat equation, and it preserves the larger features of the image, such as the overall shape of the face. It is known as the total variation (TV) flow and it is represented by a differential equation.
The result of applying the TV flow is a blurring of the image: as you can see in figure 1, more and more details are removed as you move through the image sequence, with the image becoming more and more blurred.
Figure 1: Example of the TV flow blurring an image. (Underlying image by Dyor, CC BY-SA 3.0. Image processing by Chris Irving.)
Since the TV flow is a mathematical object, you can also use mathematics to observe exactly how the flow changes over time. That is, you can see how structures of a particular level of detail (fine or coarse) respond to it — this gives you a way of getting hold of these structures, and a mathematical tool called the TV transform can be used to separate them out. Figure 2 shows a sequence of images containing structures of decreasing level of detail, fine on the left to coarse on the right.
Figure 2: The TV transform separating out structures of varying level of detail, from fine to coarse. (Underlying image by Dyor, CC BY-SA 3.0. Image processing by Chris Irving.)
Now that these components have been separated out, you can manipulate them individually, just as an equaliser manipulates individual frequencies in a song. You can also reconstruct the original picture (or the original picture without wrinkles) simply by "adding" the various components (or all the components except the ones containing wrinkles). The mathematics behind the TV transform was inspired by the mathematics behind sound processing and has resulted in a powerful tool.
We're now ready for some face fusion. Suppose you'd like to enhance (or perhaps spoil?) Barack Obama's face with some of the features of Ronald Reagan. You use the mathematics to remove a lot of the structure from Obama's face, so that you are left with a "background" image of him. Depending on what kind of effect you'd like to achieve, you might also decide to leave some of Obama's features, such as the eyes, mouth and hair, unchanged: this will ensure that Obama is still recognisable, even after Reaganification. Using the TV transform you then separate the fine and medium scale structures from Reagan's face, such as his wrinkles and the puffy cheeks. Then, after carefully aligning the images, you superimpose those features on Obama's background image.
Figure 3: Image of Obama (left) and the image equipped with some of Reagan's features (right). (Original image by Pete Souza, CC BY 3.0. Image fusion by Martin Benning, Michael Möller, Raz Z. Nossek, Martin Burger, Daniel Cremers, Guy Gilboa, and Carola-Bibiane Schönlieb.)
Similarly, you can start with a background Reagan and superimpose Obama's fine and medium scale features. Which one of the two is better looking, Ronald Obama or Barack Reagan, is up to you to decide.
Figure 4: Image of Reagan (left) and the image equipped with some of Obama's features (right). (Image fusion by Martin Benning, Michael Möller, Raz Z. Nossek, Martin Burger, Daniel Cremers, Guy Gilboa, and Carola-Bibiane Schönlieb.)
The mathematics developed for the face fusion software is likely to find a range of applications, for example in the computer gaming industry and eventually perhaps also in the movies. Next time you're impressed by some particularly spectacular computer generated imagery, give a thought to mathematics! And if you can, head down to the Centre for Mathematical Sciences on March 25th to play with the software at the Cambridge Science Festival Maths Public Open day.
You can read more about image processing using diffusion equations in this Plus article.
About this article
We'd like to thank Martin Benning, Leverhulme Early Career Research Fellow at the University of Cambridge, for his help in putting together this news story.