Virtual actors closing in on the real world

Paul Debevec does not want to put actors out of work, but the technology he will showcase at this week's AnimfxNZ conference in Wellington might do that.

Shedding light on virtual actors from nearly 500 directions, the head of University of Southern California's Institute for Creative Technologies is trying to recreate the human form in photo-real imagery, bringing computer-generated animation closer to real life.

One of several speakers at the animation, games and visual effects conference, Dr Debevec's ideas are at the heart of bringing the viewing and interactive digital experience to life. With his technology already used in movies like The Matrix, The Curious Case of Benjamin Button and inspiring techniques used in films like King Kong, Dr Debevec says the pathway he is forging is recreating the way digital humans are created for films and games.

Before getting into lighting, he was digitally mapping physical landscapes in the 1990s, such that three-dimensional recreations became possible and were adopted for the background of the freeze-frame bullet sequences in The Matrix.

"It was not lighting per se but it got me set along the path of using images more and more as first-class building blocks of how you create computer graphics," he says.

Computer imagery was traditionally built out of polygons and geometric shapes overlaid with texture maps and colouring. Mr Debevec says the idea was to try to turn 2D pictures into 3D models and use them for the basis of computer graphics, compared with trying to build it from scratch by hand.

Integrating computer-generated characters into real footage has been a developing technology for years, but University of Southern California technology, built by Dr Debevec's team, has focused on the problematic aspect of making fake actors fit next to real actors.

"I wanted to be able to link together the virtual world of computer-generated images and the real world of photographs, and mix and match them freely. Lighting was really the key to this."

While still not common in film-making circles, Dr Debevec's light-stage concept is commercially available in Hollywood. He considers it "mature technology".

The sphere of cameras and strobes was the set for the Digital Emily project, which mapped an actor's facial expressions 33 ways, in all lighting conditions.

Each "grab" takes only three seconds, as the many cameras take thousands of photographs in fast-changing light conditions, mapping the actor's face in every scenario. The photos were combined into a 3D image database that could recreate the face, and digitally manipulate it though different expressions.

"We also did some scanning of Naomi Watts for King Kong with Weta. We lit her from 480 different lighting directions, and that gave them a database of how Naomi Watts' skin reflected light so they could make the digital version reflect light in the same way."

The same technology was used to map out Brad Pitt's face for The Curious Case of Benjamin Button.

Dr Debevec says actors are not under threat from animators. His technology brings virtual imagery closer to real life but humans cannot yet be replaced for genuine expression and movement.

"We're trying to create a puppet, for lack of a better word. We haven't done anything yet to threaten the craft of acting. It might be that we can create more realistic versions of actors for stunt scenes, or if we need them younger or older, of if we need to bring an actor back to life.

"All of those things become possible because now the way the actor looks on screen doesn't have to be the way the actual person doing the acting looks. Gollum was an early example. We are not trying to get rid of the actor, we are just giving more options for what they end up looking like when they are on the screen."

Financially, the chance to recreate scenes or change an action in a frame once a shoot has finished adds a whole new creative element to the post-production process of making a movie without having to expensively reshoot a scene.

Editors and cinematic artists in movie post-production retouch frame after frame of footage to ensure continuity and smoothing cuts and textures and Dr Debevec's goal is to free them up to do what they want to be doing. "Ultimately what the technology will do is allow artistic effort to be spent more creatively. A lot of the stuff could be less labour intensive."

Editing a film is often working in a fixed environment, with all the footage "in the can" and only flexible to the point that there are alternative takes.

Dr Debevec says new technologies give editors options to change lighting, camera angles and other variables, without the "financial ruinous" cost of reshooting.

"We'll have a digital human that will does what it needs to do. At that point it's up to you what you want to do with it in terms of the storytelling. You won't have to be tweaking every little bit of it for every single shot."

While Hollywood has picked up the technology, gaming is where the true potential of Dr Debevec's ideas comes into play.

"In a video game you don't know the camera viewpoint ahead of time, because the character can walk around. You can't just use real video of people, and you need a library of different emotions and options so the characters can respond to the environment."

He says the potential to create photo-real virtual characters in games will boost the emotional engagement. "Hopefully that will elevate that art form and what people can do with it creatively."

- © Fairfax NZ News