Researchers develop 4D movie experience
The cyber-tech race to put us in the picture is hotting up, reports Tom Pullar-Strecker in the latest of our innovation series.
Imagine you were watching a battle scene in a film on a virtual-reality headset and could pick up computer-generated "sword" that was dropped in front of you, feeling it in your hand.
Then imagine you could throw it to either the hero or the villain, changing the plot which unfolds.
Researchers at Victoria University in Wellington and the Human Interface Technology Laboratory (HIT Lab) at Canterbury University, are working with academics in Korea on a joint project funded by both governments to bring that kind of "4-D" interactive movie experience to people's living rooms.
Their goal is to knock off the technology challenges one-by-one over the next three years, and get to the point where they can produce a convincing working "demo", using special "haptic" gloves that will mimic the sensations of touch and perhaps texture.
The end product of the New Zealand-Korean research effort may be as simple - and complicated - as letting someone see and feel the sensation of reaching out and picking up a computer-rendered cup.
The idea is the cup will be integrated into a movie people are watching on an Oculus Rift headset.
Leading the New Zealand research teams is Taehyun Rhee, who worked for Samsung for 17 years before swapping industry for academia.
He moved to Victoria University's School of Engineering and Computer Science two years ago.
Simulating the experience of physically interacting with other people would pose extra challenges, he says.
But he doesn't rule out that it might be possible to virtually shake Clint Eastwood's virtual hand in a 4-D western.
Your fingers and imagination could of course run wild.
"It is possible we could simulate touch with a virtual human being, but we know 'us' very well. So, if there is a slight abnormality we easily pick that out," Rhee says.
"If there is an abnormality picking up a cup we have never seen before, and isn't in our 'database' in the brain, that can be 'excused'. But human faces and movements and expressions are totally in our brains."
When virtual reality falls short, the common result is discomfort and nausea. "If we are aiming this technology at the home entertainment market, and perhaps children, we have to solve any issues around discomfort and that is a big part of our target," Rhee says.
The research project received a $450,000 grant from the Business, Innovation and Employment Ministry.
Film companies in Wellington are taking an interest, he says, but he can't name names.
The team at Victoria University is responsible for creating a photo-realistic visual experience, which will allow objects to meld with the background scene of a movie, as they are manipulated.
Canterbury's HIT Lab is working on technology needed to track the hands of viewers and reintegrate the computer-generated renditions of their hands back into the movie scene.
Meanwhile, researchers at Korean University and Ewha Woman's University, both in Seoul, are tackling some the robotic challenges, such as designing the haptic gloves.
With expectations that "virtually reality" headsets such as the Oculus Rift could take the home entertainment by storm, it is the kind of endeavour that may be needed to keep New Zealand's burgeoning $1.5 billion film production and post-production industry at the cutting edge.
It is usually assumed the biggest market for Oculus Rift, which was acquired by Facebook for US$2 billion last year, will be in computer gaming.
But this month the virtual reality headset-maker began dabbling in the movie business, setting up an in-house studio to make films that could show off the potential of viewing films in a 360 degree 3-D panorama.
One knotty issue is how 4-D film directors of the future might pick up the plot once a movie viewer had interacted with one of their virtual props.
Would viewers have to place the object they picked up back in a predefined spot so the movie could continue seamlessly to a set plot without a break in the visuals?
Or could they perhaps alter the plot by interacting with it unpredictably, as with the sword-throwing example? That would mean movie makers would have to film for different scenarios. But what would happen if the movie-viewer threw the sword over their shoulder?
Rhee admits it could get very complicated from a creative perspective. "Technology-wise it is the same. It is an open discussion."
- Sunday Star Times