Visual effects artists now have the power to bring the dead back to life through the development of digital actors – and they’d even be able to make them cry if one scooped an Oscar. Digit talks to the CG pioneers and effects makers about the march of the virtual actor.
Virtual actors – or synthespians – are looking for their name up in lights. And they’re not going to be disappointed.
It was only 18 years ago that the first full CG actor made his bit-part debut in a movie. Little more than a walk-on part, a 3D knight created by Pixar in the film Young Sherlock Holmes, had to smash through a window. The effect was simplistic, but used the then brand-new motion-blur effect for added realism. But, that dramatic leap onto centre stage set the pace for the next two decades.
[For a lowdown on the history of the digital actor, see our timeline]
Fast-forward nearly 20 years, and virtual Marilyn Monroe’s are rubbing pixelated shoulders with the likes of Gollum, Jar Jar Binks, Aki from Final Fantasy, Mighty Joe Young, and the entire crew of the Titanic. And they’re getting ever more real, reckons Scott Billups – one of Hollywood’s most accomplished visual-effects artists – who has worked on movies such as Jurassic Park, Hellraiser, Bend It Like Beckham, and City Of Ghosts.
“Gollum, Jar Jar Binks, and the like have opened up the category of virtual actors to the imagination,” he says. “I remember the first time that Jar Jar went in for a close-up – the chatter for months was about how banal his lines were and how annoying he was to watch. These are attributes we bestow on flesh-&-blood. I never head anyone say that the motion didn’t look realistic.”
There is massive appeal to work with virtual actors, says Billups: “I think the appeal of virtual actors is the same for us all – pixel wranglers and movie makers alike. It’s part of this democratization process of the motion picture manufacturing process... virtual sets us free from the bonds of budgetary limitations, and virtual actors open the door to the full realization of our imagination. The old school is scared senseless; and well they should be.”
And these budding synthespians are creating their own A Lists, and weaving their own Hollywood history, with its own stories of success and failure.
“To my knowledge, the first time a virtual actor was used in a high-profile way was in replacing Brandon Lee for the final scenes of The Crow in 1994,” says Torsten Reil, CEO of NaturalMotion, a creator of virtual actors.
“Since then, their use has become more common - in particular in crowd scenes such as those in Gladiator or Lord of the Rings. The most impressive recent example of a virtual actor is Gollum in Lord Of The Rings, which was animated with a combination of motion-capture and keyframing. I believe we will see a trend for virtual actors to move on from crowd scenes and take a more central role in certain shots.”
Stephen Rymill from the University of Cambridge Computer Lab says that CG has been used to create virtual characters in films for years – but that the appearance of “proper” virtual actors is a recent phenomenon – and that you can trace their development from path-locked stuntbots though to lead characters in today’s movies.
“The film Titanic used virtual actors to populate wide shots of the ship; these were very simple – the actors following paths to patrol a section of the ship,” Rymill says. “Jar Jar Binks, from Star Wars Episode 1, is an example of a CG character playing a large role in a film. The character Gollum demonstrates that a virtual actor can be just as much a leading actor as a real character in a film, rather than just a good bit of visual effects.”
Weta Digital CG supervisor Greg Butler – who worked on the creation of the Gollum character – worked on his first ‘digital human’ as part of the first Tippett Studio sequence for the movie Starship Troopers – and says that it seems like every project has been pushing the areas of technology related to the creation of digital human actors.
“Square’s Final Fantasy showed that is was possible to do an entire film with enormous amounts of motion capture, and solid hair and clothing simulations,” he says. “I remember the mainstream media picking up on all this activity and reporting on the imminent irrelevance of ‘real’ actors. Some famous actors have had 3D scans done – triggering stories about actors then lending out digital doubles for projects rather than learning parts themselves.”
Yet much of the hype kick-started by Final Fantasy has died down, according to Butler, but with the resurgence of movies like Spiderman, the debate around virtual actors certainly hasn’t.
“There were debates about whether studios should be allowed to use dead actors in their projects – Fred Astaire, John Wayne, and Humphrey Bogart have all shown up in modern commercials – but at some point all the media attention started to fade, and studios kept making movies the usual way,” he recalls. “Few studios have attempted a film like Final Fantasy in the years since, although human characters have begun to show up in projects such as Toy Story 2, Shrek, and Ice Age. These recent examples haven’t had photorealism as a goal, yet they’ve pushed the technology and audience acceptance in their own way.”
Yet what is a synthespian, exactly? It’s not as if you can ambush one on the red carpet, or chat to one as you request an autograph. What’s more, the industry itself has many different ideas as to what virtues a virtual has to have.
“If you don’t have to feed ’em, they’re virtual,” says Billups. And he should know: he recreated a virtual Marilyn Monroe that featured more polygons in her face than the entire bodies of the dinosaurs in the film Jurassic Park.
“An area of vactors (virtual actors) that often gets overlooked is that of celebrity recreation,” says Billups. “Once digitized, an actor can play any role, at any age, in perpetuity whether they’re available or not. They can even play multiple roles at the same time at any weight or physical condition. The problem hasn’t been the technology – although that’s improved greatly since I began working with synthetics in the late 80s – but rather the way the industry views synthetic representation.”
Chas Jarrett, CG supervisor at The Moving Picture Company (MPC) agrees that virtual actors can be lead-role material: “A virtual actor could be defined as any emoting digital character. It’s a grey area to say it only applies to humans as we see human actors wearing prosthetic masks all the time.
“They don’t look human but they’re still actors. So its form is irrelevant if it is driven by a sense of character and the internal thought process we associate with conscious beings. I think the definition becomes less valid when applied to inanimate objects brought to life through the miracle of 3D animation.”
Lead roles for digital actors certainly provide the lure for visual-effects artists, and are seen as the sexiest part of synthespian creation – and many cite Gollum as a prime example of a lead CG character.
“We’ve three main types of virtual actor at Weta Digital,” explains Eric Saindon, a CG supervisor with the company. “First, there are the main characters like Gollum and Shelob. We spend a lot of time in pre-production on these assets, in conjunction with Weta Workshop, our neighbouring physical-effects facility, and the visual-effects department, working out the characteristics for each ‘actor’.”
Yet there are two, other types of virtual actor, according to Saindon, that are having a bigger impact on the everyday production of movies: “The second type of actor is the digital double – in most cases, digital doubles are used where an actor could not do a stunt; Legolas’ dramatic leap onto a charging Mumakil in The Return Of The King is a good example,” he says. “The third type would be our crowd agents or massive agents. These agents are used when large crowds of individual actions are needed - for example, the battles at Pelennor Fields.”
MPC’s Jarrett agrees – and reckons that digital stunt doubles are worth more than CG leads. “Personally, I find the whole concept of virtual human actors a bit gimmicky,” he says. “It seems to originate from the long-standing CG tradition of pushing the technology towards photorealism, largely because it’s pretty obvious when you’ve accomplished it. But I still think ‘why bother?’ when I see photoreal CG humans. Just get a real actor.
“Digital stunt doubles are an entirely different concept based largely on circumventing the complex and expensive issues regarding shooting real stunts. Characters like Spiderman, or Neo in The Matrix, are really elaborate stunt doubles rather than virtual actors, and their freedom from physical constraints makes them attractive to some directors,” he adds.
Weta Digital’s Greg Butler says that most films that deploy digital effects use some form of digital actor – and are often digital doubles used for safety reasons – and cites examples where needing an actor to be pulverised by a car or fall of a building is better achieved through digital doubles.
“This has the added benefit that the shot can have many takes, rather than the ‘we’re only going to do this once’ mentality that is wise when a stunt person’s safety is an issue,” says Butler. “With the advancement of motion capture and artificial intelligence software, large groups of people can be done without having to deal with the logistics necessary for hundreds of extras.
This, too, has the benefit of avoiding the problem of extra number 203 staring at the camera and spoiling an expensive shot that was otherwise perfect.”
There are advantages to using virtual actors – and not just as crowd extras. You don’t have to pay them as much, and their trailer requirements are minimal.
“One advantage,” says Weta Digital’s Saindon, “is that we can do anything we want with them and not get into trouble. The battle scenes in Return Of The King required some pretty crazy stunts.
It would have taken weeks of setup and working out complicated camera angles to produce something that might have functioned on set. When brought into post production, such shots still take weeks to complete, but we have the freedom to place the camera wherever the director wants, and get away with much more extreme action.”
Assistant professor Ronald Fedkiw from the Stanford Computer Graphics Lab in the US agrees.
“The advantage is that you can put them into environments that are expensive or impossible to place their human counterparts – the moon, Hell – and you can treat them any way you want, kill them, resurrect them. The main disadvantage is that we still cannot deliver completely convincing performances. I’m impressed with the rendering and geometry, but the motion is not there yet – it’s especially difficult to create realistic facial animation and simulation,” he says.
The main challenge, Fedkiw reckons, is motion – and it isn’t just down to lousy animation skills, either. Fedkiw says that it’s hard enough for flesh-&-blood humans to act, much less a virtual human.
“How many times have you seen a real human delivering a crappy performance?” he asks. “Most of the rest of the research seems to be progressing along at a reasonable pace – although it’s not yet completely done. Motion, acting, and performance will be the bottleneck, especially facial animation.”
Butler reckons that the technical challenges are rapidly being solved – with skin shading, hair and cloth simulation, and even facial animation posting some impressive gains over the past decade. The upshot, he says, is that the virtual actor is getting closer to the audience – moving from background crowd to lead character – and they are staying in the frame for longer. Yet, the challenge for a true, virtual human is still massive.
Audiences are simply unforgiving, says Butler. If you were to stumble across CG technology that could provide, absolutely, a convincingly animated human face – complete with realistic eyes, lips, forehead wrinkles, and so on, all it would take is a split-second of unrealistic animation for the illusion to be blown apart.
“Humans are very good at detecting natural human motion, because we see it all around us all the time,” says Stephen Rymill. “If the movement is even slightly out, it will make it obvious that the actor is not real. And, given how hard it is to produce keyframe animation that looks realistic, the most popular technique is to use motion capture.”
But what about Gollum – the most ‘realistic’ of virtual actors to date?
“Gollum was successful because he was not replacing a human performance,” says Butler.
“Gollum was a fantasy character – human enough to create empathy and convey complex and powerful emotions, but not expected to achieve that 100 per cent believability test of replacing a human performer. So, in this respect, digital actors are here now in featured roles. We were lucky in that Tolkein describes Gollum as only having a vestige of clothing left on him – we often wish he had also written that Gollum was bald.”
One thing that may be crucial to virtual actors – and a complete contrast to any hard-drinking real ones – is subtlety.
Gollum excelled at this, according to Saindon, delivering the kind of details that people don’t think of, but would be missed if not included. It’s not like the cinema-going audience is munching popcorn with one hand, and holding a clipboard in the other ticking off a checklist. But the details do count, says Saindon.
“If you take a close look at Gollum, you will notice little things like wind in his hair, and translucency in thin surfaces like his ears or nose,” says Saindon. “And, if you look really closely, you can even see pores in his skin and fingerprints. These are very minor, but once they are all done well, it helps to bring him one step closer to real.”
Gollum was certainly a milestone – albeit one that won’t be doing the chat-show circuit, or making the front cover of The National Enquirer. But what of the future? Technology for creating animated characters is not only getting more powerful, it’s beginning to filter down to the desktop-level. We could all be directors of a cast of digital actors before the decade is up, some experts reckon.
“Most audiences completely bought into Gollum, who was a genuine milestone for digital characters,” says MPC’s Chas Jarrett. “But, I suspect that in a few years time we’ll look back and think ‘did I really think he looked real?’. Technology moves at quite a pace in this field and we almost don’t notice how much until we look back at past milestones and see their limitations.”
For visual-effects facilities, the emergence of the digital actor isn’t the preserve of high-cost productions, as Scott Billups has discovered. In addition to being owning one of the oldest digital production houses in the business, it’s also the smallest – and yet the demand for virtual actors is huge. It doesn’t need incredibly high-end kit, either, with Billups touting Curious Labs Poser as a weapon of choice.
“Last year I created several thousand virtual characters for six different TV shows, a nice little indie flick called Bend It Like Beckham, as well as several hundred CG bats for Matt Dillon’s directorial debut, City of Ghosts,” reveals Billups.
“I’m currently creating thousands of virtual soldiers for a History Channel mini-series on Alexandra the Great. The really close shots are being done in Alias Maya, and the legions are done in Poser and rendered in NewTek LightWave. There’s a nice mix of applications out there, but I don’t think that there’s really bigger bang-for-the-buck in virtual character creation than Poser/ LightWave. So, the future is not only here, its been here for a while.”
One thing that most people in the industry agree on is that real actors have little to fear from their digital counterparts – except that the latter look increasingly better over the years. That may be a clear advantage of binary over biological, but it isn’t enough. Yet.
“Despite all of this progress, I don’t think actors have too much to fear from the visual-effects industry for quite some while,” says Butler.
“Although we can now create photoreal images of digital people, there is much more to creating a film-quality performance that audiences will connect with and come back for more of. Acting can be a deceptively complex process,” he adds.
Rymill agrees: “A lot of people have predicted that virtual actors will revolutionize the film industry, and that actors will become redundant. I don’t see this happening. Evidence has shown that the best virtual actors at the moment are those where there has been a large input from a real actor, and this will probably continue.
Above all, though, there does seem little point in using a virtual actor if a person can equally play the role: unless the actor’s salary is very high, it will still be less than the huge resources needed to produce a realistic virtual actor.”
However, Rymill doesn’t rule out the possibility that, one day, a digital actor could scoop an Oscar.
That said, the synthespian has already had a huge impact on both the film-going public, and the people crafting the effects for tomorrow’s blockbusters. And it’s a trend that will see the continued rise of the virtual actor.
“The audience loves them,” says Fedkiw, “just take a look at the box office.”
“The audience doesn’t care if the actor is digital or carbon,” adds Billups. “All they want to do is to empathize and travel beyond their daily grind. If the character can help transport them, then they’ve got a fan club. If not, they’re only so much pixel dust.”
The making of Gollum
The famed Gollum character, which saw its debut in the 2002 movie The Lord Of The Rings: The Two Towers, is rightly hailed as the best example of a virtual actor in a major supporting role – there were even online petitions to sway the Academy Awards judges into bestowing the character an Oscar.
Fundamentally, Weta Digital – the creators of the CG Gollum – had three, back-to-back movies during which to develop the character.
“We got to the point with Gollum where we didn’t have anything left on our list to fix,” says Weta Digital CG supervisor Eric Saindon. “It’s very seldom that you ever get the chance to get a CG character to this level.”
It also meant that as technology evolved during the production of the movies, it could be deployed to boost the realism. During the first Lord Of The Rings movie, for example, the team solved a lot of hair on the characters as one large piece of cloth. It looked OK, but didn’t move like hair. Fast forward to the third movie, and computers had increased in speed by around 300 per cent, and allowed the team to solve individual hairs on Gollum’s head.
And, regardless of the technological booster rockets fitted to CG work in the movies, Weta Digital CG supervisor Greg Butler pays homage to a more biological impact: “Having a talented actor like Andy Serkis involved with Gollum certainly created a strong anchor for the consistency of the performance and emotion of the character,” he says.
Miralab in Motion
Nadia Magnenant-Thalmann is one of the world’s leading experts of CG animation and virtual humans - she founded the first MIRALab as a virtual human research lab back in 1980. Its objective was simple: to simulate virtual humans.
“Our first actors were Marilyn Monroe and Humphrey Bogart, which we created in 1987, and since then the field has grown,” says Magnenant-Thalmann. “We were the first in Canada to simulate a real actor, and then I founded MIRALab in Geneva to simulate clothes and hair for virtual actors.”
And she is attempting to take virtual actors beyond dumb animation, and into the world of self-awareness.
“I guess the main technical challenge for virtual actors is autonomy,” she says. “We need to simulate them as if they were aware of who they are, and how they should react. We need more physics and behavioural laws that should direct the behaviour of the virtual actor – but this is still difficult, as we humans are very complex.”
You can find out more at www.miralab.ch
Stephen Rymill from the University of Cambridge Computer Lab likes crowds. For him, crowds represent a good use of virtual actors in films – and he’s conducting advanced research into making virtual crowds behave better.
“Crowds are good, as they don’t have to be as detailed as the main actor,” says Rymill. However, while graphically the quality can be lower, their movements still need to be realistic. The advantages of virtual crowds are obvious: you don’t need to employ thousands of extras, they don’t need to be fed, and they won’t complain about the weather.”
Rymill’s research uses psychology, in that each actor in the crowd is given a set of rules that govern its behaviour. Traditionally, animators then tweak the rules until the animation ‘looks right’. Rymill is creating rules, such as modelling how a crowd member’s eyes move around a scene detecting other people and objects. “This approach should add more realism,” he says, “so that, for example, an actor would not notice an approaching car if he was looking over his shoulder at the time.”
You can see more of Rymill’s work on his Web site