From the beginning, computer animation has had the ability to engage viewers by giving artists a way to mix reality with fantasy. Early film examples such as the stained-glass knight in Barry Levinson's 1985 Young Sherlock Holmes and the 'water tentacle' in James Cameron's 1989 The Abyss offered a glimpse of the potential of this new art form to create memorable characters. That became immediately apparent in Cameron's later Terminator 2 (1991). He took an already great idea for a character (a terminator robot) and turned it in a new and unexpected direction: the shape-shifting T-1000 liquid-metal terminator.

The computer-generated Gollum (above, seen in 2012's The Hobbit: An Unexpected Journey) is based on performances by actor Andy Serkis (below). Credit: NEW LINE PRODUCTIONS
Credit: NEW LINE PRODUCTIONS

Computer animation is a natural extension of hand-drawn methods developed during the early-twentieth-century golden age of 'cel' animation ushered in by Walt Disney, in which a series of images is played back at speed to give the illusion of life. Just as we do today, animators used a variety of reference techniques to capture the essence of organic movement. Snow White's dance with the dwarves in Disney's 1937 film was created by matching the movements of a live, filmed dancer, using a technique called rotoscoping — in the most basic terms, tracing the motion from a film one frame at a time. This technique, although now slightly more sophisticated in its application, is still in use.

Take dinosaurs, for example. I was fortunate to begin my career at the US visual-effects company Industrial Light & Magic, then in Marin County, California, as it was gearing up to create the computer-generated dinosaurs for Steven Spielberg's 1993 Jurassic Park. Naturally, we all thought that rotoscoping dinosaurs would be a great idea, but unfortunately that was out of the question. Instead, we studied elephants to understand weight, and lizards, other reptiles and birds to get some ideas about how dinosaurs of different sizes might have moved. Digital animators did motion studies, copying the movement of these animals frame by frame until they could synthesize a convincing idea of dinosaur movement.

Two years later, computer animation took another big step forward with the astounding success of Pixar's Toy Story. Software was becoming sophisticated enough to tackle the creation of a character's performance. In traditional animation, a lead animator sets key frames or poses for a character and junior animators draw the 'in-betweens' from pose to pose. Now, artists could use the computer to do that. Pixar proved that three-dimensional computer animation could be used to create an entire film.

In 2001, I joined Wellington-based visual-effects company Weta Digital to work with Peter Jackson on The Lord of the Rings trilogy, inspired in large part by the chance to create the character Gollum. Gollum was a special challenge, because the more realistically human a character is, the more complex the animation gets. People are attuned to recognizing all aspects of human motion and behaviour, no matter how subtle. And because the characters we create are three-dimensional, we have to understand how to pose them frame by frame to achieve realistic performances.

The Na'vi in Avatar (2009) lived in an entirely digitally created world. Credit: TWENTIETH CENTURY FOX/THE KOBAL COLLECTION

With Gollum, we used the then relatively new technique of performance capture — effectively, an extension of rotoscoping. But instead of looking at an actor from a single point of view and matching the form of the motion, we looked at his performance using dozens of cameras simultaneously and matched the motion's underlying dynamics.

Andy Serkis, who performed Gollum, wore a special suit with reflective markers to show the key positions of his joints. From the multiple cameras, we could calculate his skeleton's position at every frame as he performed. Those positions were then transferred to Gollum's digital skeleton, which allowed us to make the character move the way Andy did. Traditional key-frame animation techniques still apply, however. For example, the first time we see Gollum in The Lord of the Rings: The Two Towers (2002), he is climbing down a vertical rock face, something no human can do. So that motion is based on animators observing what a human can do and using their imaginations to create a believable performance. And, in a direct throwback to rotoscoping, we created his facial performance and dialogue-related movements by hand, frame by frame, from Andy's filmed performance.

The problem with trying to capture facial motion is that there are no joints, apart from the jaw, that have movements you can track. So for Jackson's 2005 King Kong, Weta came up with a different technique. Again with Serkis, we glued small reflective markers all over his face. By using these to track the changes in skin position and tension as Andy performed, we could compute what his muscles were doing underneath. Then we built Kong so that he had the same facial-muscle layout as Andy, and used Andy's muscle movements to drive Kong's facial performance.

This breakthrough meant that we could now capture an actor's performance in its entirety. This became important for Weta's next film, James Cameron's Avatar (2009), for which we made one important modification. Each actor wore a helmet that filmed their facial movements; we then extracted the performance data from each frame and used a 'facial action coding system' solver to translate the movements into muscle activations. The knowledge of which muscles are activated in a facial expression informed our activation of the corresponding muscles in the digital characters. In addition, this process allowed the director to see the actors live through a virtual camera as they were instantly transformed into their Na'vi characters moving through the world of Pandora.

Digital characters also have to appear realistic in their surroundings, whether that is a photographed environment or a complete digital creation such as the jungles of Pandora. So we looked to understand how light and materials interact in nature. One of the best examples of this interaction is subsurface scattering. We first developed a technique to replicate this mechanism of light transport to create the translucency of Gollum's skin, leveraging pioneering research by computer-graphics specialist Henrik Wann Jensen and his colleagues at Stanford University in California (see go.nature.com/lyzuh2). The thick skin of a dinosaur can be simulated by bouncing light off the exterior. But human skin is softer and more translucent, so light enters and bounces around dozens of times before exiting. These properties, which are easily observed by putting your hand in front of a bright light, are crucial to a realistic portrayal.

Realistic animation depends on knowledge of how skin, muscles and hair move independently of a character's performance.

Realistic animation also depends on knowledge of how skin, muscles and hair move independently of a character's performance. These secondary motions are achieved through intensive simulations that compute all of the mass, dynamics, tensions and interaction of each part of the body as a character moves. The simulations help to create the complex visual cues that the human brain processes when taking in an image. They also ensure that the physiology of creatures (real or fantastical) has a ground truth and is believable. Combining this new level of detail with motion-captured performances of talented actors has enabled computer-animated characters to become lead actors.

Rupert Wyatt's 2011 Rise of the Planet of the Apes featured a chimpanzee named Caesar who was not just the protagonist; he was the emotional centre of the film. This entirely digital character is a great example of how the advances in animation work together, from muscle simulation, fur and realistic lighting, to motion-captured body and facial performance.

The following year saw all of these developments come full circle when we were able to once again present Gollum in Peter Jackson's The Hobbit: An Unexpected Journey. This new Gollum benefited from a much more detailed digital model, new subsurface scattering techniques and all of the advances we have made in the past ten years. If you watch closely, you will see the muscles moving under his skin and the light refracting in his eyes. And you will get a glimpse of the worlds we can create from the mix of all this art and science.