Principles of VFX
Parasite (2019)
Interstellar (2014)
Everything Everywhere All At Once (2022)
Avatar (2009)
2001: A Space Odyssey (1968)
Miniatures and matte paintings were used to create the space shots in 2001: A Space Odyssey.
Director: Stanley Kubrik
The Abyss (1989)
One of ILMs first computer generated VFX shots. ILM had to create new software specifically for this shot.
Director: James Cameron
Jason and the Argonauts (1963)
Stop motion skeletons were placed in front of the clean footage of the actors and then animated to match the footage.
Director: Don Chaffey
Pirates of the Caribbean (2003)
Each scene featuring the skeleton pirates was shot twice, once with the actors for reference, and another without. The actors then recorded the scene again in a mocap studio.
Director: Gore Verbinski
Luxo Jr. (1986)
Luxo Jr. was Pixars first film and one of the very first fully computer generated short films.
Director: John Lasseter
Tron (1982)
Tron was one of the first films to make use of computer generated imagery, and is cited by the head of Pixar, John Lasseter, as the movie that showed him the potential of CGI, stating "without Tron, there would be no Toy Story".
Director: Steven Lisberger
A Trip to the Moon (1902)
Georges Méliès is widely thought of as the creator of visual effects, he pioneered compositing multiple shots together.
Director: Georges Méliès
King Kong (1933)
King Kong is known for its groundbreaking use of special effects, mainly matte paintings and rear projection to superimpose their actors into different scenes.
Director: Merian C. Cooper, Ernest B. Schoedsack
The Lord of the Rings (2001)
Andy Serkis helped revolutionize motion capture with his performance as Gollum.
Director: Peter Jackson
An American Werewolf in London (1981)
An American Werewolf in London was praised for its amazing makeup special effects.
Director: John Landis
Prince, S. (2010) 'Through the Looking Glass: Philosophical Toys and Digital Visual Effects'. Projections, Vol 4 (Issue 2), pp. 19-40.
In this document Prince lays out how the worlds of art and science have been an interconnected part of cinema all the way from its invention to today’s digital effects. He says Visual Effects are an illusion meant to fool the viewer into believing what they are seeing is reality, even if they know its not possible. Prince labels this effect as “Perceptual Realism”, the idea that the context clues of traditional cinema can be replicated and used within VFX to allow the viewer to suspend their disbelief in a way that traditional special effects cannot.
Prince then discusses the evolution of how scientific studies into human vision eventually led to the rise of cinema. He describes devices made for the study of vision as “toys” with purpose in both scientific study and entertainment. Many believe that science and art took separate directions in the early 20th century, but Prince argues that they have always remained connected, which is evident with today’s digital effects.
A Pseudoscope, a device that reverses depth perception when looked through.
Invisible vs Spectacular
Invisible Effects
_gif.gif)
FORD VS FERRARI
1917
KNIVES OUT
Spectacular Effects
AVENGERS: INFINITY WAR
AVATAR
PLANET OF THE APES
Prince, S. (2012) Digital Visual Effects in Cinema : The Seduction of Reality. London: Rutgers University Press.
In this book introduction Prince challenges the notion that visual effects are a just a “spectacle”. He first relates this back to early cinema, with Lumiere’s actualities versus Melies's magical “special effects”. Prince argues that these two halves of cinema are profoundly connected, and that modern visual effects are simply more tools the filmmaker can use to convince the viewer what they are seeing is reality.
Prince counters the idea that visual effects is the death of the actor as actors are still used for motion capture performances, and in the case of hand animated films, the animator themselves is the actor. Prince then goes on to describe how visual effects helps filmmakers raise the bar of realism when applying special effects, and allows for much deeper storytelling without the need to cut-away due to fear of the viewer “recognising” that what they are seeing is not real.
Gunning, T. (2006) 'The Cinema of Attractions: Early Film, Its Spectator and the Avant-Garde', in Strauven, W. (1.) The Cinema of Attractions Reloaded. Amsterdam: Amsterdam University Press, pp. 381-388.
In this essay Gunning outlines the pre-1906 “Cinema of Attractions” that existed before narrative film overtook as the predominant genre. Early cinema was dominated by the debate of whether it should follow in the footsteps of the existing theatre, or in a new avant-garde direction with effects and spectacle never seen before by the audience.
Gunning also speaks of how involved the audience is in these types of cinema, with narrative cinema keeping the audience more separated in order to allow them to become immersed in the fiction of the film. Whereas in the cinema of spectacle filmmakers like Méliès would often look straight at the audience (seen as immersion-breaking in narrative cinema), in order to invite them to pay attention to the upcoming spectacle. Early cinema was primarily driven by this spectacle due to the rapid evolution of filmmaking techniques; each new technique would be another thing for the audience to discover.
Ray Harryhausen
Ray Harryhausen is a legendary animator who inspired generations of directors and animators with his combination of live action footage and stop motion (dubbed "Dynamation").
His style of animation and creature design can still be seen today in movies like Spielberg's "Jurassic Park" or Peter Jackson's "King Kong" especially in the way the creatures move.
Harryhausen combined the footage by rear projecting the live-action footage onto a screen behind the stop motion models.
The animation table was then masked out with a matte screen in front, which would then be replaced with the live action footage using a reverse of the mask.
Manovich, L. (1995). What Is Digital Cinema? Cinema, the Art of the Index.
"We usually think of computerization as automation, but here the result is the reverse: what was previously automatically recorded by a camera now has to be painted one frame at a time."
Manovich answers the question “What is Digital Cinema” by first laying out his 5 principles of digital filmmaking. These principles lay out the idea that digital cinema shifts live action footage from being close to the final film, to being just another raw source of data alongside computer generated imagery. He also suggests that digital filmmaking has brought the acts of editing and special effects much closer together, previously an editor would simply set the order of already completed images, now their job can include the manipulation of said images.
Manovich then says that “Digital cinema is a particular case of animation which uses live action footage as one of its many elements.”. He then relates this back to early cinema, and how it was born from animation using the Phenakistiscope and the zootrope. Essentially meaning cinema has come full circle, from the imagination of artists, to live action footage in the 20th century, and then finally back to the artists with digital cinema.
Lev Manovich is an author of many books about digital culture and new media. His work is widely cited in the digital media field. He is currently a Professor at City University of New York.
CGI Remakes
Yoda in The Phantom Menace (1999) vs the 2011 rerelease
Star Wars 1977 vs the 1997 and 2004 re-release
Pixar Founders
Alvy Ray Smith is a computer scientist who co-founded Pixar alongside Ed Catmull.
He is responsible for many inventions in the computer graphics space, including the alpha channel, HSV color space and the first RGB paint program "Paint3".
Smith created and directed the "Genisis Demo" sequence from Star Trek II: The Wrath of Khan, which is one of the first fully 3D sequences in film.
Smith left Pixar in 1991, before they had released any films, after an alleged falling out with main financer and CEO Steve Jobs.
He is now widely called the most overlooked founders of Pixar, even though he was responsible for most of the technological breakthroughs.
Ed Catmull is a computer scientist who co-founded Pixar alongside Alvy Ray Smith.
Catmull discovered and invented texture mapping, bicubic patches, Z-buffering and much more while at university.
Catmull also created a one-minute 3D animation of his hand "A Computer Animated Hand", which was picked up and included in Futureworld (1976).
Smith, A.R. (1998). George Lucas discovers Computer Graphics.
In this article Alvy Ray Smith talks of the early years at Lucasfilms computer division, and how George Lucas (rather unknowingly) allowed him to build a team out of the best computer scientists in the country. In the article he describes people like Lucas and Steve Jobs as “accidental visionaries”, as they did not fully understand the potential of the space they were investing in and did so only on intuition.
Smith then talks about the process of creating a 3D sequence for the film Star Trek II: The Wrath of Khan. This essentially acted as a commercial for George Lucas to invest further into the computer department and use them on films like Return of the Jedi. They designed the sequence specifically to "Knock George's socks off", they did this through creating special camera moves with a 6D spline, designed to look like a spacecraft flying past the surface of a planet.
Uncanny Valley
Lay, S. (2015). Uncanny valley: Why we find human-like robots and dolls so creepy. [online] The Guardian
In this article Lay outlines Masahiro Mori’s “Uncanny Valley” theory, which theorises that the more human-like a robot, the more acceptable people find them. But just before they reach full human-likeness there is a huge valley where people develop a sense of unease and discomfort. While Mori originally made this theory for robots, it also applies to video games and CGI.
Lay goes on to outline the increase in research in the field in recent years. Some research theorises that the Uncanny Valley doesn’t exist, and some clearly show a relationship between human likeness and a sense of unease when approaching full likeness. Lay mentions that a lot of this research has found that a mismatch between aspects of a robot’s appearance and behaviour, like speech speed and facial expressions can have a huge impact on whether it falls into the uncanny valley.
Rules of Animation
Disney sets out 12 rules for animating their characters in 2D animated movies, but these rules still apply today in 3D animation.
Their animation has a flowing style with characters squashing and stretching and deforming in exaugurated ways in order to give them larger-than-life characterisation.
In recent years Disney have remade lots of their classic movies in 3D, which gives us a direct comparison between the 2 styles of animation, although many feel that Disney have abandoned these rules in favour of photorealism.
Squash and Stretch
Anticipation
Pose-to-Pose and Straight Ahead
Staging
Follow Through
Slow-In Slow-Out
Arc
Secondary Action
Timing
Exaggeration
Solid Drawing
Appeal
McClean, S.T. (2008). Digital storytelling : the narrative power of visual effects in film p48-51. Cambridge, Mass. ; London: Mit.
In this chapter McClean analyses the relationship between the virtual camera and storytelling and how this relates to existing rules between the camera and narration. Previously a camera might follow a cable from a phone and then reverse this shot on the other side of the call, but with VFX the camera may jump inside the phone and then follow the call down the line. This can be used to emphasise the importance of the phone call in a more effective way than traditional camera moves.
McLean also outlines James Monaco’s observations of the relationship between camera and narration and the extra abilities a virtual camera gives to you, for example you can place the camera in “impossible” positions, like in a fridge or even a “POV” shot of a character in the mirror, allowing you to see the narrative from the perspective of the character. He notes that directors need to consider the extra abilities given to them by VFX while also following the same rules as traditional filmmaking.
Tinwell, A. (2015). The Uncanny Valley in games and animation. Boca Raton, Fl: Crc.
In this text, Tinwell explores the concept of the self and its connection to the psychological phenomenon known as the uncanny. The uncanny is the feeling of unease or discomfort that arises when one encounters something that is human-like but not quite human. Tinwell discusses the work of William James, who introduced the concept of the self in the late 19th century, and argues that the self is made up of both conscious and subconscious processes that help us establish our sense of identity.
Tinwell also examines the role of the ego in building self-esteem and pride, and suggests that people often seek out objects and people that remind them of themselves in order to boost their ego. Tinwell also discusses research on how children and animals respond to positive and negative interactions, and the long-lasting mental effects of these interactions. This research, Tinwell argues, may help explain the phenomenon of the Uncanny Valley, where virtual characters that are almost, but not quite, human can result in this mental anguish that is innate to us.
Breakdown Videos
This video quickly breaks down some of the methods the VFX artists used to create the effects for Everything Everywhere All at Once on a small budget
This video breaks down how the animators on Spider-Verse created the unique stylized look of the movie.
I liked how they used various animation frame-rates to tell a story, with Miles swinging at a lower frame-rate than Peter when learning to web swing.
This is a longer video (25 min) of the VFX supervisor talking about the process of making the visual effects for No Way Home, from on set to compositing.
Composite Breakdowns
This composite is being used for a set extension, where the rooftop was filmed on a bluescreen and the distance was shot separately
This composite was used to add rubble to the rooftop, these were added via 3D models being overlaid on the live action footage.
Here a 3D model is being composited into the scene to add buildings
Assignment 2 - Question & Reading List
What is the Uncanny Valley? Citing specific film examples, discuss its relationship to special effects, computer generated imagery.
For Assingment 2 I have chosen the question on the Uncanney Valley as that is what interested me the most.
Below is a list of texts I will read as research for the assignment:
Tinwell, A. (2015). The uncanny valley in games & animation. Boca Raton, Florida: CRC Press.
Mori, M., MacDorman, K. and Kageki, N. (2012). The Uncanny Valley [From the Field]. IEEE Robotics & Automation Magazine, [online] 19(2), pp.98–100. doi:10.1109/mra.2012.2192811.
Lay, S. (2015). Uncanny valley: Why we find human-like robots and dolls so creepy. [online] The Guardian
Mori, M., MacDorman, K. and Kageki, N. (2012). The Uncanny Valley [From the Field]. IEEE Robotics & Automation Magazine, [online] 19(2), pp.98–100. doi:10.1109/mra.2012.2192811.
In this article Masahiro Mori coins the term "The Uncanny Valley". The uncanny valley is a term to describe the phenomenon where a person's response to a humanlike robot abruptly shifts from empathy to revulsion as it approaches, but fails to attain, a lifelike appearance. The idea is that as a robot becomes more human-like, our sense of affinity for it increases, but as it reaches a point of almost but not quite being human, it causes a sense of discomfort.
The article also notes that the uncanny valley applies to prosthetic limbs, and that as technology improves, it becomes harder to distinguish between a prosthetic and a real limb. The piece highlights that while the uncanny valley is a concern, robot and AI researchers are continuing to work towards creating more human-like robots.
Assignment 2 - PowerPoint
What is the Uncanny Valley? Citing specific film examples, discuss its relationship to special effects, computer generated imagery.
Have you ever seen a lifelike robot that gives you an eerie feeling when you see it? Or maybe a CGI character from a movie or video game? This phenomenon is known as The Uncanny Valley, theorised by Robotics Professor Masahiro Mori in 1970, and it has gained new relevance with the rapid development of CGI in both video games and movies. Mori theorised that as a robot got closer to 100% human-likeness, a person’s affinity towards the robot would take a sharp dive and it would become unsettling or disturbing. The theory then suggests that once the robot reaches human likeness this affinity returns, and the robot is no longer considered unsettling. He named this “The Uncanny Valley” (Mori et al., 2012). In this essay I will be exploring the uncanny valley as it relates to film, audio, and motion, and will investigate whether we have now crossed the valley, or just avoided it.
To Understand the Uncanny Valley first we need to understand what makes an image uncanny, more particularly why it looks uncanny while in motion, as this is extra important when it comes to film and visual effects. One thing I find interesting is how little it can take to send an image into the uncanny, for example in the 2017 film Justice League, they had to use visual effects to remove Henry Cavill’s moustache, resulting in an uncanny image despite very little of the overall image changing. “The effect being attached to the otherwise real Cavill is the key. We know what Superman looks like, so the contrast of human and inhuman is obviously more pronounced; whenever he looks different it'll stand out” says movie reviewer Alex Leadbeater (2017). Evidence has shown that motion can increase the effect of the uncanny valley, in fact Mori observed this in his original 1970 study. Mori used the example of a prosthetic hand, which sense of eeriness only increases with movement (Mori et al., 2012). In Angela Tinwell’s book The Uncanny Valley in Games and Animation she sets out to understand the relation of uncanniness and motion in the Uncanny Modality study. The results from this study “showed that characters were perceived to be more uncanny when there was a lack of human-likeness in the character’s facial expression, particularly in the upper facial region, with the forehead and brows being of particular significance”. She placed a lot of significance on NVC (Non-Verbal Communication) as it has a large effect on how we interpret the state and possible actions of others (Tinwell, 2015; Darwin, 1872). This can be connected back to the example of Cavill’s moustache, the removal of movement from his upper lip prohibits his NVC, leading to an uncanny feeling performance.
Another relationship to think about when it comes to the uncanny is the relationship of audio and motion. A performance may not be uncanny on its own but mismatching audio can set off alarm bells, or in the case of Zemeckis’s Beowulf, a semi-photoreal Angelina Jolie with the voice of Jolie herself ends up being a hard sell for the audience (Plantec, 2007). Tinwell (2015) also found this in her Uncanny Modality study, she found that over exaggeration of mouth movements during speech increased the uncanny, and that characters with perfect lip-synchronisation were regarded as less uncanny than those with asynchronous lip movements. When creating virtual characters for movies great care needs to be put in to make sure the character works as a cohesive. The audio, the visuals, and the animation must work together to sell us on a character, any mismatch in “human likeness” between these parts can amplify the uncanny valley. Jolie voicing a photoreal version of herself in Beowulf heightens our senses as our brains try to pick apart the illusion in front of us, whilst if she were to play a less-photoreal cartoon version of herself this may bypass the uncanny valley as our brain no longer feels “fooled” by what’s in front of us (Plantec, 2007).
The final thing to think about when it comes what makes something uncanny is audience expectation. A lot of the uncanny can be attributed to what an audience expects from a character. For example, an audience won’t expect human-like movement or features in a character like Sonic the Hedgehog, so when they were presented with exactly that in the initial trailer for the 2020 film there were cries of “uncanny valley!” from the audience who thought the character was eerie and unnerving mostly due to his very human-like teeth (Greenspan, 2019). So much so that the studio had to go back and recreate him in a less “human-like” fashion, which ultimately brought him back out of the uncanny valley to great effect (Richards, 2019), with the main visual differences being the size of his eyes, and the visibility of his teeth. Another place where audience expectation can increase the effect of the uncanny is in the recent surge of de-aged or even deceased actors in movies. For example, both Carrie Fisher and Peter Cushing appear in the film Rogue One despite the latter being dead for over 20 years. Whilst the Visual Effects were outstanding the audience are immediately scrutinizing and searching for clues of fakery as they know that what they are seeing cannot be real, whereas if this digital character had been of another more unknown actor, people may not have noticed at all. When a character isn’t attempting to fool us, it becomes much easier for our brains to accept, but when our brains even suspect that it’s trying to be tricked it becomes impossible for us to ignore (Plantec, 2007).
The final question I want to ask in this essay is, have we crossed the valley? And if not, what will it take to do so? Now this is a hard question to answer as opinions change greatly from person to person, some would argue that we are getting close to crossing it, while others would argue that we are avoiding it all together. A movie like 2009’s Avatar could be said to be “crossing the valley” as its human-like Na’vi aliens do not illicit a feeling of uncanniness, but you could also argue that these characters are simply avoiding the valley by using design choices like larger eyes and wider noses. Cameron himself said in interview (Boucher, 2009) that:
“Our goal right from the get-go is that we had to get over the uncanny valley. These characters have to be real, they have to be alive. And what the actors do has to come through 100%. We didn’t want to get in and come back and muck around with a lot of key-framing where we would be animating over what the actors did.”
This quote implies that he thought the movie had successfully crossed the uncanny valley, but many believed that he had simply avoided it all together with the Na’vis character design, in something some have dubbed as the “intentional artificialization” of reality (Simpson, 2010). One of the biggest barriers to realistic looking characters in motion is the methods of capturing an actor’s performance, Dr. Parag Havaldar, lead software engineer on Beowulf found that during production of the film they would often have to go in a tweak the animation of the characters faces manually using keyframes and that even after those tweaks much of the actors performance was lost, which becomes evident when watching those performances side-by-side with the final product (Plantec, 2007). But this is something that has come on leaps and bounds over the past decade or so, especially as it was needed for the ever-growing video games industry, who needed a way to capture actor’s performances for use in game, with games like The Last of Us (Naughty Dog, 2013) being praised for its use of mocap.
In conclusion, the uncanny valley in film is a theory that, as a character becomes more human-like the more it risks feeling uncanny to the viewer, and filmmakers need to be careful when approaching that valley. The uncanny valley is still a large issue in filmmaking today, but with research like Tinwell’s 2015 Uncanny Modality Study helping us better understand the relationship between audio, motion and the uncanny, we can now better navigate the valley and avoid its pitfalls. The rapid development of facial capture technology is also helping reduce these valleys by greater capturing the actor’s facial performance, leading to a less uncanny feeling when translated onto a 3D model. Finally, we need to keep in mind the audience’s expectations when coming into a movie, if an audience suspects they are being tricked they are going to view the work sceptically, greatly increasing the chance of feeling the uncanny valley. Further research is needed into what it will take to fully cross the valley, as most films that have claimed to cross it are simply just avoiding it all together by making their characters less human instead of facing the problem directly by trying to improve technology and pass it.
Bibliography
Mori, M., MacDorman, K. and Kageki, N. (2012) ‘The Uncanny Balley [From the Field]’, IEEE Robotics & Automation Magazine, 19(2), pp. 98–100. Available at: https://doi.org/10.1109/MRA.2012.2192811.
Justice League (2017) Directed by Z. Snyder. [Feature film]. Burbank, CA: Warner Bros. Pictures
Leadbeater, A. (2017) ‘Why was superman’s cgi-erased mustache so bad in justice league?’, ScreenRant, 18 November. Available at: https://screenrant.com/justice-league-movie-superman-henry-cavill-mustache-reshoots/ (Accessed: 2 February 2023).
Tinwell, A. (2015) The uncanny valley in games & animation. Boca Raton, Florida: CRC Press.
Darwin, C. (1872) The expression of the emotions in man and animals. London: John Murray. Available at: https://doi.org/10.1037/10001-000.
Plantec, P. (2007) Crossing the great uncanny valley, Animation World Network. Available at: https://www.awn.com/vfxworld/crossing-great-uncanny-valley (Accessed: 31 January 2023).
Beowulf (2007) Directed by R. Zemeckis. [Feature film]. Burbank, CA: Warner Bros. Pictures.
Sonic the Hedgehog (2020) Directed by J. Fowler. [Feature film]. Los Angeles, CA: Paramount Pictures.
Rogue One (2016) Directed by G. Edwards. [Feature film]. Burbank, CA: Walt Disney Studios Motion Pictures.
Greenspan, R. (2019) ‘No one can look away from sonic the hedgehog’s teeth in this trailer’, TIME, 20 April. Available at: https://time.com/5580304/sonic-the-hedgehog-movie-teeth/ (Accessed: 1 February 2023).
Richards, W. (2019) ‘Second time lucky - redesign of controversial live-action Sonic The Hedgehog revealed in new trailer’, NME, 12 November. Available at: https://www.nme.com/news/second-time-lucky-sonic-the-hedgehog-shares-redesign-in-new-trailer-2574435 (Accessed: 1 February 2023).
Simpson, J. (2010) ‘Avatar and the Uncanny Valley’, Running After My Hat, 5 January. Available at: https://johnesimpson.com/blog/2010/01/avatar-and-the-uncanny-valley/ (Accessed: 1 February 2023).
Boucher, G. (2009) ‘James Cameron: Yes, ‘Avatar’ is ‘Dances with Wolves’ in space. . .sorta’, Los Angeles Times. Available at: https://www.latimes.com/archives/blogs/hero-complex-blog/story/2009-08-14/james-cameron-yes-avatar-is-dances-with-wolves-in-space-sorta (Accessed: 2 February 2023).
The Last of Us (2013) PlayStation [Game]. Naughty Dog. Sony Computer Entertainment: Japan.