VR is here - and you should be getting creative with it. With VR hardware moving from prototypes to widely available hardware - and YouTube and Facebook letting you watch 360-degree footage on your computer, tablet or phone - it seems pretty obvious that the time is right to be getting involved in design and creation of 'immersive' content ('immersive' may sound like a wanky marketing term, but it's the best we've got as a catchall for both interactive and non-interactive 360-degree content that includes that produced for VR headsets and for online viewing). But how do you approach the 360-degree world?
The first question – and the one's that's hardest to answer right now – is conceptual: what is VR for? Who will want to 'experience' it ('watch' and 'play' don't quite seem to be the right terms here), where will they be and what do they want to get out of the experience?
“A year and a half ago, we were talking about a concept you could refer to as 'first-person cinema',” says Simon Robinson, chief scientist at The Foundry, which has built some of the first generation of tools to edit and add effects to 360-degree footage. Examples of 'first-person cinema' include both short-form and feature films such as Catatonic and Zero Point, where you watch a story unfolding around you as you move your head around.
For the viewer, this has an immersive effect – you feel more like that you're there than with conventional films – which can make both fictional stories and documentaries more powerful.
However, while we've seen some brilliant examples of these, it seems we need to go in different directions to get people to invest time and/or money in the hardware necessary to experience VR. Getting people to try the cheaper end of VR such as the Google Cardboard VR-viewing box for your phone - or even 360-degree videos on YouTube or Facebook - seems to be best done with projects linked to brand and concepts they already know and love. The New York Times experiment with creating films and documentaries (below) for Cardboard is a great example of this – though it did send out 1.1 million Cardboards to subscribers for free – and Simon notes that "we were talking to ILM about mini Star Wars VR episodes that you can see on Google Cardboard, to tie in with the film release."
For more expensive hardware such as Oculus Rift or Steam/HTC's Vive VR, you have to raise the bar higher.
"It’s not entirely just first-person cinema, where you’re embedded in the middle of some action, but you’re involved to some extent in things that are happening around you – and you are required to participate," says Simon. "It’s not quite gaming, but you are guided through a story by participating. It’s a bit like immersive theatre [such as a Punchdrunk production] – where you are neither an active participant, nor a passive one. There’s clearly a script laid out for what’s going to happen but you are involved in the process.
"You feel like you have choices but you don’t, necessarily.”
VR is not just a film in 360-degrees
So once you've got your concept sussed, how do you create an amazing experience? Well, the first thing you shouldn't do, according to Mativision CEO Anthony Karydis – who’ve created VR and 360-degree projects for MTV and Vodafone – is start of thinking like you would for a conventional video or animation project.
“Samuel Taylor Coleridge coined the term 'suspension of disbelief' for how people should be immersed in a story,” he says. “Creators of VR experiences need to exploit this same thing – the viewer must be able to suspend disbelief to be able to get the most out of the experience. It needs to be presented in such a way that the viewer can believe that they are experiencing the truth.
“All too often you see examples of where people have produced something in an environment, but because of the lack of cues the viewer has no idea where they should be looking, so they miss what’s going on. This makes the experience very disappointing. It’s a common artistic problem. We need to remember that 360 video and VR is not like a cinema. Here you have to teach the viewer to receive cues and turn in the correct direction at the right time.
"The VR experiences that will be most successful will be the ones where people are fully immersed and feel they have seen the things they wanted to. Otherwise it will be simply about creating a world/rooms where people go and just wander aimlessly around. That would kill VR.”
Rewind creative director Daryl Atkins says VR designers need to be asking questions like ‘how are we using space?’, ‘how do we want our audience to feel?’ and ‘what’s the point of it being there at all?’
Rewind has worked on some of the most high-profile VR experiences so far: including Björk’s Stonemilker 360-degree music video (top), the Red Bull Air Race and BBC's 360-degree video experiment from the centre of the Strictly Come Dancing show floor (which you can watch a bit of below).
“On Strictly, we delivered a ‘best seat in the house’ experience that gives the user a personal viewpoint that wouldn’t have never before been possible,” he says.
Daryl describes immersive content is a deeply personal and engaging experience. ”It’s a powerful format in which to deliver narrative," he says. "We have the opportunity to be extremely direct to our user without the visual contention that other media can suffer [from], allowing for our cues and design choices to be absorbed far more sensitively by our viewer.”
As with any new medium, Daryl notes, it's important to understand that you haven't got the wealth of experience you have with other forms - so you need to spend more time experimenting and evaluating the results at every stage of the project.
"This often requires lots of pre-production, blocking out acting, greyboxing experiences, and testing, testing and testing again,” he says.
A new venture of the Manchester-based dock10 studios is also helping the BBC get immersive. The 360VR content service there recently unveiled five different immersive shorts for the BBC programme The Voice. As the show is also made at dock10's MediaCityUK studios, these shorts employed a broadcast-quality level of post-production - immersing viewers in highly detailed footage captured on multi-camera 4K rigs. As with Rewind's work on Strictly, the aim was to give the viewer the experience being part of the production - this time also going backstage.
“With our work on The Voice, the viewer was given entry to some of the backstage areas that aren’t part of the traditional broadcast experience,” says Richard Wormwell, head of 360-degree production. “They could walk in the shoes of a contestant, stand in front of the big red chairs and be in the best seat in the house while the four coaches perform the opening song to the series.The ability to give people this sort of ‘access’ works really well, it’s had some great response from the public.”
How to design an immersive experience
But how do you build these experiences? It’s a fairly new world after all.
Richard says the dock10 team considers a few key things before it embarks on a 360? project.
“When content is viewed through a VR headset we have to make sure that people are not confused and overwhelmed with their new surroundings,” he explains. “We have to give people the time to understand their locations and allow them to get comfortable before the action starts. Once they’ve had enough time to settle in we don’t want to rush the pacing: the sort of quick cuts and fast camera movements that we use in traditional TV and film don’t work in VR.
"Instead we use techniques to guide people though their experience - but it’s also important to let go, to step back and let the viewer take part in discovering the story for themselves. We have to be spatially aware; we can’t just fill one area with action. People can, and will, look everywhere, so we try to make the whole scene interesting. Every project we work on teaches us more about how the medium works but there’s a long way to go and the content we create will only get better and better.”
Daryl says that there's a difference from desiging for a traditional media approach to concept work, such as a frames in a storyboard showing the action that needs to be conveyed. “Thinking in frames ultimately inhibits the scope of this medium,” he says. “Traditionally directors take the choice out of the hands of the audience, but this is no longer an option. Now we must show intent rather than explicit framing.”
“Immersive video also suffers badly from 'forward bias': the context of the location is quickly disregarded and the user becomes fixated to a single static focus throughout. We often have great narratives pasted into a corner of their surroundings - just because we have the technology to do so.”
“There is a need to develop a better way for designing VR in a digestible way,” he continues. “We must extend our widescreen way of thinking and start thinking ‘spherically’. By designing our focal points in context of the environment image we discover a relationship with that environment. Thinking this way promotes thoughts about secondary interest features within the scene, rather than embellishing shots in the production phase. I believe this must work by presenting contextual framing in a hierarchical structure of primary and secondary features.”
What works, and what doesn't?
Rewind learned a number of interesting things about how humans interact with VR when testing projects with the HTC Vive - the VR headset created in conjunction with game developer Valve and its sister game distribution platform/online store Steam - which allows the user to roam freely within a space, rather than confining them to a seated VR experience.
“Unless there’s a really good reason not to, you should let physics do what the user would expect it to,” says Daryl. “If you can knock over one object but not another, it’s a frustrating experience which can break the sense of presence you have lovingly created. Likewise, if you lightly tap something, it shouldn’t fly off at great speed. If you can pull a lever, you should be able to pull others. You must spend a lot of time getting the balance of performance, rich interaction and lean design. When it works, it’s a greatly rewarding experience.”
However, you can’t always abide by these rules, especially when objects exist in the VR view that don’t exist in the user's real world vicinity.
“We have to look at ways of elegantly dealing with this,” says Daryl. “One scenario could involve a [real] tool in the player’s hand intersecting with a rock [that's only present] in the scene. One approach people have experimented with is to ‘ghost’ the tool as it passes through the surface. Another would be to have the tool drop from the player’s grip gently teaching them to avoid doing so. Experimentations with sticking the object to the surface can feel somewhat clumsy and incorrect as it becomes detached from your proprioceptive expectation [the expection at your body will feel movement through your proprioceptive sense - the 'sixth sense' of spatial awareness that's based information 'felt' in the inner ear and in your muscles].”
Daryl advises that you also need to be consistent and thorough with your implementation of audio.
“The material interactions should not only be correct, varied, appropriate and spatialised - but they play an important audioceptive role in helping the player master their interactions and finesse user motor control,” he says. “If audio triggers fractionally early or late, it can have a surprising impact on the user’s ability to judge their movement and thus their effectiveness at utilising learned skills.”
That, by it's very nature, virtual reality isn't solid can also cause problems that you have to design around. While we will instinctively avoid walking into virtual walls or tables, we're still aware they're not real - so can consciously try to go through them just because we assume we can (which breaks the immersion). However, the creatives we've spoken to say that people are much more likely to move around virtual obstacles if it proves directly advantageous for them to do so in the context of the game - for example, if they're 'punished' in a game for touching boundaries.
“We have learned to care for ourselves in the real world and that seems to translate really well into a virtual one,” says Daryl. “That's great news for experience design as it allows more than just a flat environment space. However, this subconscious desire to preserve objects and ourselves can knock people unsteady if it’s a permanent feature of the VR-space, even more so if it’s a trip hazard. Also inevitably people will eventually test objects by trying to pass through them. Some consideration might have to be made to handle this neatly to preserve the fifth wall.”
Into a VR future
So what will mastery of the VR form allow us to do? Richard forsees an exciting future: “As the amalgamation of 360-degree content, CG effects and creative storytelling improves - and we start to add further depths of realism through binaural audio, light field cameras and displays - the levels of immersion will literally be out of this world.”
“While VR can be used to mirror real life, it can also be used to immerse people in worlds that are purely fantastical,” agrees Daryl. “Imagine VR-ready representations of your favourite painting; you’d be able to enter that world and explore in a way that traditional media wouldn’t allow. You could be walking through a surreal Dali landscape, or exploring Monet’s garden. Similarly, artists are increasingly able to create content inside of virtual reality.
"Google’s VR painting app Tiltbrush [below] allows the user to paint within a 3D space and walk around their creation. The potential applications for design and conceptualisation are enormous, and will bring a new avenues for creativity and collaboration to multiple industries.”