Push the boundaries of digital design using projection-mapping, Arduino, Microsoft Kinect & more

VVVV (vvvv.org) is another open-source toolkit that creatives can make good use of. It’s “great fun” too, says one of its firm fans, Stuart Dearnaley, 3D artist and innovation developer at Manchester-based agency The Neighbourhood (the-neighbourhood.com). “It’s great for prototyping an idea before really getting down into the design.”

The Neighbourhood is still best known for motion design projects, such as the title sequence for BBC show The Magicians (learn how it was created in our After Effects tutorial). It was only a few months ago that the agency ventured into the world of interactivity. 

“We started using a head-tracking set-up with a converted webcam to track an infrared source, then used that data to control a virtual camera,” Stuart says. “The virtual camera would mimic the viewer’s head movements, causing the image on screen to appear to [be] 3D.”

Stuart also managed to get the rig working using stereoscopic 3D, which he says greatly increased the depth perception of the ‘virtual window’. “As with the Christmas tree project, another of our internal R&D experiments, the goal was to discover and learn new techniques and apply another creative layer to make it more of a piece of art than a technical demo,” he says.

The Christmas Tree Project – aka the Cubic Light Tree (above) – was a 3D projection-mapping project which The Neighbourhood ran over the recent festive season. It used a stack of white boxes as a screen onto which the designers projected their animated creations. 

“I used VVVV to map out the faces of the cubes and in real time we could test out different image resolutions, colours and so on,” Stuart explains. “It was a great help to be able to work like that – flexibility is key to any type of innovative project where the goal may not be completely defined at the start.”

The team is now working towards creating an interactive installation, using projection mapping to display details of an architectural development in a physical space. “We plan to use an iPad for the interactivity and control,” says Stuart. “It’s a great device, and we will no doubt be using it in a few of our future projects and experiments. There are already a few free apps available that we can use to get input data across into VVVV.”

The Neighbourhood’s experiments with experiential design have included head-tracking and a project for Heston Blumenthal’s Fat Duck restaurant, in which the user manipulates a tile in front of a webcam and sees a real-time display of a sweet shop rotating on top of the tile

At the opposite end of the hacking scale is Twine, a little green box from US startup Supermechanical (supermechanical.com/twine). This low-power module can connect to a Wi-Fi network and thence to the Internet, allowing you to control it using a simple web app. It also has sensors to monitor the world, and its connectivity allows it to feed output back to the world via email or Twitter. Having enjoyed a successful crowdfunding programme via Kickstarter (kickstarter.com), Twine boxes should start shipping in May.

Another box that’s generating a lot of interest is Berg’s Little Printer, which uses the creative agency’s forthcoming BergCloud platform (bergcloud.com) to send information ranging from to-dos to a breakdown of your last exercise session. The Little Printer is fun if limited in scope, but BergCloud has the potential to add interactivity to all manner of objects. However, Berg isn’t ready to discuss the platform yet, an oddity in a community where sharing how projects were created is inherent in being part of it.

Alongside boxes expressly designed for engineered projects, there’s a growing trend to incorporate consumer electronics products – some of which boast sophisticated interactive technology – into hacked projects. Leviathan has customised the Microsoft Xbox 360 Kinect controller for control and visualisation purposes in the Amon Tobin shows and at last year’s Leo Burnett Leovative innovation event.

These aren’t isolated examples – the design world is brimming with people exploiting Kinect’s capabilities. One wily programmer has created a tool to motion-capture animation from Kinect devices to drive real-time character animation in Autodesk’s MotionBuilder application. 

The Design Zoo used the Arduino microcontroller in a device that fired fake snow at their office Grinch. A microsite let users launch the ‘snow’ and view a live feed of the consequences

There are more Kinect hacks on show on YouTube (bit.ly/xcGCWV), but Microsoft trumped most of them when it when it demonstrated a Kinect hack of its own last year. KinectFusion (bit.ly/qFookG) uses the depth-sensing cameras in the device to create a 3D representation of the immediate environment, then maps CGI visuals over them in real time. The PlayStation Move motion controller has been exploited too, by Studio Output, to interpret a scene and output real-time projection mapping images.

This use of games controllers has resonance in the field of haptics, the branch of technology related to touch/feel response in devices. The redoubtable Arduino has itself been incorporated into haptic projects, for example the Cryoscope. Another case in point is Cloud9, 3D ‘touch-modelling’ software from Edinburgh-based Anarkik3D (anarkik3d.co.uk) that uses the Novint Falcon device (novint.com) for haptic sculpting. Once you’re happy with the work you’ve created, it can be turned into a physical object by outputting it to a 3D printer.

“Haptic design is probably the next step up from touchscreens,” says Rita Mantler, one of the senior developers from Studio Output. “A really cool utilisation of this technology took place in 2009 when Japanese researchers [from the University of Tokyo] tried to make a hologram ‘touchable’ by using ultrasound vibrations . They could then ‘sculpt’ the hologram with their hands.”  

If you’re itching to get started on your first hacking project, don’t be afraid of the technology, advises Matt Daly, Leviathan’s principal and chief scientist. “Try a few approaches to break barriers and find a comfort level. Get your hands dirty – take apart and rebuild equipment. Spot the masters of the craft and study their approach and work.” 

You don’t need massive experience in coding to get started, according to The Neighbourhood’s lead creative, Rob Millington. “You can find examples [of code] and tweak them to learn what is going on,” he says. “To take something to the next level and invent a custom particle engine or a weird gravity simulation, I would personally need to get a proper programmer in. I can’t learn in a month or two what years of study, practice and play can give someone. 

“Collaboration is key,” Rob adds. “My advice would be to make friends with a good coder who shares a similar passion to yourself.”

Selling such nerdy concepts to the client might seem daunting, but as Millington says, a lot of clients have seen interesting work on YouTube and Vimeo and want to get involved. The Neighbourhood wants to do research to understand the technical aspects of the different projects and then come up with a delivery framework, he says. “Some of these R&D projects we are running in tandem with conversations with clients. “Others we are just getting into without any preconceptions of where we will go with it. We just want to develop our ability to tell stories in different locations.”

“The conceptual approach is still the most important first step,” says Leviathan’s Jason. “We pitch our experience projects much like we would a high-end broadcast spot: we envelop the message within motion graphics and chart the story along a timeline.

A dial selector allows the Albion bakery in Shoreditch, London, to tweet what is fresh out of the ovens, via an Arduino-based device by neighboring Poke Studio

“The commercial aspect can often be the recorded experience of the event, cut into a well-crafted story to demonstrate the impact and excitement of the event itself.” 

Adrian of The Design Zoo says experiential design is ultimately a smarter way of working. “Our research into this area, including the growing [number of] add-ons, developers and [availability of] open-source code, cuts down the project cost and development time for our proposals,” he says. “This is why I foresee more small businesses developing projects of large scale that would have been seen as extremely niche before and only capable [of being tackled] by the few.”

Note: We may earn a commission when you buy through links on our site, at no extra cost to you. This doesn't affect our editorial independence. Learn more.

Read Next...