The interactive installation by Jeroen Snik and Sandipan Nath

‘Watch, you’ll see my movements on the screen immediately’, says 3D animator Jeroen Snik (36), as he walks around in front of a camera on a tripod. Indeed, a shining star floats from left and right to up and down on the large screen in the meeting room of film producer Redrum. The digital star hangs in front of cloud cover over a fictional mountain landscape. It looks like a computer game, but it is a simulation of how museum visitors will soon experience an interactive installation, Snik explains. ‘The visitor stands in front of a screen on which he sees himself depicted in a digital landscape. The body movements are followed by this digital image.’ In this case, it’s a star above the clouds, but later it could be a sailor on a seventeenth-century Dutch East India Company ship.

To create a realistic experience, it’s essential that the sound also follows the image over surrounding speakers, says sonic researcher Sandipan Nath (30). For now, these sound movements are only visible as pulsing circles on the screen of his laptop; each circle marks a speaker. ‘The sound must not only travel over the speakers but also change in volume and pitch. This is because as sound gets closer, the frequency also gets higher. This so-called doppler effect must be synchronized with the user’s movements’, says Nath. This synchronization of image, sound and a person’s real-time movements is what the duo-for-the-occasion has been working on for the past three months in the context of the Building Talent grant scheme. ‘We’re actually making a virtual-reality experience without the awkward glasses.’

smart 3D camera

For the time being, Snik and Nath’s research project requires a certain amount of empathy from the future users – and from the image and sound artists themselves, for that matter. ‘After all, we don’t yet know where this multimedia installation will be located or what story the user will end up in’. Besides, the software to link the different data does not exist. ‘That makes this experiment rather abstract, even for us’, Snik says. Fortunately, the equipment is familiar to the two professionals: a smart 3D camera to capture body movements, an ultra-fast laptop to calculate the sound sequence and a computer with the widely used software Unreal Engine to create the animations. Snik: ‘The crux is to make the right connections.’

The two have synchronized image and sound, but now comes the refinement. ‘At the moment, the animation responds only when the user moves. But we want the surround sound to be activated as well by just waving the arms, for example. Each movement is then accompanied by a different soundscape.’ Nath says, with a smile: ‘The only way to find out is to lock up a 3D animator and a sonic researcher in the same room.’

commercials

The duo works about two days a week on this hands-on experiment. Partly online – ‘it’s all digital, after all.’ But they would prefer to sit next to each other in Redrum’s office; it means you can respond to each other faster. Non-verbal communication in particular is essential for creative chemistry. Both of them provide specific input to the project. ‘Sandipan can use his software to synchronize a sound document with live footage. We have no experience with that.’ Conversely, it was a breath of fresh air for the independent media artist to work with Redrum’s high-tech equipment. ‘It wouldn't have even occurred to me to link my software to the live footage from a 3D camera.’

What also comes into play for Nath is the commercial approach of a sophisticated film company like Redrum, which employs 40 people and produces commercials, animations, and educational and cultural films. ‘It’s an enriching experience to work on a user-friendly application of technology alongside autonomous art. In this respect, the user’s experience outweighs the intentions of the maker. It’s less personal, anyway.’

army barracks

Redrum is housed in a former army barracks in Ede that has been transformed into a creative hub and cool residential lofts. In the elongated business space, there are tables full of iMacs and flip charts with scripts. ‘For us, this collaboration with a young talent was our first introduction to the Creative Industries Fund NL,’ says co-founder Carlijn Pluijmakers Emmen. ‘Admittedly, we pay a lot of attention to training and innovation, including with our own Reducations, an in-house academy for film talent. Young makers in particular bring fresh inspiration or the latest technological gadgets. We also look at other disciplines as much as possible, such as music or installation art. As a company that is constantly committed to innovation, this exchange of knowledge is crucial.’

Although the two Building Talent participants each bring specific expertise – image for Snik, sound for Nath – they share a background in which art and technology converge. Snik swapped his Mechanical Engineering studies at TU Eindhoven for a successful career as an international DJ/producer under the stage name Icicle: ‘I was already making music videos, but when my DJ work came to a halt due to the COVID-19 pandemic, I turned entirely to digital animation.’ Indian-born Nath moved to The Hague to study Industrial Design at the KABK. ‘I ended up collaborating with my friends from the Art Science programme, where you learn to collaborate as a designer with other disciplines such as science or visual arts – in my case sonology.’ He graduated in 2020 with the installation Interference 53°N,42°E, in which the user experiences in a sensory way the noise nuisance from seagoing ships and oil rigs on the North Sea floor. ‘Only there was no visual.’

pop festivals

The goal of the experiment is not to design a tangible installation but to develop a multimedia template that can serve as a starting point for different applications, Snik says. For the proof of concept, a museum setting was specifically chosen. According to him, the cultural sector simply offers more freedom than a commercial assignment. ‘But who knows, maybe this technique can soon be applied at pop festivals or even a commercial trade-show presentation.’

As a result, the study does not have a clear end point. The animation software and also hardware such as the 3D camera are constantly improving, making new applications possible. ‘What if the user wants to customize the story they’re in with voice control? Or perhaps a 3D camera is no longer needed because haptic gloves are worn – gloves that convert hand movements into computer commands.’ Snik is exploring potential possibilities. Nath adds: ‘These gloves can also exert physical pressure that makes it feel as if you’re actually picking something up.’ But at the end of the day, the two creatives agree, it’s not about the technology. ‘We want to create a sensory experience.’

Text: Jeroen Junte