George Lucas once received a hearty round of applause back in 2011 when he declared that “sound is half of the experience of a motion picture.” If that’s true, what happens to the equation in the setting of VR and AR, where the vicarious experience of a 2D cinematic transcends into experiential 3D immersion?
After all, immersive worlds are creative imitations of the real world, in which sound and the auditory function play far more of an operative role than we tend to realize.
“Sound is usually an afterthought. Sound is invisible; it is vibrations of molecules and we cannot hold it in our hands, it exists only in time, and is the sonic expression of an object in motion.” said Laura Sinnott, a sound designer and doctor of audiology, at the NYVR Expo I attended.
Indeed, it is important to acknowledge the fact that in the process of growing up in a modern world filled to the brim with jumbled layers of noise, our brains respond and adapt to the chaos by numbing and tuning out (synaptic pruning) much of the full spectrum, and all for the sake of streamlining our daily lives.
Triggering an auditory renaissance
Sinnott kindly educated me recently on how fundamental hearing is to our very existence; that it serves not only as a medium for communication, but is key to our emotional well-being, cognition, and, perhaps more relevant in the past than our present reality, but even to our survival. Audio, she says, is the first sense to be developed in the uterus, primarily hooked to our instinctive center, and although it is an organ that we have lost significant sensitivity to, XR has the potential to help restore it.
“The sense of hearing is one of the most primitive senses; it likely evolved before vision, although after the balance system. It behooves sound designers, whether designing immersive audio for web or even traditional mono radio, to understand a little bit about just how powerful sound is.” Sinnott said.
Even if it is the case that this primary sense organ has somehow been relegated in our collective minds to playing the role of second fiddle, the panel last month at the NYVR Expo that was filled with experts in spatial audio made me a believer that XR can open the door to a kind of auditory renaissance in humanity’s appreciation for it. In the backdrop of the panel, the slides read: “Our eyes can only see what’s in front of us, our ears can take in the world all around us.”
Spatial audio design is key
To recognize that sound is not a second-class citizen and never was is something XR creators encounter quite soon in their attempts to create digital simulacrums that replicate reality sufficiently enough to trick the totality of our being into believing that the fantasy is real. They realize that, as Larry Chang, a Taiwanese composer and sound designer, wrote last month in a blog post I highly recommend, “without a proper audio design, the narrative experience will lose its charm and the world won’t be as convincing.
If the spatialized audio is absent or, even worse, misaligned in some shape or form, than the quality of our presence will suffer in proportion to that act of neglect or ignorance. Our physicality or instinctive programming is engaged in immersive content just as much, if not more so, than our minds and emotions are, so poor spatial audio design can prove as unseemly and uncomfortable as, for example, shoddy video stitching.
Sound is a core experiential layer
In fact, spatial audio serves as a fundamental layer to our cognition and it’s inclusion has a significant impact on our capacity for storing and recalling experiences.
“As the ‘Cone of Experience’ image illustrates, as soon as you add audio to an experience, learning retention goes up.” said Jeff McSpadden, a composer and multi-instrumentalist for film, television, advertising and new media. “Combining sight and sound creates a further increase (which is why video has become so popular), but then involving the learner in those sights and sounds, as is the case with most XR, creates a 90 percent certainty of learning retention!”
Above: The Cone of Experience
McSpadden is a pioneer in immersive audio design, serving as both the CEO of Composure (a transformative tech company creating immersive experiences that uses spatial and interactive audio to reduce stress and increase well-being), as well as Prime8, a creative lab that develops immersive audiovisual projects focused on the power of sound.
“With Composure, we’re studying how powerful interactive spatial audio can be when used as the focal point for an immersive experience. We aim to create custom music experiences that involve the user to help decrease stress and promote positive mood states.” McSpadden said. “There is plenty of evidence showing that passively listening to music is useful for managing stress, but the relaxing and focusing effects are increased when you involve the listener.”
There is also merit in going Daredevil and relying on AR wearables that forgo the visual component altogether; an approach that McSpadden describes as enhancing the aural experience instead of just replacing it as is the case with most headphones. Take the Bose AR glasses, for example, that were announced last March along with the $50 million venture fund to foster an ecosystem around it.
McSpadden sees Bose’s entrance into the market as a major win for everyone, but especially the blind, who can benefit from spatial audio for training, education, and entertainment with a depth and level of engagement that has never before been possible. It’s the kind of stepping stone that can accelerate the timing of augmented audio becoming a part of our daily lives.
Extending psychoacoustic research
XR as a medium also serves as an extension for the scientific and research community, which for its part has spent decades applying large-scale instruments like anechoic chambers (“free from echo”) for the purposes of better understanding the acoustic experience. They are, as Sinnott described to me, what happens to an environment when you remove the physical space and there are no objects to reflect sound waves.
Above: “The Air Force Research Lab’s test facility aims 277 loudspeakers in a 14-foot sphere toward a test stand, where a subject or sensor array sits.”
Image Credit: Eckel Noise Control Technologies.
“The anechoic chambers alters the frequencies of sounds that are typically heard.” Dr. David M. Greenberg said, psychologist and musician at City University of New York and University of Cambridge. “XR technology has the capability to control the frequency and vibratory properties of music to make true enhancements on sound and music.”
But whether it be for consumer-facing applications, enterprise implementations, or psychoacoustic research, it’s clear that immersive design will invariably elevate our understanding of how sound behaves in the world and hopefully also increase our awareness of the value and potential that this not-so-passive sense organ has to offer.
Source: VentureBeat
To Read Our Daily News Updates, Please visit Inventiva or Subscribe Our Newsletter & Push.
