Microsoft Research helped ‘Gears of War 4’ sound so good

Popping in and out of cover has been a hallmark of the Gears of War franchise since the first game came out in 2006. It hasn’t changed much because it didn’t need to. What’s always been an issue though is how thin the game sounds — a shortcoming of the underlying tech, Unreal Engine, powering it. But Microsoft owns the series now and has far more money to throw at it than former owners/Unreal Engine creators Epic Games did. With help from Microsoft Research, Redmond’s Gears of War factory The Coalition found a high tech way to fix that problem. It’s called Triton.

Two years ago Microsoft Research’s Nikunj Raghuvanshi and John Snyder presented a paper (PDF) titled “Parametric Wave Field Coding for Precomputed Sound Propagation.” The long and short of the research is that it detailed how to create realistic reverb effects based on objects in a video game’s map, to hear it in action pop on a pair of headphones and watch the video below.

At its simplest, Triton looks at an entire video game level and calculates the reverb properties of every material. From there, it applies realistic echo/reflection effects to the soundscape. This means incoming fire passing over a wooden crate sounds different than it would a brick wall.

It’s a much different approach compared to the way other games tweak audio to sound more realistic. Those rely on comparatively basic volume ducks and one-size-fits-all muting effects to create the illusion of occlusion and obstruction between you, the player, and the source of the sound. Peek your head up to take a potshot and the sound is unobstructed, but because you’re sitting behind a wooden garden box in a greenhouse instead of a cobblestone plaza it’s going to sound distinctly different.

Obstruction and occlusion are what keeps dialogue or battle sounds in adjacent rooms from playing at full volume, confusing the player as they walk around an environment, too. Rather than adjusting the levels of audio clips for distance, Triton considers the simulated materials and acoustical properties of the surrounding gamespace. Sound bounces off of a cave wall differently than a deserted hospital, so Triton does its best to simulate that.

If you need an example of obstruction and occlusion gone horribly wrong, fire up the recently released BioShock Collection — also based on a version of the Unreal Engine. Early on in the first game, a splicer enemy is singing a lullaby to the revolver in her baby stroller. Yes, she’s crazy, and the scene is absolutely creepy. But her voice sounds clear as day from two rooms over when there’s 30 feet (and a wall) between you. This is the type of thing that robs a scene of its immersion and atmosphere and undermines the developer’s intent: to make you forget you’re playing a video game.

“[Triton] is quite the complicated beast,” Gears 4 audio director John Morgan said. “The occlusion and obstruction values that Triton gives our game are really, really accurate to every listener position permutation possible on the map.”

A gunshot sounds basically the same in any situation in BioShock, but in Gears 4, a shotgun blast will ring out differently in the middle of a cathedral than it might in the entry way. Why? Because the ceiling is higher. It sounds like an insignificant detail, but getting it right makes a huge difference in terms of immersion — especially if you’re shooting a gun literally thousands of time in a game and rarely hear the same sound effect twice.

“A lot of people would be like, ‘Why does it sound like I’m in a bathroom when I’m going through this hallway?’ People playing the game didn’t understand why it was wrong, but it felt wrong to them,” Morgan said. “A big part of that is how does the dialog sound? How do the weapons sound? How does the reverb of that space, or the acoustic properties of that space sound? “If you get it wrong, people will notice right away.”

“People playing the game didn’t understand why it was wrong, but it felt wrong to them.”

Triton probing for sound data; player in the middle.

One of the examples Morgan is most proud of is near the game’s beginning. The new Gears have just finished fighting a few waves of robotic sentry guards and have to cut through a hospital’s maternity ward. Outside, trees sway in the breeze that descends from the mountains surrounding the deserted city; leaves rustle along the cobblestone street as the wind whispers around brick facades. The area feels like large open space because it sounds like a large open space.

That all changes once you bash through the door. Dialog has more echo as it bounces around the tiled floor and off of the propaganda posters lining the walls. Indoors, the shotgun has a more menacing bellow. Crossing the threshold back outside or into a glass-domed foyer and this changes because there isn’t a ceiling, or the one that’s there is higher and more reflective.

It’s this type of nuance and subtlety that set Gears 4 apart from its competition — even within Microsoft. But Triton’s fancy tech is a double-edged sword: Morgan said that a bulk of The Coalition’s early audio and engineering work was just getting the toolset up and running at parity with its previous project, the Gears of War Ultimate Edition remaster.

And because it’s specifically tailored to work with Unreal Engine and the third-party Wwise audio toolset, Triton’s benefits won’t make their ways into games using other design tools (even those made within Microsoft) anytime soon. Morgan said it’d be somewhat easy to port Triton into another Unreal game, but wouldn’t work with a custom game engine like, say, what Halo or Forza uses without a ton of additional labor.

If you get [reverb] wrong, people will notice right away.

John Morgan

What’s here is a far cry from the common tools used across everything Electronic Arts makes with its Frostbite engine.

“It’s not something we can just go. ‘Here, we’re handing this off to you.’” Morgan admitted. “That’s the tricky part right now if other game teams are looking at it. We’re trying to make it easier for them.”

That’s a shame, because save for Uncharted 4, there simply aren’t many other games that sound this good. Yet. A feature that’s exclusive to one game or toolset eventually finds its way elsewhere. Just look at the HDR audio system EA’s DICE studio pioneered with the Battlefield series or the lens flare effect that permeated practically every game in the late ’90s. That’s thanks in part to developers sharing their secrets at venues like the Game Developers Conference.

Microsoft has done the financial and physical heavy lifting to get Triton up and running, now it’s up to every other developer to follow suit and implement it or something similar.

“Nikunj [Raghuvanshi] has given a Siggraph presentation on [Triton], it’s not a secret anymore,” Morgan said. “People know we’re doing this, and that’s how it works.

“I think it’s a matter of going, ‘Do we want to spend the money on trying to make this happen?’”