Matthew Marteinsson on Serving the Game with Sound

From Harmonic Content 005

From working on Need for Speed earlier in his career to his nearly 15-year tenure at Klei Entertainment (Don’t Starve, Oxygen Not Included), Matthew Marteinsson has been around the block a few times when it comes to game audio. When I put out the call for this issue, Matthew brought up the concept of “serving the game,” which immediately reminded me of “serving the song,” an idea I’ve heard over and over from musicians and producers. Serving the song means quieting that voice in your head that wants to show off your skills and instead tuning into what the song needs to sound its best. So what’s serving the game? I’m glad you asked. Or are you glad I asked?

What does it mean to you for sound to serve the game?

I like to say that every sound needs to be doing one of two things: it needs to inform the player or build the world—or both, hopefully. To me, that is serving the game. From there, you can better serve the game when you start to think about things like the fidelity of the game, which is a new thought that was brought up to me in my work. I’m still processing what exactly that means.

That sounds interesting. Let’s start with fidelity.

Think of that Super Mario video where somebody dubbed in all realistic sounds.

I haven’t seen that, but it sounds hilarious.

It’s really weird. It’s all proper human grunts and real squishes, not eight-bit or 16-bit video game sounds, so there’s a giant disconnect because that’s not the fidelity that game needs or wants. Something super cartoony wants super cartoony sound to go along with it. It doesn’t want hyper-realistic. If you took a cartoony shooter and dropped in all the sounds from that giant gun battle in the movie Heat, you’d be like, “This is just weird.” You need to serve the fidelity of that game.

The other thing I think a lot about is how audio people often like to overcomplicate things and make really cool systems that don’t necessarily serve the game. It’s serving us to be like, “I made this cool thing,” but did the game actually need that? Maybe you could have gotten away with something simpler. The game team I’m currently working with is 12 people total and I’m the one sound designer, so I have to balance my bandwidth, and serving the game then becomes “What’s the most important thing?” I might have a really cool idea that’s going to take me a bunch of time, but is that system really important to the game, or can I do something quick and spend more time on something more meaningful to the player?

What are some examples of sounds that serve the game well, or ones that don’t?

UI sounds seem really easy on a surface level: just inform the player. You hit a button, make a button sound, and you can have something that informs the player in a positive or negative way quite quickly. I think any of us can do that, but just grabbing a generic accept or negative sound isn’t serving the game or building the world. This is where we can get into informing the player and building the world by thinking about the context of the game.

I was working on some UI stuff for Invisible, Inc. and I was concentrating on just quickly getting something in. It was a totally fine accept sound, but one of the designers was like, “Hey, I just don’t feel like this really works. It doesn’t fit the game.” When I thought about it, I was like, “Yeah, this isn’t serving the game because it’s not coming from the world in which this game exists.” Often what I’ll do is take action sounds from the world that I’ve already created and turn them into UI sounds, or take the effects processing I was doing for other things and put them through that. If it comes from something in-world, it melds better with everything and becomes part of that world.

Do you ever slip into that classic sound designer habit of getting lost in the weeds and spending way too much time on something, only to realize it’s just not working?

Less and less over the years, but yeah—even though I’ve been at this for 22 or 23 years, I still get it wrong. To combat that, I like to try to get stuff into the game as soon as possible. Even if it’s like 50% done, just get it in the game and you get a better idea whether it’s working or not. It’s so easy to spin our wheels and just fiddle with things, but when it’s in the game, that’s when you know. If it’s a fantasy game, you swing a sword, you hit a creature, and it screams, right? There’s the hit, there’s the swing, there’s the scream. All three of those things need to work together and you’re not really going to know until you have all of them together.

We’re making informationally dense sounds all the time and we’re trying to inform the player of all these things, but what’s the most important thing to know in that situation? The first is, “Did I do something? Yes, I swung a sword.” The second is, “Did it connect with the enemy? Yes, it did.” The swing is no longer the most important thing, so it needs to get out of the way of the hit. And then the hit and the scream can be like, “Well, how badly did I damage them?” Then, maybe then the hit needs to get out of the way of the big scream. What should we be informing the player of changes every second, and that’s where we can get into complex mix systems to maneuver all these things so that the swing sounds really big and cool when it doesn’t hit anything, but then we can get it out of the way when it does hit something because we have other sounds that are now more important.

How do you approach a new project when everything’s fresh and you want to make sure that your sounds will serve the game?

First, I’m going to try to start making stuff as early as possible. This probably stems from being the only audio person on a team. It’s not like a triple-A scenario where there’s 20 of you and you all get your own specific area. So I get to making sounds as quick as I can, and I try to get a sense of the world. I’m probably going to start with ambiences first because I can look at concept art as a starting point. Maybe we have a dark and creepy forest or we’re on a moon with next to no atmosphere or we’re in a creaky old horror house. It doesn’t take much; I just need some concept art and a couple of sentences and I can start making something before we’ve got a game running.

So, your sounds usually evolve alongside the art direction?

Totally. I’ll start with those ambiences and get them working in whatever game we’ve got at that point so that when you fire it up, there’s at least some sound there. Then, I’ll move on to actions and UI, but I might be throwing out all those ambiences as the world changes. As I’ve gone on in my career, I’ve tried to embrace throwing stuff out.

In audio, we don’t really have a concept art stage like artists go through, where they’re just drawing a bunch of stuff and they throw a lot of it away because that’s how they get to what works. Why don’t we do that with sound? I feel like we’re under pressure to get it right with the very first sound we make. The first idea we have is the one that gets into the game, and it’s not really fair to think that we’re going to get it right the first time, every time. So give yourself the grace and leeway to screw around and make some shitty stuff, because that all leads to what does work. You need those failures.

I’ve heard of at least one example of that, in a blog by Steve Green about creating an “audio narrative” for the game Abzû. It’s a diving game, so he experimented with a diving helmet filter, but ultimately realized it was getting in the way of the player experiencing the underwater world. Thanks to that sonic “concept art,” they ended up ditching the helmet and making this really immersive free-diving game. Is that kind of experimentation more accepted in the indie space than at a triple-A game factory?

I think it depends on the game and the team. If you’re making a big, many-year project, you’re going to have that pre-production time at the beginning. But on the indie side, a lot of teams bring in audio people late and you’ve just got to hit the ground running. Whether you’re on a small team or a big team, you need to know what you’ve got the resources for, and then make hard decisions about what doesn’t fit so you don’t end up in a crazy crunch situation. And you’ve always got more ideas you’d like to get into a game than you possibly can, so it’s a bit of a hard decision when you’re like, “Fuck, I got this really cool thing, but if I do that, it means I don’t have time to do shield hits and it’s a combat game so that’s more important.”

How does collaboration with other disciplines come into play? Are there times where you think your sounds are working great, but you get some unexpected feedback from art or animation?

I’ve got an example from my latest game, and this is where the fidelity thing came up. I was thinking of a system and I went to the programmer and said, “Hey, I want to do this cool thing. Can you support me in building this?” And he’s like, “That’s cool, but I don’t think it fits the fidelity of the game we’re making.” I thought about it, and I realized it didn’t need something that in-depth. It just needed something simpler. Since then, I’ve thought a lot more about what the fidelity of our game is and how the sound can serve that and not my ego.

There’s not a lot of examples of art or animation pushing back on me, but there are quite a few instances of me asking for a little bit more in an animation so that I can put sounds on it. There’ll be an animation when a monster gets hurt and I’m like, “Can you make the mouth open when it gets hit so I can do an open-mouthed scream?” We’re very open to feedback in our studio, so anybody can have a thought about any area and it’s valid.

Do things ever make it all the way to the player before you get that kind of feedback?

We have historically worked in early access for quite a while, so we’re basically developing the game with the public input coming in. I’ve always found that the public at large are great for identifying where a problem area is, but they’re not good at the solution. When Don’t Starve was in early access, there was some severe hate on the pickup sound. There’s a “tink” when you pick up an item, and people just despised it. Somebody even asked for a mod to turn off just that one sound. So I went to our designers and I was like, “Is this a bad sound?” And they’re like, “No, no, it’s a good sound.” I’m like, “Okay, I’ve got other people saying it’s a good sound, but there’s still something wrong with it.” I think I just turned it down six or nine dB.

That was enough, just turning it down?

It was just way too loud. It was just so in-your-face and it cut through everything and it just needed to be really quiet and subtle.

I remember something like that in Morrowind, which I was obsessed with as a kid. For some reason, the sound effect for the player getting hit was ridiculously loud and abrasive in the mix, so I actually ended up searching online how to go into the game files and edit that one sound.

That’s one of the great things about games that have modding systems: you can see what players are doing. If they’re going in and adjusting sounds, they really care. It’s not necessarily an easy task to mod a game, so if they’re willing to do that, it’s an area of interest. Maybe you look at it and go, “Okay, some people like it that way and we’ll just leave it like that,” but in some cases you’re like, “I should really do something about this.”

I’ve seen complete sound overhaul mods for games like Mechwarrior 5 and Elite Dangerous, where people try to make it sound more like you’re inside the cockpit of a ‘mech or a spaceship. One person even made custom EQ presets for their music player so it would sound different inside each ship. People want an immersive experience and sometimes they think they can do it better—or maybe it’s just that they can afford to spend way more time on it than the devs could.

Yeah. That can go back to the scope that your team has. Maybe you initially had plans to make every single spaceship have its own ambience, and then more and more things got added to the game, the timing got tighter, and you’re like, “Okay, maybe now we’re doing one ambience per class of ship instead of every unique ship.” You just run out of time for the cool ideas, but it’s rad that players can then go do that if they want.

Thinking about mods like that really helped me come to terms with offering more options to the player to adjust audio. I definitely started out thinking, “This is my mix. It’s the perfect mix; you shouldn’t touch it,” and then I was like, “Okay, I’ll put in a sound effects slider and a voice slider and a music slider.” Now, I’ve gotten to the point where I’ll have those big sliders and then a sub-menu of even more sliders. However you want to play the game, I’m all for it. It becomes an accessibility thing—some people really need those adjustments to play better, and some people just like it. If you like playing my game with just the UI sounds and all the player movement sounds and ambiences turned off, whatever. As long as you’re enjoying the game, that’s cool.

It’s like you’re not just serving the game, but serving the player. In Battlefield 6, you can adjust the frequency of the tinnitus sound if it bothers you, which was a cool thing I hadn’t seen before. Things like that should be standard.

Yeah, and different mix settings if you’ve got a 5.1 stereo or just your TV speakers or headphones. Just giving the player the option is rad, I think.

Finally, do you have any advice for the game audio folks out there who want their sounds to better serve the game?

It’s really just about putting the game at the forefront and checking our egos. It’s not about us, it’s about the game, but I think it’s quite easy to get caught up in our own egos. I mean, we have a cool job. Making sounds is super fun and creative and you want to flex that all the time. I’m always trying to come up with new, creative ways to make a creature sound or an ambience or whatever, but there’s a checkpoint that you need to go through. Take a step back and go, “What does the game actually need? Am I informing the player of everything? Am I building the world? Are all the pieces coming together to serve the game?”