Rethinking game audio: from sound effects to player experience

A sound ultimately belongs to its listener.

Image generated with the Image Creator from Microsoft Bing

When you read my blog, you probably notice that I use the fundamental terms “sound” and “sound design” in a slightly unconventional way. It is time to explain why I do so. This is probably the most personal post on my blog, where instead of exploring how sound affects video game players, I explain my attitude and professional values.

I am a practicing sound designer, and my job is to make video games, not search for the ultimate truth, or explain specific phenomena (although I enjoy attempting that!). When thinking and writing about complex topics, I’ve learned to approach definitions pragmatically, prioritizing utility over accuracy. Consequently, I tend to accept definitions that help me make better games, even if they are questionable, inaccurate, incomplete, or borderline blasphemous to some people. I’ve realized that the definitions I use have a significant influence on my professional priorities, interests, ethics, and motivations. So, I’ve decided to elaborate on the most important ones in this post.

Sound and soundwave

The centuries-old thought experiment “If a tree falls in a forest and no one is around to hear it, does it make a sound?” still sparks debate. The answer ultimately depends on your definition of sound.

There are a few, but the most common ones are something like:

1. Sound is a pressure wave created by objects moving in physical space.
2. Sound is a perceptual entity, emerging in the mind of a sentient being when they receive the pressure wave described above.

Even though both definitions are correct, I believe the second one is more useful to game sound designers, because it broadens the perspective of what we do. There is also an anthropocentric aspect to it: if sound ultimately belongs to the listener, we need to focus our attention on the human, not on the pressure wave. This is quite relevant to game audio because, as in the same way sound doesn’t exist in the absence of a listener, the gameplay experience cannot emerge in the absence of a player.

Thus, by sound I always mean perceptual entity, and following Grimshaw and Garner, whose work ultimately convinced me to use this definition, I call the physical phenomenon a soundwave to differentiate between the concepts. But when working in game audio, there are other terms I need to work with. I depict their hierarchy in the following way:

• Audio file is a piece of digital data containing information for the speaker to emit a soundwave.
• Sound effect is one or more audio files arranged together to communicate something. A sound effect can contain multiple audio files if they are supposed to play together as layers to convey a single message.
• Sound event is a term I borrow from audio middleware terminology but use in a slightly different way. It contains one or more interchangeable sound effects with a shared meaning — sound effect variations. The difference with middleware terminology comes from the fact that, for example, a Wwise event can contain much more than that.
• Soundwave is the pressure wave that comes from the speaker when the gameplay system triggers a sound event.
• Sound is a perceptual entity that emerges in the player’s mind, usually elicited by a soundwave.

This perceptual definition transforms sound into a high-level, subjective concept, that depends not just on the information encoded in the audio files, but also on other sensory inputs, listening conditions and ultimately the listeners themselves — their life experiences, expectations, abilities, emotional state, and myriad other factors. But since most of these factors are beyond our control, wouldn’t it be more practical to focus on replicating the physical phenomenon?

Perhaps yes, but the perceptual definition of sound motivates me to always stay aware of the larger context. It would be ridiculous to discuss, for instance, ludic and narrative function of a sound meaning just a pressure wave. The context, and thus the sound, is influenced by non-acoustic factors, including visuals and haptics, narrative, gameplay systems, and more. This context awareness helps me conceptually decouple sound and audio file, leading to a few observations below.

1. Different sound effects can create the same sound. When we talk about character footsteps in a game, we usually refer to them as “the sound of footsteps”, not as “the sounds of footsteps” because they are variations of the same sound.
2. The same audio file can create different sounds. A child’s laugh on a playground carries a different meaning from a child’s laugh in a scary abandoned mansion, even if it results from playing back the same audio file.
3. The absence of an expected sound is a sound. If you walk in the forest and suddenly don’t hear your footstep, you look at your feet to see if you stepped into something nasty. You didn’t hear what you expected, and the absence of a soundwave created a perceptual entity that redirected your attention to your feet.
4. The absence of any sound is (most of the time) still a sound. If the game is completely silent, the player still hears their surroundings or something they chose to hear. And this could be an intentionally designed experience.
5. Any change in the auditory environment is a sound, even if no new audio file has been played. A change in the mix elicits the same perceptual entity as a sound effect. If your character’s health is low and you hear a low-pass filter muffling the mix, it is a sound.
6. Sometimes an audio file plays at a significant volume level but remains unnoticed and unheard, arguably not producing any sound. Or it produces a subconscious sound we are not aware of.

The perceptual definition of sound sparked my curiosity about many things beyond audio hardware, software, and methods of crafting an audio file. I am still near the beginning of this journey, and it will likely lead me to more conclusions, but it already heavily altered the way I approach sound design. Which is another term I redefined for myself.

Sound design

The term “sound design” has different meanings in different media, but in most cases, it revolves around one fundamental idea: sound is an object of design. Sound designer designs sounds, right? Disagreeing with that is a bit delusional, given the practical reality of our work.

However, I still want to challenge and reframe this definition, repositioning sound as means, not the object. An air fryer doesn’t produce fried air; it heats the air to cook something else. Sound designers record, synthesize, mangle, and implement sounds, design and build playback systems, but the goal is also to create something else. This is why I prefer framing sound design not as designing sounds for a creative product, but as designing a creative product through sound.

Game sound designer and an air fryer. Image generated with Adobe Firefly (Beta)

This perspective positions game sound design as not just making sounds for games (which is still an essential part of the job) but as collaboratively making games through the meaningful use of sound. This simple idea has completely changed my professional identity, leading to the following statements:

1. Sound design, like any design, starts with “why”. If we treat sound as an object of design, we are tempted to start with “what”, which can be a misleading question. The “why” defines the “what”, sometimes even cancelling it in the first place.
2. A sound without any purpose and meaning is called noise. Every “some sound” (as in “Hey, let’s add some sound here!”) is either a noise or a missed opportunity.
3. Making a sound that elevates player experience is more important than making a pleasant, impressive, or unique sound. It is a simple statement repeated by every experienced sound designer I know, but the definition I adhere to implies it with extreme clarity.
4. Sound has problem-solving potential, but to utilize it you need to know where the problem is. Understanding games as a medium is crucial to perform game sound design, because even though you can design sounds for something you don’t understand, you cannot design the experience you don’t understand.
5. The definition implies that audio is an integral part that shapes the experience alongside other integral parts, not a detachable component that can be applied on top of an existing experience. Some people disagree. Accepting this definition of sound design effectively blocked me from working with them.
6. Sound design doesn’t exist in a silo not only because you cannot co-design a game without a good understanding of what co-creators are doing. Exploring and building synergies with other creative disciplines and having genuine curiosity about what they do is a necessity.
7. Game sound design consists of sound effect creation, implementation, and collaboration. These three components are equally important, but most learning resources tend to focus on the first two.

To summarize, this definition of sound design is about understanding the broader context and using sound as a means to design a creative product, rather than focusing solely on the creation and implementation of individual sounds. This perspective has changed how I approach my work, placing greater emphasis on understanding the game’s goals and collaborating with the development team to create a compelling experience for players. The change brought one unexpected consequence: most of the skills I have barely transfer to any other medium. If for some reason I am unable to practice game sound design, I’d likely be more successful at doing some other work in games than doing sound design for film, theater or other medium.

Both definitions here are inspired by other people’s work, so please don’t attribute their origin to me. Even if you disagree with everything I say here, I hope this post has helped you understand the main idea behind my blog. And I now have a separate entry to link to, avoiding awkward digressions in my future writing.