The Hidden: Affective Conditional Modifiers in Adaptive Video Game Music
ATTENTION - This is a simplified overview of the following publication:
Dissertation Project (Computer Science MSc)
Research Overview
Video games have long used music to enhance player experience, from grand orchestral scores highlighting dramatic moments to subtle audio cues providing vital gameplay information. However, traditionally this music has been composed to convey emotions that match the game’s scenes and narrative, without considering the player’s real-time emotional responses. Our research explores a novel approach called Affective Conditional Modifiers (ACMs) - a technique that adapts video game music based on the player’s emotional state to either reinforce or modify their current emotional experience.
The Technology
We developed a system that measures players’ emotional arousal through electrodermal activity (EDA) - essentially how much their skin conducts electricity, which changes based on emotional intensity. This biosignal is processed in real-time to create an “arousal score” that indicates how stressed or calm the player is at any given moment.
The system uses this score to modify the game’s music through three different conditions:
- Help Mode: Reduces music intensity when players become stressed, attempting to calm them
- Hinder Mode: Increases music intensity when players are calm, attempting to stress them
- Control Mode: Maintains consistent music intensity regardless of player state
The musical changes themselves are implemented through dynamic blending between different instrumental layers and timbres, using the Wwise interactive audio middleware platform, creating smooth transitions that respond to the player’s emotional state.
The Experiment
To test our ACMs, we created a horror game called “The Hidden” where players must escape a creature that can only be seen in direct light. The game features two types of scenarios:
Search Tasks: 3-minute sequences where players must find items while being pursued by the creature in an open area. Players use a flashlight to spot the creature and a radio that emits static when it’s nearby.
Chase Tasks: 24-second sequences where players must escape the creature while avoiding hazards. The creature’s proximity to the player is influenced by their arousal level.
The game was designed with minimalist visuals to ensure that player responses were primarily influenced by the audio rather than visual elements. We also implemented a “super-task” health system that prevents player death while maintaining an illusion of danger.
Key Findings
While our participant pool was smaller than intended due to external factors, we made several interesting observations:
-
Game Familiarity Effect: Players generally became calmer in later stages of the game, regardless of the music condition. This suggests that game familiarity might have a stronger impact on emotional state than musical intervention.
-
Musical Integration: Players reported that the adaptive music felt natural and fitting to the game, even when it was working against their current emotional state. This suggests that ACMs can be implemented without breaking player immersion.
-
Emotional Impact: Participants strongly agreed that the more intense music made them feel more stressed, while the less intense music made them feel more relaxed - even though most didn’t consciously notice that the music had ever changed.
Future Directions
Our research highlights several important considerations for future work in this area:
- The need to account for individual players’ emotional resilience and response profiles
- The importance of appropriate trigger thresholds for different game contexts
- The potential benefit of including calibration phases to establish baseline emotional responses
- The value of varied gameplay elements to prevent predictability
Impact
This research represents one of the first systematic attempts to implement and study emotion-based adaptive music in games. While further research is needed, our findings suggest that ACMs could be a valuable tool for game developers looking to create more personalised and emotionally engaging experiences. The techniques we’ve developed could be particularly valuable in horror games, but could potentially be adapted for any genre where emotional manipulation is a key component of the experience.
The research was supported by the UKRI Centre for Doctoral Training in Artificial Intelligence and Music and OVOMIND SA, contributing to the growing field of affective computing in gaming and interactive entertainment.
Share on: