In this video, I used again Adobe After Effects and its Trapcode plug-ins to try and improve on the visual interest part of these experiments. I wanted a more complex particle animation, close to particle simulation and different variations to try and add diversity to the visuals.
For the music analysis, I focused this time on low frequencies to isolate different beats and percussions types which where prominent in this track.
An interesting thing I noticed while trying to extract various sound features and link them to different visual features was that it got confusing very quickly. Especially when visualizing different things that vary at the same time. It actually worsened the readability of the visuals I was going for. I found that as the visuals got more complex, the music features represented had to get simpler (more intuitive) to compensate. While it doesn’t mean that having various music and visual features to play with is bad, I think it is important that only one (or very few) thing vary at the same time and that variations are introduced one by one if going for maximum readability.
In this video, I used Adobe After Effects and the Trapcode plug-ins Sound Keys and Particular to generate particles, based on the music frequencies.
The spectrogram is divided into three frequency bands corresponding to low, mid and high frequencies. Each of these bands is linked to a particle emitter and the band’s amplitude controls the spread of the particle emission.
As observed in previous spectrogram-based visuals, beats related to low frequencies are easily recognized in the red emitter spread. However, I find it hard to accurately tell what the two other emitters are reacting to as it is a mix of harmonics from instruments and human voice.
The idea of these music visuals experiments is to explore different ways to generate visuals, as automatically as possible, directly from a music track. Ideally, those visuals should be entertaining and reflect what stands out in the music from a listener point of view.
In this video, I used Matlab to compute the spectrogram of a music track and display it as concentric color rings. Small rings in the center correspond to low frequencies, whereas larger rings show higher frequencies. The ring brightness is directly linked to the corresponding frequency amplitude in the spectrogram.
While the spectrogram contains a lot of information on the frequencies of the sound, it is not intuitive to link what it shows to what we hear. The low frequencies corresponding to beats and percussions are easily distinguished, however higher frequencies contain a mix of harmonics from different instruments such as piano and human voice. While we can easily distinguish them by ear, it is fairly unintuitive to do so in the spectrogram visualization where both are entwined.
In order to improve the visualization readability, more complex information can be extracted from the spectrogram such as melody, pitch, tempo, etc… These are intuitive properties used to describe music to human listeners and therefore should be used to guide an intuitive visualization. However, these properties are not always straightforward to compute and extract from the sound wave data. It gets even harder when the data is a mix of different instruments playing different melodies.
Research on these topics is being done and tools are already available to try and extract these properties from sound data. For example, the Matlab MIR toolbox provides methods for different feature extractions, and other challenges like melody extraction can be tackled using advanced algorithms. While these methods provide impressive results in some cases, they do not work with any input and still fail when dealing with challenging cases.
These tools are promising ways to improve the music analysis part of this, though my priority for future experiments will probably be to improve on the visual generation part to try to get interesting visuals even for simple and easy to compute music features.