Experiments With Musical Automata
I've recently gotten back into designing things with the Godot game engine, which I've been using and learning since 2018. I've started many projects over the last seven or eight years, and while most of them are still unfinished, all of them were fun to work on. Right now, I'm working on creating a sandbox tool for designing and enjoying musical automata.
A few weeks ago, I read about Brian Eno's work in generative music (not related to generative AI, which it pre-dates by several decades) and that inspired me to try my hand at it myself. (See this interactive explainer for an overview of generative music, and this transcript to read Eno talk about his work.) I'd like to avoid associations with AI, so I'm not referring to this project as "generative music", but it very much owes its inspiration to Eno's compositions.
My first idea was to take Conway's Game of Life and apply its principles to a music theory tool called a Tonnetz. The Game of Life is an example of a cellular automaton, which basically means it's a grid of cells which change state ("alive" or "dead") based on very simple rules. In Life, these rules say that whether a cell is born, survives, or dies depends on how many living neighbors it has. Too few, and it dies as if by isolation; too many, and it dies as if by overpopulation; just the right number, and a new cell is born. From these simple constraints, all kinds of wild patterns emerge - "gliders" that travel across the screen, "oscillators" which repeat the same positions over and over again, and even "glider guns" that create more gliders.
The Game of Life is Turing-complete, which means that you can use it to program or model anything you want (heavily simplifying here). People have built computers inside the Game of Life. People have used those computers to program the Game of Life inside the Game of Life.
Anyway, I wanted to know what would happen if you took something like a Tonnetz:
and used it as the grid for a game of Life. In a Tonnetz, each vertex (point) represents a musical note. It looks way more complicated than that, but that is literally the whole thing. Where it gets interesting is when you examine how those notes are arranged and how they relate to each other. I won't get into the music theory details, but just know that the notes are in the positions they're in for a reason, and that reason has to do with how they sound when you play them one after another.1
After a detour into digital audio encoding technology, learning how to synthesize audio in Godot, and a bunch of sloppy lines of code, I had something hacked together:
Not much to look at, but it worked! You could click cells (notes) to turn them on, then press "Start" and let the simulation run. Just like in Conway's Life, all kinds of cool things showed up, flowing across the screen in mesmerizing patterns and shapes. And the rudimentary audio synthesizer I wrote was performing really well, even in the slower programming language2 used by default in Godot.
Just one problem: It sounded awful.
Yeah . . . that's not really what I had in mind. It's cool in its own weird way, but not very nice to listen to.
Admittedly, I kind of saw that coming. I was hoping that since the Tonnetz is not just a random jumble of notes and is arranged in a way that has foundational western music theory ideas behind it, it might produce something that sounded interesting, if a little odd. Instead, it pretty much just sounded like a cat rolling on a piano. Oh well!
Fortunately, I had a great time building it and was in good spirits. After showing the "Life Tonnetz" to my dad and seeing his fascination with it, I knew there was a possibility of getting something really cool out of it. So I regrouped and went back to the drawing board.
I started trying to think of ways to encourage the system to produce better-sounding results. In physics, there's a concept called the harmonic series.
When you play a note on a musical instrument, there's actually a whole bunch of notes present in the sound that comes out. We just perceive them all together as a single note. The harmonic series describes the patterns that we see in these bunches of notes, and it also helps us understand why certain notes sound good (consonant) together and certain notes sound bad (dissonant) together.3 What if I could use the harmonic series to create rules for the system in some way? For example, what if instead of just looking at a cell's neighbors, the rules considered cells that were related to each other by the harmonic series?
It's a little odd to think about how this might work on a grid like in Life or the Tonnetz, so instead I started imagining sets of notes in rows, then chaining those sets together, almost like each set was a different instrument in an orchestra. What if there was a set of notes that was the low sounds, the bass notes (very important in making chords in music)? That set could have just one or two notes playing at a time, and then it could tell other (higher-pitched) sets about what it was doing. Then the higher sets could play certain notes depending on what the bass set was doing, making sure to pick notes that sound better with the bass notes.
I started seeing each set of notes as a layer in a system, with each set feeding into the next one above it (the lower sets feeding into the higher sets). Then I realized: that's basically how an extremely common machine-learning tool called a neural network is constructed. Now that's kind of exciting.
The super basic idea of a neural network is that it's a bunch of neurons (what we've been calling "cells") organized into layers, where each neuron in a layer sends information to each neuron in the next layer. Those receiving neurons then take that information, do something with it, then send that new information to the next layer. This repeats until the information reaches the output layer, and the algorithm has produced a result. The input at the beginning could be something like "Here's a picture of a handwritten number, what number is it?", and the output layer would spit out an answer to that question.
In the musical network I imagined, though, there is no output layer - each neuron/cell simply has a note associated with it, and depending on which notes are active in the previous layer, those notes can be playing or not playing. The (hopefully) musical result is the overall state of the network, not just the final layer. Over time, as the input (maybe just the state of the bass layer) changes, the whole network changes and thus the music changes.
So that's where I'm at right now. I've been rewriting my code to organize notes into layers, which are organized into "graphs" (collections of layers). The plan is to use a combination of the harmonic series and information from the Tonnetz about relationships between notes to influence how each layer affects the next. I'd like to make it look pretty while it's running too. And, hopefully, it sounds nice!
More soon-ish!
Image credits4
Specifically, the Tonnetz is meant to help visualize voice-leading patterns and show transformations that allow chords to neatly transition to other chords.↩
Godot has its own custom scripting language called GDScript, which is based heavily on Python. I love GDScript (and Python), but it's undeniably less well-suited to certain high-intensity processing tasks than other languages. To compensate for this, Godot also supports C#, as well as a bunch of other options if you really want to get your hands dirty.↩
The pedantic musicologist in me feels the irresistible need to clarify that the perception of consonant and dissonant being governed by the harmonic series in this way is very much not universal. Many cultures feel completely differently about certain combinations of sounds than we do in the west!↩
Gospers_glider_gun.gifvia Wikimedia Commons, Lucas Vieira BY-SA 3.0. The Tonnetz is from the blog listed in the image, but that blog doesn't seem to exist anymore. All other images public domain or created by me.↩