Condenser Mic in Development at A/B Labs

About Synthia

Synthia is the first musical instrument to be released from Project Ludi at Almer/Blank Labs.

You can watch R Blank’s talk, ‘Hearing Pictures’, about Project Ludi and the creation of Synthia (recorded at FITC Toronto on April 26th, 2010).

Project Ludi is a long-term endeavor based around the challenge of successful execution of cross-modal metaphors — translating from English into Russian has a certain set of challenges; translating English into a color, or a musical score, or a weather pattern has entirely another set of concerns. And that’s what Ludi is about.

As a part of Ludi, Synthia is an instrument that composes infinite musical canons by using software to translate pictures into a score, which is played back in real time with the mixing of pre-rendered audio files.

There are three important facts about Synthia:

  • Synthia will compose music from any image
  • Synthia will compose the same music from the same image
  • Synthia will compose similar music for similar images

The reason that Synthia works boils down to the repetition and variation of short musical phrases. So Synthia translates an image into short musical phrases, and then loops those phases with variations (with the specific variations also determined by the image).

In broad strokes, the rules that Synthia applies are:

  • Synthia composes music for three instruments, all in the same key;
  • Key is determined based on overall color;
  • The actual instruments selected depend on the relative color in the picture;
  • Each instrument receives a color assignment;
  • Within a key, Synthia determines which notes are playable for each instrument by the color range — one instrument inherit’s red’s histogram, one plays from the green histogram, and one plays a subset of notes determined by the blue histogram;
  • Tempo is determined by a crude metric of variation within the image;
  • Each instrument plays a short musical phrase that repeats;
  • The notes of the phrase (selected from the playable notes for each instrument) are determined by analyzing how the average color of a block relates to the other colors;
  • Similar metrics determine volume adjustments and the frequency of rests;
  • Synthia loops the short tracks, applying chord progressions determined by the pattern of color repetition in the image

You might ask, is it possible for a human to predict what Synthia will compose from any given image? To a certain extent, yes. Darker images are lower in scale. Images with fewer colors produce fewer notes. Images with lots of blue are in a minor key. Images with more variation produce faster work with more rests. And so forth.

But the algorithms are still young. This is the first version. We would ideally like it to become more predictable, more truthful, over time.

About Almer/Blank

Almer/Blank is an interactive production agency, based in Venice Beach, CA, that creates digital experiences for Fortune 500 brands including Apple, IKEA, Microsoft, Mattel and the House of Blues.

For more information, please visit AlmerBlank.com.