Artificial Creativity: Sounding AI

»Cosmologies III«, for fixed 3D electronics, situates the listener inside a virtual grand piano to experience its secret inner life. The piano interior, recorded with a 32-channel microphone array, is complemented by 3D field recordings of Paris’s Place Igor Stravinsky, a glimpse of the reality outside the concert hall, fusing the piano-world with our own. These recordings are highlighted and underlined with computer synthesis using artificial intelligence to reproduce the spatial presence of acoustic instruments, while the microcosm of the piano’s inner space expands larger-than-life. Presented in a binaural version at ZKM’s »inSonic 2020« festival, this is the premiere of the multichannel version.

The modular work is preceded by »Cosmologies II«, an interactive 3D sound installation in which the audience becomes the performer of the virtual instrument. The ambient sounds and movements of the public, unintentional or intentional, trigger sparks of sonic potential energy that grow in intensity. The public is invited to explore and move about the space of the virtual piano interior before quietly taking their seats as the lights dim and »Cosmologies III« begins.

Author: Aaron Einbond

Production assistance: Benjamin Miller (sound director at ZKM | Hertz-Lab)

The production of these pieces took place within the frame of the project EASTN-DC, co-funded by the »Creative Europe Programme« of the European Union.

»Poetica Liquida« is a multichannel piece especially conceived for the Kubus at ZKM | Karlsruhe. Leveraging different tools for music generation and hybridation of timbre, this piece explores the notions of metamorphosis and hybridation, fluid identity, and essence. Machine Learning techniques like »Variational Auto-Encodeing« (VAE) have been used to transform generate sounds via variational neural audio synthesis. Tools based on recurrent neural networks (RNN)  have been also employed to freely explore the algorithm proposals as well as possible developments of the same pattern, searching for unexpected outcomes in the perspective of a round trip with the machine via supervised process.

»Sonata IV« consists of 4 parts, each separated by a shorter intermezzo:

 1. »Black With Silver«
 2. »Largo - Potential Sites«
 3. »Più mosso«
 4. »(Comment)«

One chosen issue while composing the piece was selecting a set of machine-learning techniques, and exploring ways to adapt and use these in creative workflows.

The "Largo" section includes a non-fixed layer, to be prepared for each new performance, and mixed in according to instructions in the score. This layer should be a sound recording related to events shaping world history in significant ways, either contemporary or historically. For this performance a demonstration in support of the current resistance in Ukraine was chosen.

Many thanks to the soprano Silje Marie Aker Johnsen.

Author: Anders Vinjar

How did your project came into being?

As a result of interest in the recent developments of machine learning algorithms for music generation.

What is your background for getting involved in these projects/residencies and your intention?

I attended the IRCAM cursus in 2015 and became interested in new developments of technologies for music generation.

How long, why, and how you use AI?

I use AI in the perspective of an »augmented creativity« and dialogue between the composer and the machine.

What is the advantage of using AI for your pieces and why was it mandatory for developing your pieces?

The advantage is that AI tools allow us to move away from habits and comfort zones and explore new territories. The machine’s output can be sometimes surprising and it allows us to look at the object used as input from different and new perspectives.

How did your project came into being?

IRCAM invited me to this residency after a period of collaboration with some of their researchers, notably Jean Bresson. Our collaboration concerns developing and experimenting with how various techniques based on modern days Machine Learning can be useful in creative workflows, esp. those of composers.

What is your background for getting involved in these projects/residencies and your intention

Since the start my work as a composer has been exploring contemporary developments in computer-technology in artistic ways. The current residency is part of this continued search to find artistic potentials in the technology of every-day life.

How long, why, and how you use AI?

Since the term became widespread during the 1950s, AI has always been much in use in creative work and contemporary composition work, by me and others. I started exploring AI techniques while studying musicology around 1990, at that time to explore new and potentially more meaningful ways to represent foreign musical traditions, which seldom fits very well within western musical concepts or ways of notation. This work soon continued in a creative direction, using the same techniques in my own composition work.

What is the advantage of using AI for your pieces and why was it mandatory for developing your pieces?

By itself AI can not be said to provide a general advantage over other approaches in my own composition-work. It comes with alternative possibilities, other potentials, other questions, new problems – things which are very valuable to explore in the search for new and relevant artistic expression.

I find at least 2 immediate potentials by using modern AI:

 • possibilities to work controlled with more complex data
 • many of the algorithms and techniques involved are – for a person with a normal composition background like me – difficult to fully
    comprehend. By itself this is an interesting possibility to move forward as an artist, hopefully making art which is relevant in contemporary societies.

How did your project came into being? 

The project arose out of a STARTS Residency completed at IRCAM in 2019 - 2020 in which I collaborated with researchers Jean Bresson, Diemo Schwarz, and Thibaut Carpentier on a novel approach to interactive sound spatialization using higher-order ambisonic (HOA) and artificial intelligence (AI) techniques based on the radiation patterns of acoustic instruments. The artistic result was a new composition »Cosmologies« for piano and 3D electronics produced by IRCAM at Centre Pompidou with soloist Alvise Sinivia. After completing that work, I saw the potential to enlarge it with two more modular movements that approach three-dimensional electronics in the ZKM_Kubus in two complementary media that can be presented together or separately: an interactive sound installation in which the audience can move freely to explore the 3D electronics, and a fixed electroacoustic composition situating the audience in the midst of the microcosm of the piano interior expanded larger-than-life to fill the Kubus.

What is your background and your intention for getting involved in these projects/residencies?

While I have worked on projects with music technology throughout my artistic career, this project made more complex demands than before on computer programs, algorithms, acoustical science and technical thinking. This requires collaboration to produce a project that is broader than I could conceive as a composer alone.  Working in residencies at IRCAM and ZKM with collaborators Jean Bresson, Diemo Schwarz, Thibaut Carpentier, Alvise Sinivia, and Benjamin Miller (sound engineer at ZKM | Hertz-Lab) I was able to conceive sonic situations beyond what I could imagine alone.

Some additional information that is not included in your work descriptions?

The title takes the imagery of physical cosmology as an artistic metaphor. Derived from the Greek words kosmos, (world) and logia (study) the title suggests an analogy between the microcosm of the piano interior as it expands out to the space of the concert hall surrounding the listener, and the radiation expanding from the early universe over billions of years to reach us at our current vantage point.

Maybe some words on how long, why, and how you use AI?

Artificial Intelligence (AI) research is ubiquitous, yet often ignores the embodied presence of the live instrument and performer. This project instead centers on situated listening, using machine learning of audio features to decipher the intricate interdependencies of timbre and space that bring an acoustic instrument to life. While I have only been focusing on AI in recent work since I began my »Cosmologies« series in 2020, I will continue to develop it in the future. I am currently working on a new composition for percussion and 3D electronics as part of a European Research Council project “Music and Artificial Intelligence: Building Critical Interdisciplinary Studies”.

What is the advantage of using AI for your pieces and why was it mandatory for developing your pieces?

I began the composition process by making 3D audio field recordings and 3D recordings inside the grand piano using the Eigenmike 32-channel spherical microphone array. While this allowed me to capture and reproduce spatial acoustic phenomena with high resolution, I wanted to go further to complement the recordings with new, unpredictable responses. Using machine learning, the computer “learns” from 3D radiation patterns of acoustic instruments to respond to and spatialize new and processed samples based on their timbral features. These unpredictable reactions can be heard in real-time the installation »Cosmologies II«, where the computer responds to the ambient sound from the audience and concert hall, as well as during the fixed electronics of »Cosmologies III«, where it responds to the piano and field recordings.

Everything else that you think would be interesting to read?

The natural acoustic phenomena used to train the machine learning algorithm come from a study at the Technische Universität (TU) Berlin of fourth-order Ambisonic radiation patterns for 41 acoustic instruments. I implemented this with an interactive Max patch using software packages Spat and MuBu for Max to map samples to these measured HOA patterns in order to “learn” from the instruments’ unique spatial characteristics.