“I see machine learning along a continuum” Holly Herndon’s take on AI music
After just three albums, Holly Herndon has already established herself as one of the major names in the AI music scene. Despite creating sounds that seem anonymous and impersonal at times, Herndon says making AI-powered music is still a very human endeavour.
At some stage through the two and a half year process of making her latest album Proto, Holly Herndon worked out her favourite method of sound production with her AI-powered system, which she lovingly named Spawn.
“Basically, I read a tonne of text and sang some passages and trained Spawn on my voice and that became Spawn’s entire universe, my voice,” she explained in a recent panel talk for Couch Lessons, a weekly series for essential conversations about artificial intelligence.
The method, which Herndon, calls “the voice model” is especially audible on the 12th track Godmother which sees Spawn trying to beatbox a composition by music producer Jlin. It’s a hectic song with blips and beeps and squawks, like a machine is trying to crawl out of a record player.
“I didn’t teach her to beatbox, none of the training data was me beatboxing in the studio,” Herndon adds, referring to Spawn as always with the female pronoun. “When she is confronted with something like a percussion sound and she tries to make sense of it, it comes out in a kind of beatbox sound.”
Herndon has become one of the biggest names in the area of AI and music in recent years | Photo credit: Boris Camaca
Originally from Tennessee, in the southern United States, Herndon went on a high-school exchange to Berlin and became fascinated by electronic music. She then studied the subject at university and recently completed a PhD in Music and Acoustics at Stanford University’s renowned Center for Computer Research.
Now, the 40-year-old lives in Berlin, where she worked for some years producing Proto with a small ensemble, as well as her husband and long-time collaborator Mat Dryhurst and Spawn’s designer Jules LaPlace.
While the process of working intensely on the album was a group effort, she says getting AI involved in the music-making was a natural progression.
“We really see her as part of our ensemble, an ensemble member, a performer,” Herndon explains. “We chose to use the baby metaphor (when referring to Spawn) because we see this as a kind of a nascent technology, which takes an entire community to raise, whose entire worldview is really based on the training material that we provide for her.”
The cover of Holly Herndon's latest album 'Proto' is an image made up of a mix of all the contributing artists | © Holly Herndon, 4AD
“Streaming platforms are treating us like we are algorithmically consuming music, as if we don’t really care about where something comes from or the community that makes it,” she says. “I think that’s a really important aesthetic aspect of music.”
“If we are just hoovering up our shared past archive and then we have a corporate recreation of subcultures that we can consume and never pay attribution to – to those people who come up with the ideas – then we have a really dystopian vision of the future of music.”
Holly Herndon performs with backing singers in Lisbon in November 2019 | Photo credit: Vera Marmelo
“I see machine learning as a really interesting and wonderful new human coordination technique,” she explains. “I see it along a continuum of something like early vocal traditions, which are also a human coordination technique.”
For the moment, she says, AI technology still needs considerable human contribution to make meaningful music.
“It is still something that is very human-laborious and something the humans are enjoying and finding expression and creativity through - but that art is something that is expressing our experience of our here and now.”
Holly Herndon’s Couch Lessons talk on music and AI, featuring Claire L. Evans and Ash Koosha, took place on September 30th, 2020 and is available to view on Goethe-Institut’s Youtube channel. Further talks can be viewed at the Couch Lessons website.
“Basically, I read a tonne of text and sang some passages and trained Spawn on my voice and that became Spawn’s entire universe, my voice,” she explained in a recent panel talk for Couch Lessons, a weekly series for essential conversations about artificial intelligence.
The method, which Herndon, calls “the voice model” is especially audible on the 12th track Godmother which sees Spawn trying to beatbox a composition by music producer Jlin. It’s a hectic song with blips and beeps and squawks, like a machine is trying to crawl out of a record player.
“I didn’t teach her to beatbox, none of the training data was me beatboxing in the studio,” Herndon adds, referring to Spawn as always with the female pronoun. “When she is confronted with something like a percussion sound and she tries to make sense of it, it comes out in a kind of beatbox sound.”
Herndon has become one of the biggest names in the area of AI and music in recent years | Photo credit: Boris Camaca
A long-lasting fascination
Holly Herndon’s Proto, released last year, has been her most ambitious album yet and firmly established her as one of the major names in AI music. Before it came Movement (2012) and Platform (2015), both electronic music albums but less AI-driven. The path to exploring machine learning and AI’s music creative capabilities started in her teenage years however.Originally from Tennessee, in the southern United States, Herndon went on a high-school exchange to Berlin and became fascinated by electronic music. She then studied the subject at university and recently completed a PhD in Music and Acoustics at Stanford University’s renowned Center for Computer Research.
Now, the 40-year-old lives in Berlin, where she worked for some years producing Proto with a small ensemble, as well as her husband and long-time collaborator Mat Dryhurst and Spawn’s designer Jules LaPlace.
While the process of working intensely on the album was a group effort, she says getting AI involved in the music-making was a natural progression.
“We really see her as part of our ensemble, an ensemble member, a performer,” Herndon explains. “We chose to use the baby metaphor (when referring to Spawn) because we see this as a kind of a nascent technology, which takes an entire community to raise, whose entire worldview is really based on the training material that we provide for her.”
The cover of Holly Herndon's latest album 'Proto' is an image made up of a mix of all the contributing artists | © Holly Herndon, 4AD
The good, the bad, the ugly of music tech
While Herndon admits now is an exciting time to be an AI musician because “what maybe mattered 50 years – or even 10 years ago – is dramatically changing,” she also mentions her concern about the copyright and reimbursement issues associated with using data to create new AI-produced music. In the making of the album Proto, Herndon personally made sure that Spawn’s data was not compromised for instance. The neural network built inside the humble computer tower she used was fed musical data only of Herndon singing, her ensemble and occasionally larger choir groups that were brought in just for the purpose of training the system. It’s all part of Herndon’s broader belief of valuing the context of a piece of music, she says.“Streaming platforms are treating us like we are algorithmically consuming music, as if we don’t really care about where something comes from or the community that makes it,” she says. “I think that’s a really important aesthetic aspect of music.”
“If we are just hoovering up our shared past archive and then we have a corporate recreation of subcultures that we can consume and never pay attribution to – to those people who come up with the ideas – then we have a really dystopian vision of the future of music.”
Holly Herndon performs with backing singers in Lisbon in November 2019 | Photo credit: Vera Marmelo
Don’t lose hope
Herndon’s hope for the future is instead that technology, and specifically AI, won’t be dehumanising or overwhelming for musicians and music lovers alike. She also says that she doesn’t want her work with Spawn to be the start of humans being “automated off stage.”“I see machine learning as a really interesting and wonderful new human coordination technique,” she explains. “I see it along a continuum of something like early vocal traditions, which are also a human coordination technique.”
For the moment, she says, AI technology still needs considerable human contribution to make meaningful music.
“It is still something that is very human-laborious and something the humans are enjoying and finding expression and creativity through - but that art is something that is expressing our experience of our here and now.”
Holly Herndon’s Couch Lessons talk on music and AI, featuring Claire L. Evans and Ash Koosha, took place on September 30th, 2020 and is available to view on Goethe-Institut’s Youtube channel. Further talks can be viewed at the Couch Lessons website.