Dr. Matthias Röder, Chief Executive Officer,, Karajan Institute; Co-founder and Managing Partner, The Mindshift; Director, Beethoven X – The AI Project
Category
🤖
TechTranscript
00:00Thank you, Jeremy.
00:01Hey, everyone.
00:02200 years ago, a mildly grumpy, middle-aged man with wild hair
00:10stepped out of his apartment in Vienna,
00:13briskly walked down the street mumbling, humming,
00:17and singing from time to time, standing,
00:20writing something in his notebook,
00:22and continuing on his way.
00:25Yes, you guessed right.
00:27I'm talking about Ludwig van Beethoven
00:29at the time the most well-known composer of music in Europe,
00:34and almost completely deaf at this point.
00:39Later at home, he would take out the melodies
00:43he found during his walk and copy them
00:45into a big sketchbook that was lying on his desk.
00:49This is where the real work started,
00:52where he started to work on the musical problems
00:56that the material posed to him.
00:58How should I develop this melody?
01:01How should I combine it?
01:02Which key should it be modulating to?
01:06What is the setting and the structure
01:09that is good for this type of material?
01:12This was the actual work.
01:14And Beethoven had amassed many of such books,
01:17and he would go back and forth.
01:19This was his creative Bible.
01:22The 10th symphony that he was working on at the time,
01:25that didn't materialize.
01:27Beethoven died two years later,
01:30having left the work unfinished.
01:33195 years later, I got a phone call,
01:36not by Beethoven, but Michael Schult
01:40from Deutsche Telekom asked me,
01:41Matthias, can we finish this 10th symphony
01:44of Beethoven with AI?
01:46I don't know, Michael.
01:48We got to put a team together.
01:49We got to start working on it,
01:51and we got to see how far we can get.
01:53Well, do it.
01:56Two months later, I was pretty certain
01:59that it was possible to do something.
02:00Whether it was the big thing, I don't know.
02:03So Beethoven left 45 sketches for a 10th symphony,
02:08some of which were a little longer, others were shorter.
02:12A musicologist named Barry Cooper, a couple of years ago,
02:15used the longer ones to finish one potential version
02:19of a 10th symphony, and that worked pretty well.
02:22But with the short material,
02:24no human could ever possibly take that
02:27and turn it into something bigger.
02:29Why?
02:30Because you would naturally put your own ideas,
02:33your own style, your own way of composing into the material.
02:40I called Ahmed Elgamal from Rutgers University,
02:43who is a computer scientist specialized
02:45on creating artworks with the help of AI,
02:48and asked him if he wanted to be part of the project.
02:51He said, yes, but I've never done anything in music.
02:54So I sat down with him, taught him everything I knew
02:56about the creative process of Beethoven
02:58and how composition works,
02:59and he said, let me think about it.
03:01Called me back next week and said,
03:03Matthias, I thought about it.
03:06It's quite straightforward.
03:07Composing music is just like email auto-complete.
03:13Whoa, you can imagine that moment,
03:15I thought like, okay, well, maybe the wrong person
03:18and this is never gonna work.
03:20But then he told me, no, but look,
03:23every note is a function of all of the notes
03:26that came before.
03:27And it makes sense.
03:30When I have a piece of music,
03:31the next note is related to what came before.
03:34We can actually try it out now together.
03:36I'll sing you a melody, I'll stop at some point,
03:39and you sing the note that I left out, okay?
03:42Are you ready?
03:43Okay, da-da-da-da, da-da-da.
03:48Yeah, you see?
03:49Every note is a function of all of the notes
03:52that came before.
03:53So we set out, we said, well, let's use this technology,
03:56the transform architecture that Google invented in 2017,
04:01and create a GPT for music
04:04that's trained on late 18th century,
04:06early 19th century music,
04:08and then fine-tuned with the works of Beethoven.
04:11And then we use the sketches
04:13that Beethoven wrote down as prompts.
04:16And we let the machine continue these ideas.
04:19And then the group of musicologists
04:22combined the material, go through it,
04:25improve on it, give it back to the machine,
04:28and this way, iteratively,
04:29create one possible version of a 10th symphony.
04:32At the time, most people thought, oh, they are crazy,
04:35this is never gonna work.
04:36Even some Google folks said that
04:38to the CEO of Deutsche Telekom.
04:40But we were pretty certain that at least, you know,
04:42we would create something that would pass
04:46as a potential musical work of that time.
04:50So I wanna play you an example of that 10th symphony,
04:55which starts with one of these sketches
04:58that Beethoven had written down.
04:59It's basically one-to-one what he wrote down in his book.
05:02And at some point, the AI will take over.
05:05And I invite you to play this game with me,
05:08where you raise your hand when you think
05:10this is the moment where the AI comes in.
05:12Of course, I'm gonna give you, you know,
05:15the solution afterwards.
05:17But this is one minute of music.
05:19Close your eyes, don't get distracted
05:22by the people around you, and just be bold.
05:24Raise your hand whenever you think
05:26this is where Beethoven stops and AI begins.
05:30Was generated by this algorithm.
05:32Granted, it was then, you know,
05:35going through other algorithms as well.
05:37We not only produced an AI that could continue the sketches,
05:41we produced one that would find the suitable accompaniment,
05:44one that would stitch together pieces of AI-generated music,
05:48and then one final AI that would take the output
05:52of all of the other AIs and say,
05:54this note will be played by that instrument,
05:57this note by this instrument, an orchestration engine.
06:00This is where we really reach the end of the line,
06:04because that AI wasn't that successful.
06:06So we actually hired a human in our team
06:09to orchestrate the final version that you just heard
06:13being performed by a real orchestra with real humans.
06:18This collaboration started in 2019, as I mentioned.
06:23In 2020, we were supposed to premiere it,
06:27but COVID came along, and so we had to postpone.
06:30We had one additional year to work on it,
06:32and in that timeframe, I got a second phone call
06:34from Deutsche Telekom.
06:35Hey, Matthias, Robbie Williams has heard
06:39about your Beethoven AI, and he thinks
06:42he would like to compose with Beethoven.
06:44Is that possible?
06:45I said, well, you know, what can I say?
06:48I don't say no if Robbie Williams wants to use an AI
06:51that we produce, so we'll find a way to make it happen.
06:54Of course, the question was, how could we do that?
06:58I flew to London, I explained it to Guy Chambers,
07:01the composer that worked very closely with him,
07:03and Robbie's manager, exactly what we were doing,
07:07and they said, well, we'll think about it,
07:09and they called me back the next day.
07:11We wanna do angels, can you recompose angels?
07:15In the style of Beethoven, and I said, I don't know.
07:19I really don't know if that's possible.
07:22The problem was, you see, we had this AI
07:25that was trained on 18th and 19th century music,
07:27and that could very well deal with Beethoven's style,
07:30but how do we get Robbie Williams in there?
07:33This is when the composer on our team,
07:35Walter Wärtsow, had this amazingly genius idea.
07:38He said, well, we'll take a famous melody of Beethoven,
07:42and we combine it with bits and pieces from Robbie's angels,
07:46and this combination is what we then use as a prompt,
07:51and so by prompting the AI with something
07:53that had already the main motivic content of angels in it,
07:59it was capable of producing music
08:02that combined Robbie Williams's angel
08:05with the style of composing of Beethoven.
08:08There was one problem.
08:09We thought, okay, well, our AI's pretty good
08:12at creating long stretches of music,
08:14but Guy Chambers told us, no,
08:17you have four seconds for the intro,
08:20and you have another 30 seconds for where the solo is,
08:23and everything has to happen in there,
08:25so we would try out different types of combinations of music,
08:30and finally, we found one that worked very well.
08:33We combined the moonlight sonata with,
08:37well, some people are laughing, yeah, with angels,
08:40and I have another excerpt for you
08:43that is a little mashup of what we did with Robbie.
08:47You can listen to the whole thing on his newest album,
08:50and please, let's listen to angels with Beethoven AI.
08:56The point of this was really to collaborate,
08:59and we started out to create this one AI
09:02that could do something very, very specific,
09:06and we ended up providing a tool set
09:09to one of the great artists of our time to collaborate with,
09:13and I think this is the move that we see
09:15in a lot of what's happening
09:16in generative AI and music right now.
09:19We see tools that become collaboration partners,
09:23and that are really instances of inspiration,
09:28of transformation, of working bodies for composers,
09:33and if you ask me what the future of all of this is,
09:36then I'd say all of the tools that right now
09:39are sort of text to finished song,
09:42where you basically say I want to have a track
09:45which is 15th century style
09:48with green sleeves and heavy metal combined,
09:51and then you get a finished track.
09:52This is all nice and very good,
09:55but it will be superseded by tools
09:58that will be directly integrated into the software
10:02where we compose music,
10:03where we have our inspirational AI tools.
10:07Now, I want to leave you with another excerpt,
10:10this time from the Beethoven 10th Symphony.
10:12It's actually a work of 25 minutes,
10:15and I condensed it for you here to a couple of minutes
10:18for you to get some sense of how the piece sounds
10:21when you hear it performed by an orchestra.
10:24We can roll the tape.
10:27This was done a couple of years ago
10:30in Hamburg in the Elbphilharmonie.
10:31This is the Beethoven Orchestra conducted by Dirk Kaftan,
10:37and there's a little surprise there,
10:39an instrument that you wouldn't expect
10:41in a Beethoven symphony, so enjoy.