The collaboration between human intelligence and artificial intelligence allows bots to imagine and propose new forms of music
A Chinese computer scientist and professional musician is redefining the level of human-computer interaction in music creation.
Professor Gus G. Xia, Visiting Professor at Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), uses machine learning to make robots improvise and bring ideas to create new musical works. If a human drummer or pianist doesn’t feel right to participate in a performance, Professor Xia will save the day with his AI robot.
“I use interactive intelligent systems and technologies to expand human musical expression and abilities,” said Professor Xia, who is currently an assistant professor of computer science at NYU Shanghai. He received his PhD from the Department of Machine Learning at Carnegie Mellon University, USA, where he studied machine learning and computer music.
Professor Xia’s research equips the Auto Accompaniment System with machine learning, robotic embodiment, and improvisation on folk, pop, or popular songs or melodies. The existing auto accompaniment system is one where the computer is fed with a predefined score and once a person plays a piece, the system delivers a performance that complements the dynamics and tempo.
AI robot musicians to improvise
Professor Xia has been researching human-computer interaction for about 10 years.
“There are several levels of application. At the first level, the computer knows what I’m going to play. In real time, I can speed up or slow down, the algorithm follows my tempo, and delivers the complement. At the second level, when I do my expressive performance, the machine can learn from repetition to better anticipate my next move. At the third level, there is the incarnation of the robot. The robot can have its own body gestures and facial expressions based on my musical expression and timing,” said Professor Xia, Director of Music X Lab.
Professor Gus Xia interacts with gesture expressions
Currently, Professor Xia is taking it to the next level by allowing the system to improvise, merging his ideas with the musical performed by a human to produce a new composition.
“At the fourth level, which I do now, is improvisation. In this case, the robot does not know exactly what I am going to play. It is not a predefined score. It’s a pre-made lead sheet (a form of musical notation) only with a rough sketch of what I’m going to do. I can improvise and play an existing piece or something new, and based on this material the robot will improvise. He’s going to merge some of his own ideas with my music and deliver something new. There are different layers, and I build the fifth layer.
System to help, not replace, humans
Professor Xia’s research work involves testing the system on piano, saxophone and flute. It uses algorithms, which are rooted in the field of computer vision and computer graphics.
“We can use algorithms to analyze the music, to tell what genre it is, where the song is, chords, from very low level to high level.”
By creating such a robotic system, Professor Xia aims to assist humans and play with them.
“I have no intention of replacing humans with my system. But the system could be part of a group.
He is optimistic that the AI robot system can meet a demand from the public and play a melody.
Students at MBZUAI
I couldn’t afford a pianist, I created a robot
Professor Xia started learning music at the age of 3, started playing the saxophone at 10 and the Chinese flute at 12. He was the first soloist of Chinese Music Institute of Peking University, China, and performed a solo concert. in 2010. His inventions were among the feats and experiments performed during his doctoral days.
“One of the reasons I created this robot to complement me is that during my doctoral studies, I couldn’t afford a professional for my performance and I decided to invent my own. I plan to organize a thematic concert in Dubai next month. I hope the system will be part of it.
In the coming months, Professor Xia will interact with students and conduct research at MBZUAI in Abu Dhabi.
READ ALSO :