List of Projects
- Score Alignment
"Score Alignment" helps you play together with machine
What is Score Alignment?
Score alignment tells you where in a piece you are playing right now. It is like a GPS on your music score: it keeps track of where you are, whether you play really fast, really slow or even if you skip a line.
In this video, a famous character Hatsune Miku is dancing along with keyboard performance.
When the performer plays fast, she dances fast, when he plays slow, she dances slow, as if she is listening to his performance. She even dances along when he makes a mistake because she can still comprehend his performance. If you look carefully, when he plays too fast she cries, and when he plays too slow she seems a little irritated, maybe it was too slow for her. As shown in this demo video, this technology opens door to a whole new kind of interaction between music performance and a machine.
How is it done?
First, you specify the music that you are about to play. Then, as you play into the microphone, our system infers what point in the music score best describes the performance data coming in from the microphone. To deal with fast playing, slow playing, pausing, skipping and so on, it combines various musical knowledge, such as how widely the tempo is expected to vary, what kind of sound is expected at each point in the music score, or how the system should coordinate the machine interaction to best fit the performance.
There are many other uses. For example, we are already developing an auto accompaniment system that allows accompaniment data to synchronize with the performance. Another possibility is a page turner for digital scores that turn the page for you at appropriate points in the score. It is also possible to make a piano fingering video that synchronizes with the user's playing.
If you are interested in this technology, please contact us.