Y2 PROJECT

Audio Beat Sync

Audio Beat Sync: A beat tracking technology for live music performance

Imagine a world where machines listen and lock in to the beat played by human musicians. Imagine, for example, a stage lighting or a VJ system that synchronizes its playback how fast a drummer is playing; as the drummer gets excited and plays faster, the lighting or VJ system would automatically accelerate, or spice up the visual effect. Or, imagine having a toy humanoid robot that dances with how a child claps beats.

These kinds of coordination require machines to understand and track beats. Thus, we developed Audio Beat Sync, a real-time beat tracking technology geared towards live music performance.

A tad more detail

Audio Beat Sync detects beats of a music audio signal that is played in real-time. It is capable of tracking sounds of percussion, drums, bands, club tracks and so on. By coordinating external devices(*1) to the detected beat, machines could track and synchronize with what the human players are doing, instead of humans having to track and synchronize to machines.

Infographics-blockdiagram.png
It is difficult to track beats from audio played by humans because humans cannot play with a machine-like precision, and the beat may be produced from a variety of audio sources. To analyze the beat, Audio Beat Sync analyzes audio signal to extract features that represent note onsets and short-term tempo trends. In addition to this kind of audio analysis, the tempo and beat positions are estimated jointly through real-time statistical inference. Furthermore, a mathematical model of human perception of beat is employed to create a natural estimate of the beat. The upshot of all these is that Audio Beat Sync is capable of analyzing beats to a wide variety of audio signals, even when the beat interval wavers.

*1:For example, Open Sound Control (OSC) or MIDI clock signals may be used.

What kinds of uses are there?

There are many ways to use Audio Beat Sync, and the following list only scratches the surface:

Music Performance
- Synchronize arpeggiator and audio effects of synthesizers to external audio, such as the drum part (e.g. MONTAGE
- Synchronize sequencers and DAWs to a human drummer.
- Synchronize VJ effects or playback rate to a jam band

Content Production
- Creating a motion data that is synchronized to music (e.g. controlling a real-time dance computer graphics creation system)
- Choreographing robot motion to music
- Production of lighting, fireworks or video data that is matched to a piece of music

Healthcare / IoT
- Synchronize music playback to how fast a runner is running

Audio Beat Sync runs on a wide variety of platforms, including SH4 microcontrollers, smartphones, tablets and PCs. If you are interested, please feel free to contact us.

If you are interested in real-time music interaction, also check out Score Alignment technology, which tracks where, in a given music score, a human player is playing.