Personal orchestra: a real-time audio/video system for interactive conducting

摘要

We present the first multimedia system to conduct a realistic electronic orchestra. Users can control tempo, dynamics, and instrument emphasis of the orchestra through natural conducting gestures with an infrared baton. Using gesture recognition and tempo adjustment algorithms, the system plays back an audio and video recording of an actual orchestra that follows the user's conducting in real time. A major achievement of this system is its ability to vary playback speed in real time while avoiding audio artifacts such as pitch changes. The system has been deployed as an exhibit and has become a major attraction of a large Vienna-based music exhibition center.