The Human Theramin is an interactive audio experience in which participants place sounds in 3D space, then explore those sonic landscapes. Our goal was to explore the sensation of painting with sound. Accomplishing that experience required a great deal of technical hacking.
The user wears headphones and holds two Wiimotes. A Kinect tracks the location of their head and hands. When the user actuates the Wiimote, a virtual sound emitter is created at that location in 3D space. The effect is enhanced using binaural audio. An OpenFrameworks app compares the position and orientation of the user's head to the position of each sound, tuning them appropriately in the headphones via a custom MAX patch.
The sounds themselves are created using granular synthesis, which transforms stock samples (e.g. refrigerator groan, toilet flush, car crash, motorcycle engine) into unique sounds for each button press. The controller's roll, pitch, and acceleration additionally affects the sounds' LFO, distortion, and pitch bend, granting a high degree of creative control.
A key computational challenge was in connecting OpenNI to MaxMSP. The patch used a head-centered “interaural-polar” coordinate system, which requires we convert the position data from the world coordinate system given by the Kinect. The system is described in detail by the CIPIC Interface Laboratory, which supplied the HRTF data for the binaural patch. The azimuth, elevation, and distance of the hands are relative to the user’s head, which is assumed to be level in the XZ-plane and facing in the same direction as the user’s shoulders. This computation allows users to wander around a large area, facing any direction, and hear the sounds as if they were suspended in the virtual space.