Genetic Algorithm for Generating Spectral Harmonies

 

In my work as a research assistant with the GRASSP project under Dr. Robert Pritchard, I have developed a Max/MSP-based, gesturally-controlled electronic interface that enables a performer to capture a sound and manipulate its spectral characteristics during a live performance.  The instrument employs a commercially-available glove controller, the P5 by Essential Realities, in conjunction with the computer keyboard, giving the performer intuitive controls over both continuous and discrete parameters.  By providing individual control over each of a spectrum’s significant partials, the performer can create gradual transitions between a spectrum that appears fused into a single timbre and a texture in which each of the individual partials are perceived as independent elements.

 

The philosophical motivation behind this instrument stems from ideas that I began to explore in my thesis work; both have grown from a desire to draw strong aural connections between the internal structures of sonic phenomena and the semantic structures of the composition.  While the techniques of spectral music are typically applied at the pre-compositional stages of a piece’s development, this interface extends these concepts into real-time performance.  By allowing a performer to sample a source sound, deconstruct its overtone spectrum, and manipulate its individual components in a live performance environment, the instrument aims to heighten the listener's perception of the sound's spectral characteristics, drawing audible connections between timbral features and harmonic structures.  Furthermore, the expressive gestures of the performer’s hand serve to visually reinforce the instrument’s aural effects, facilitating audience comprehension.

 

The P5 glove interface affords the performer simultaneous control over four dynamic parameters (X, Y, Z, and the hand’s “degree of openness”), which are intuitively synthesized by gestures of the hand in a “sonic space”.  The controller allows the performer to easily generate complex interactions between the four parameters with a gestural image that is intuitive and easy to visualize and recall.  Since each region of the space corresponds to a different quality of sound, the performer need only reach for the desired sonic area rather than consider how each of the individual parameters should be altered.  

 

 

This project centers on the development of an environment for interactive composition and performance, integrating Dr. Keith Hamel’s notation software NoteAbilityPro with Max/MSP-based score following and responsive sound generation.  I presented a paper (co-authored by myself and Dr. Hamel) at the ICMC ’07 conference entitled “A Score-Based Interface for Interactive Computer Music”, detailing the operation of the environment.  In a live performance, the system is able to follow the live musicians and trigger soundfiles, send MIDI messages, and control complex processes at specified score locations, regardless of the performer’s tempo fluctuations.  Extensions were made to NoteAbilityPro (forming the Integrated Interactive Music Performing Environment, or IIMPE) that allow a NA score to send control messages to 16 different IP addresses and ports during score playback, and 8 ports are available in NoteAbilityPro for receiving network messages from the connected applications.  Modules created in Max/MSP receive messages sent by NA, and allow Max/MSP to control the NA score playback.  My involvement in the project has centered around the implementation of IRCAM’s suivi.score and antescofo objects for score following.  In this configuration, a score following patch tracks the pitches played by the performer, and sends score location and tempo information to NoteAbilityPro, which in turn adjusts its playback tempo to align with the performer.  

 

I am currently developing a musical interface for electronic sound generation and manipulation, controlled in real-time by a performer's physical movements using video tracking.  This interface will be designed for performance in a live concert setting, as an independent electronic instrument that may be used alone or in ensembles with other instruments.  This Video-Tracking Electronic Instrument, or VTEI, will be developed in the graphical programming language Max/MSP/Jitter, and will require no specialized hardware beyond consumer-level video cameras for its operation.  In playing the VTEI, a performer will stand in the field of vision of two video cameras, and by moving his or her hands in particular gestures will be able to control the various operations of the system.  Images from the two cameras will be analyzed using a robust collection of computer vision algorithms (implemented in Max/MSP/Jitter by Jean-Marc Pelletier), providing the system with three-dimensional position data of the performer's hands and fingers.  The core of the VTEI will be a software interface that will parse, filter, and otherwise interpret the video-tracking data, and use this information to initiate and control a variety of sound synthesis procedures.

 

The short video below demonstrates the instrument’s general approach, using simple colour-tracking to follow two points.  The distance between the points, as well as their angle of rotation, is calculated and this data is applied to a “sonic space”; in this demo, this space consists of the sounds of crumpling paper and ankle bells, and is navigated by means of an FFT phase vocoder (left-right), spectral pitch shifter (up-down), gain control (point distance), and bandpass filter (rotation angle controls center frequency).  The final project will use two cameras for tracking in three dimensions, will be able to track all five fingers, and will offer a variety of sound production methods.