Concept Ideation and Process
After extensive research, the team responsible for the sounds decided to use TouchDesigner, a popular software for creating interactive multimedia content, as the host for sound interaction. TouchDesigner provides hundreds of operators for data processing and networking, enhances projects through Python scripts, and syncs real-time effects and video elements to music, making it a versatile and powerful tool. The Sound Design subteam designed a prototype to test the feasibility of the program by demonstrating its capability to interact with sound elements such as volume, speed, and pitch.
There were three important considerations in ensuring the sound would be successful:
- Realtime, continuous message relayed via UDP (User Datagram Protocol) from the Sensor Processing subteam
- How the message should be used to manipulate a multi-layered composition from MIDI components
- Finalization of the composition, the software to manipulate the composition, and the software that receives the information sent via UDP
User Datagram Protocol and MIDI files
TouchDesigner was being used to receive a UDP message from a dummy application sending mock sensor data in a format determined by the Sensor Processing team. To manipulate the audio and MIDI files, Ableton Live was used with a TouchDesigner plugin to utilize parsed data from the UDP input to change audio and MIDI outputs in Ableton Live. However, with more experimentation, the subteam realized that TouchDesigner would not be able to receive real-time information from a UDP sender. Thus, Cycling 74’s Max 8 was a better solution for receiving UDP messages—the subteam was able to successfully receive real-time data via UDP with Max 8. The MIDI and audio layers were controlled from parsed information retrieved from the sent UDP message.
To test their solution, the Sound Design subteam set up a webcam and integrated the Python program for launching MediaPipe made by the Sensor Processing subteam that was supposed to run in conjunction with Sound Design subteam’s. This program connected a UDP receiver, set to listen to a port on the computer’s localhost IP address, to a JavaScript program that received and translated the message from Unicode to a string. From there, it parsed the string for the appropriate value, tested that value against a threshold number, and outputted a boolean to start or stop the audio and MIDI files depending on the result of the test output, which was opened and added to the program. This test worked with a participant waving a right hand in the air, which was the motion resulting in values passing the threshold number. With this successful test, the following technical specifications have been determined: to allow a participant to produce or stop sound by raising or lowering a right hand above the shoulder, 3 the pipeline_patch.maxpat program must be run in Max 8 along with the MediaPipe, UDP sender program as provided by the SP sub-team on a computer with a Microsoft Windows operating environment with a webcam. Once the pipeline_patch.maxpat program is opened, audio files and MIDI files must be opened inside the program using the labeled objects. The audio and MIDI files must be saved in the same location as the Maxpat program to be accessible. Locking the .maxpat file using the locking button in the lower left-hand corner will allow the program to run. With the MediaPipe python script and the .maxpat files running, functionality will be achieved.
Finalizing the Composition
The composition should be layered to allow for different compositional stems to be manipulated individually. The subteam chose to split the sounds into high-register and low-register tones—similar to an orchestra in which violins and woodwinds would usually carry the melody while bass, low brass, and drums would create and round out the background music. This variation would allow for responsiveness to different environments within the exhibit. The higher tones were to be played through a directional speaker over the umbrella, while the lower tones would envelop both environments of the exhibit. Thus, under the umbrella, the participant would hear a fuller composition. To create cohesiveness in a single, strong, layered composition, composer Paul James Cornish helped overlay compositions on three different parts in MIDI files. This would solve the solution of the paucity of upper register tones that might form a melody.
Sounds were recorded on physical instruments and later edited through Logic Pro and Max 8 to emulate the “transformation to a more tangible reality”. For example, several tracks were recorded on the drums that emphasized sharp and bright hi-hat elements, while repeating arpeggios filled the violin and piano parts to create a lively, angelic effect.
Section 1
In Section 1, composer Paul James Cornish’s keyboard runs were spliced and combined to create staggered layers; this played when a participant’s hand was raised. Logic Pro’s ‘Celtic harp’ and ‘delicate bell’ instruments were used to inhabit a higher register tone. This helped create a medium theme that was abstract, yet moody.
Section 2
Section 2 featured bird sounds which included chirping noises and a flock of birds flapping their wings. When the hand height increased, more bird sounds would be layered over top of one another, increasing the bird noises to give users a feeling that a whole flock of birds was flying next to them. At the highest point of the user’s hand, the sound would play in an overhead speaker to give the illusion of birds flying over the user’s head. This section is unique in that two compositions were played, rather than just one. The full track used for Section 2 can be found below.
Zuyu Chen’s Composition
MaxMsp was the main tool used to create the composition. Max is used only for parsing incoming JSON-formatted pose data and sending midi CC messages to control the music played in Ableton Live. The MSP part was not utilized.
The three types of interaction with hand height changes are adding/subtracting instrument tracks, birdsong density increasing/decreasing in combination with the density changes of certain instrument tracks, and timbre changing. These three interactions, which result in distinct changes in the soundscape, are set up in different sections to allow one to invoke only one type of sonic change at a time. Each distinct sonic/music effect responds uniquely to the audience’s hand motion without confusing individuals when multiple people are interacting with the sound.
However, auditory perception is undermined when competing with visual and other sensory inputs. A higher volume and more straightforward sound changes would yield greater attention from the audience who enter this mysterious place and have no idea of what’s going on. Though designed to be distinct, the sonic interactions we made are not significant enough to make the installation more immersive and engaging.
Section 3
Section 3 emphasized a darker theme, so the sound was bass-heavy. It included nature sounds and a woosh, low-register noise to utilize the subwoofer capabilities while adding depth to the calm exhibit. The base increased as the hand height increased, and the inverse was true.