Raise Your Hand: Visual Projections
The exhibition featured five distinct rear projections (the projector was behind the screen), one for each of Sections 1 and 3, and three in Section 2. The projections were themed around a mystical forest with interactive animated responses according to participants’ hand motions. The outer two sections, Sections 1 and 3, were intended to be dark and mystical, while the center section, Section 2, was intended to be a bright and realistic scene – a sort of refuge defined by a pool of light.
Section Projections
Section 1
Section 1 consists of an abstract scene, embellished with randomly appearing animated swirling floral sprites appearing on a cool-toned background to convey an aimless and eerie ambiance. The sprites appear at random times on the screen and dynamically respond to the participants’ hand motions, becoming larger, brighter, and more frequent as the hand is raised.
Section 2
Section 2 provided two different perspectives in the same 3D Unity scene of a flock of birds flying in a realistic forest. There was the “ant’s view” looking-up perspective created by two projectors on each side of an overhead domed screen. The other perspective was a “horizon view” created by a projector onto a wall screen. The domed overhead screen used the cloth from a wedding umbrella held open by clear nylon spokes suspended from the ceiling. Together these different perspectives created an immersive visual storytelling experience.
The source of the flock was unseen and behind the participant. The flock would fly overhead and towards the horizon. As the participant’s hand was raised, the number and density of birds was increased. Simultaneously, corresponding bird sounds would multiply, creating a multi-sensory experience for the participants.
Section 3
Section 3 was a low-poly “unreal” forest, set in a night-time, moon-lit scene with a rocky terrain, dispersed sporadically with luminous mushrooms matching the physical mechatronic ones in front of the projection. The trees were placed in a regular grid, like pine trees planted along highways in rural Georgia.
As the participant’s hand was raised, both the digital and mechatronic mushrooms would begin glowing and spinning, brighter and faster in proportion to hand height. Furthermore, the interactive panoramic scene added an additional element of realism by shifting the perspective on the scene from side to side, according to the degree of the participants’ torso tilt, that is, their lean to the left or right. As the participant leaned, the participant had fleeting glimpses through different corridors of trees, similar to the sensation of driving by a moonlit planted pine forest in Georgia.
Projection Creation
To create each interactive projection, the Visual Arts and Design team (VAD) utilized C# programming language within the Unity game engine—an industry-standard platform. The process begins with receiving the Ethernet UDP packets from sensors and computers, and then a C# program in Unity parses through the incoming packets to assimilate the data within Unity environment.
After collecting the Normalized Hand Height (NHH) from the parsed packets, the Unity program would create a responding video projection to react to users’ movements. Additionally, VAD employed MadMapper, a software designed specifically for mapping projections, to seamlessly align scenes with their corresponding projections and the umbrella dome in Section 2. Finally, the resulting video stream is projected onto the walls by a BenQ Short Throw projectors.
Challenges
The Visual Arts and Design team (VAD) encountered various challenges throughout the projections’ developments, primarily centered around the synchronization of projections to properly respond to user movements using the UDP receiver and integrating the programs with each other. In Section 1, VAD had trouble ensuring and timing the randomization of the floral sprites’ appearances. In Section 3, Unity3D faced difficulties in handling multiple inputs and camera motion concurrently since Section 3 uses both hand height and torso tilt to change the scene. Furthermore, the presence of multiple public variables from the data received meant that numerous entities across programs modified them in order to generate real-time visual interactions.
This concurrent access to shared variables introduced the risk of data race conditions where the simultaneous attempts to modify the same variable could result in data loss and potential Unity failure entirely. To combat this, VAD implemented a thread-locking mechanism to ensure only one entity can access a variable at time to prevent conflicts and maintain orderly control over data access.