Designing & prototyping an XR creative tool for embodied musical development on Apple Vision Pro
Live coding lets performers make music by writing code in real time, but it only engages the mind and ears, not the body. We asked whether XR could change that: what happens when rhythm is visually present in the air around you, and your gestures shape the music?
Using the HCI framework of design probes, we're investigating how spatial computing can support creative flow during live composition. We built a prototype on Apple Vision Pro that connects hand gestures and spatial visuals to live-generated music, as a probe to explore how embodiment reshapes creators' sense of connection, agency and attention. We're now running user studies.
ScreenRecording_09-07-2026 11-29-16 PM_1.mov
Live coding is the practice of writing and modifying musical code in real time. It has gained popularity in recent years through events such as algoraves, where performers generate sound by typing symbolic patterns while audiences listen and watch the evolving code on a projected screen.
Although compelling, this format only allows for cognitive and auditory engagement with liveness. It lacks physical involvement.
Our team wondered whether XR, with its immersive spatial audio, precise hand tracking, and expressive 3D visuals, could bridge this gap and reimagine how music is felt.
Our key research question: how can embodied XR live-coding environments function as design probes for deepening one's connection to music? By letting performers see, touch, and move through musical structures, embodied XR transforms live coding from a symbolic interaction into a physically grounded practice.
Our project is informed by four key areas of HCI research.
| Area | What it gave us |
|---|---|
| Design probes | Objects intentionally designed to invite participant reflection, as tools for both design and exploration. We designed around four properties: thematic openness vs. boundedness (enough freedom to explore, enough constraint to direct attention), materiality (form, materials and affordances that scaffold reflection), pace (how quickly or slowly participants engage) and challenge (the degree of difficulty or provocation). |
| Cognitive load theory | A distinction between intrinsic load (the inherent complexity of musical structures) and extraneous load (unnecessary effort introduced by the interface). This gave us a lens for evaluating how XR might redistribute cognitive effort, and informed how complex the design should be. |
| The live coding movement | The principles of liveness and transparency of code, referencing figures like Alex McLean, Nick Collins, Julian Rohrhuber and Roxanne Harris. |
| Embodied interaction | How XR introduces spatial, gestural and multisensory feedback that shapes how users experience their body in immersive spaces. |