Inhabiting Neural Activity: A Closed-Loop VR System for Generative Sonification and Visualization of Spike Data
Source of Publication
Proceedings of the 31st International Symposium on Electronic/Emerging Art (ISEA2026)
Conference Year
2026
Abstract
This paper introduces an end-to-end, closed-loop pipeline that transforms neural action potentials from extracellular electrophysiology recordings into an interactive, stereoscopic VR experience. Spike events, stored in Neurodata Without Borders (NWB) files, are converted into MIDI using the Neuro-MIDI Python package. In Max/MSP, these MIDI events drive a poly-timbral, semi-generative audio engine where the data’s own structure dictates the performance. The system interprets MIDI events in parallel streams: generating harmonic chords from single notes, routing different notes to distinct VST instruments, and employing a meta-control structure where a specific range of notes (MIDI 60-69) triggers its own dedicated VST sound layer while simultaneously altering the main playback tempo in a feedback loop. Probabilistic gating and real-time mixing contribute to a variable, non-identical performance. Concurrently, a generative particle system in TouchDesigner creates an evolving visual field. The system features a bi-directional feedback loop where 3D particle coordinates control the binaural spatialization of audio layers, while audio analysis modulates the particle system’s physics. The experience is rendered for a Meta Quest 3 headset, with camera control driven by head-tracking and VR controllers.
Document Type
Conference Proceeding
Publication Date
11-6-2026
Recommended Citation
Uhm, SiHyun and Saunders, Jonny L., "Inhabiting Neural Activity: A Closed-Loop VR System for Generative Sonification and Visualization of Spike Data" (2026). ISEA 2026 Proceedings. 42.
https://zuscholars.zu.ac.ae/isea2026/42