Inhabiting Neural Activity: A Closed-Loop VR System for Generative Sonification and Visualization of Spike Data

Source of Publication

Proceedings of the 31st International Symposium on Electronic/Emerging Art (ISEA2026)

Conference Year

2026

Abstract

This paper introduces an end-to-end, closed-loop pipeline that transforms neural action potentials from extracellular electrophysiology recordings into an interactive, stereoscopic VR experience. Spike events, stored in Neurodata Without Borders (NWB) files, are converted into MIDI using the Neuro-MIDI Python package. In Max/MSP, these MIDI events drive a poly-timbral, semi-generative audio engine where the data’s own structure dictates the performance. The system interprets MIDI events in parallel streams: generating harmonic chords from single notes, routing different notes to distinct VST instruments, and employing a meta-control structure where a specific range of notes (MIDI 60-69) triggers its own dedicated VST sound layer while simultaneously altering the main playback tempo in a feedback loop. Probabilistic gating and real-time mixing contribute to a variable, non-identical performance. Concurrently, a generative particle system in TouchDesigner creates an evolving visual field. The system features a bi-directional feedback loop where 3D particle coordinates control the binaural spatialization of audio layers, while audio analysis modulates the particle system’s physics. The experience is rendered for a Meta Quest 3 headset, with camera control driven by head-tracking and VR controllers.

Document Type

Conference Proceeding

Publication Date

11-6-2026

This document is currently not available here.

Share

COinS