Overview
The **Gesture Music Generator & Live Spatial Audio Sandbox** is an advanced biophysical computer vision framework that transforms real-time structural coordinates of the human body into active mathematical modulations for synthesized Web Audio architectures. By processing non-invasive pixel streams directly inside the client browser, this application converts active spatial trajectories into expressive tone, routing, and envelope metrics. Rather than relying on rigid hardware controllers or pre-recorded loops, the platform creates an direct anatomical linkage between physical gestural articulation and digital signal processing networks.
Within this interactive workspace, dynamic metric parsing translates movement into acoustic parameters. By tracking key joints, the system models the biomechanics of spatial coordination. The spatial canvas is treated as a continuous, coordinate-gated field where tracking frames continuously update coordinate values for oscillator synthesis networks.
This laboratory has significant applications in clinical rehabilitation, sensory substitution research, and biophysical performance. By turning movement coordinates into immediate auditory feedback, users can interact with neural and muscular pathways in a clean closed-loop setup. Connecting visual motion signals directly to digital signal processors explores the intersection of computer vision, biomechanical coordinate systems, and real-time electronic sound synthesis.
Technical Details
This platform uses decentralized, deferred MediaPipe pipelines to calculate landmark coordinates, routing output values directly to a custom Tone.js multi-oscillator network. The visual engine tracks coordinates across the normalized bounds of the camera area, $W \times H$. To prevent false tracking spikes when hands overlap the face, the system applies a face-overlap boundary check. This check discards hand signals that fall within the estimated facial bounding box $[X_{\text{min}}, X_{\text{max}}, Y_{\text{min}}, Y_{\text{max}}]$.
Pinch distance $d$ is calculated using the Euclidean distance formula between the thumb tip ($Thumb_4$) and index finger tip ($Index_8$):
$$d = \sqrt{(x_{t} - x_{i})^2 + (y_{t} - y_{i})^2 + (z_{t} - z_{i})^2}$$
This computed displacement scales the dynamic envelope filters using an exponential mapping function, where $\tau$ represents the decay constant:
$$\tau(d) = \text{clamp}(5d, 0.1, 2.0)$$
To ensure steady coordinate tracking, the raw coordinates are processed using a low-pass exponential smoothing filter ($\alpha = 0.35$):
$$y[n] = \alpha \cdot x[n] + (1 - \alpha) \cdot y[n-1]$$
Audio outputs are routed through gain limits and brickwall limiters to protect hardware and dynamic range. To ensure smooth performance, tracking calculations run inside a optimized `requestAnimationFrame` loop. Canvas dimensions are normalized against `window.devicePixelRatio` to prevent layout shifts and keep the interactive canvas crisp on high-DPI displays.