v0.4.0 — BSL 1.1 OPEN SOURCE

NeiroSynth

Differentiable neurophysical audio engine.
Neural networks predict physics, not audio samples.
KAN + GPU FDTD. Train on any sound. Synthesize anything.

25K+

Lines of Rust

WGSL Shaders

Python

~2x RT

M1 Realtime

View on GitHub Learn More ↓

fn adjoint_backward(&self, state_history: &[Vec<f32>]) // 12 WGSL compute shaders — zero CPU readback pub struct KanBasic { layers: Vec<KanLayer>, rbf_centers: Tensor } let stiffness_grad = reverse_time_fdtd(&adjoint_state, &laplacian); // Courant number: C² < 0.5 for stability fn spectral_loss(pred: &Tensor, target: &Tensor) -> f32 LSTEF gaussian_excitation(x, y, t, sigma, amplitude) fn morph_stiffness_maps(a: &Grid, b: &Grid, t: f32) -> Grid // The stiffness map IS the instrument fn progressive_upsample(grid_32: &Mesh) -> Mesh64 pub fn dispatch_batch_mesh(&self, pass: &mut ComputePass) // Kolmogorov-Arnold Networks — interpretable nonlinear mappings fn adjoint_backward(&self, state_history: &[Vec<f32>]) // 12 WGSL compute shaders — zero CPU readback pub struct KanBasic { layers: Vec<KanLayer>, rbf_centers: Tensor } let stiffness_grad = reverse_time_fdtd(&adjoint_state, &laplacian); // Courant number: C² < 0.5 for stability fn spectral_loss(pred: &Tensor, target: &Tensor) -> f32 LSTEF gaussian_excitation(x, y, t, sigma, amplitude) fn morph_stiffness_maps(a: &Grid, b: &Grid, t: f32) -> Grid // The stiffness map IS the instrument fn progressive_upsample(grid_32: &Mesh) -> Mesh64 pub fn dispatch_batch_mesh(&self, pass: &mut ComputePass) // Kolmogorov-Arnold Networks — interpretable nonlinear mappings

The Problem

Why NeiroSynth Exists

Current neural audio synthesis generates waveforms directly — fast but physically meaningless. Physical modeling is parametric but not differentiable — you can't train it on audio. NeiroSynth bridges this gap.

∇

Adjoint-Driven Physics

Exact gradients through the wave equation via the Adjoint State Method. Reverse-time FDTD computes ∂L/∂s with O(N) memory — no autograd graph, no backprop-through-time explosion.

KAN Parameter Prediction

Kolmogorov-Arnold Networks with Gaussian RBF basis predict 25 physical parameters. Interpretable nonlinear mappings replace black-box MLPs. Every output has physical meaning.

⊞

GPU-Native FDTD

12 WGSL compute shaders run the full 2D wave simulation on Metal/Vulkan. Zero CPU-GPU readback in the training loop. 0.01ms per dispatch on Apple M1.

∅

Zero-Allocation Inference

All GPU buffers allocated at init. Per-frame: shader dispatch + ping-pong swap. No allocations, no GC pauses, no Python overhead. Pure metal performance.

Signal Flow

Architecture

From MIDI input to physical audio output — every stage is differentiable, every parameter has physical meaning.

MIDI / Control Input

Note, velocity, pitch bend, CC → normalized input vector

KAN Encoder

5 variants: basic, complex, resonance, controller, volumetric. Gaussian RBF basis → 25 physical parameters

stiffness_map damping_map excitation_shape waveguide_params pickup_position

2D FDTD Scattering Mesh

64×64 grid, per-node stiffness + damping. CFL-stable, energy-conserving. LSTEF Gaussian excitation. 12 WGSL compute shaders on Metal/Vulkan.

batch_mesh.wgsl adjoint.wgsl fdtd_live.wgsl ion_core.wgsl

Output Pipeline

Stereo pickup → DC-block → Watermark → Audio out @ 48kHz

stereo L/R ultrasonic watermark zero-alloc

LIVE FDTD SIMULATION — 64×64 mesh

Physics Engine

The Stiffness Map
IS the Instrument

A 64×64 grid of learned material properties defines how sound propagates through a virtual body. Feed it a recording of a guitar — it learns the physical body that produces that sound. Swap maps between instruments to cross-synthesize.

u[i,j]ⁿ⁺¹ = 2u[i,j]ⁿ − u[i,j]ⁿ⁻¹
+ C² · s[i,j] · ∇²u[i,j]ⁿ
− d[i,j] · (u[i,j]ⁿ − u[i,j]ⁿ⁻¹)

C = cΔt/Δx (Courant number, CFL-stable: C² < 0.5)
s[i,j] = learned stiffness · d[i,j] = damping coefficient

Training Pipeline

Four Training Modes

From gradient-free robustness to exact adjoint gradients — choose the convergence profile that matches your target.

⟐

SPSA

M=4 Rademacher perturbation. 8 FDTD passes per step. Robust gradient-free optimization through non-differentiable GPU kernels.

∇

Adjoint

Exact gradients via reverse-time FDTD. O(N) memory for state history. Cosine LR schedule 1e-4 → 5e-6.

⇌

Hybrid

SPSA → Adjoint transition. Explore globally with SPSA, then refine with exact gradients. Best of both worlds.

▦

Progressive

32×32 mesh (40%) → bilinear upsample → 64×64 (60%). Coarse-to-fine refinement for faster convergence.

terminal — train on any .wav

# Clone
git clone --depth 1 https://github.com/Neirotunes/neirosynth-open.git
cd neirosynth

# Train on your recording
cargo run --release --bin train_replicator_sota -- your_guitar.wav

# GPU real-time stream
cargo run --release --bin stream_live

# Morph between instruments
cargo run --release --bin physics_morph

# VST3/CLAP plugin
cd neirosynth-plugin && cargo build --release
      

Benchmarks

vs. PyTorch Equivalent

Same algorithms. Different universe of performance.

Metric	NeiroSynth (Rust/WGPU)	PyTorch Equivalent
FDTD step (64×64)	0.01 ms	~0.3 ms
Training iter (SPSA, M=4)	~300 ms	~2-4 s
Memory (adjoint, 4096 steps)	~64 MB	~500 MB+
Binary size	~8 MB	~2 GB
Dependencies	7 crates	50+ packages
Cold start	<1 s	5-10 s

Audio Demos

Trained & Morphed Samples

Every sample below was synthesized by NeiroSynth — trained on real recordings, then rendered through the GPU FDTD mesh in real time.

Guitar E4 — SPSA

Trained 3000 iters on acoustic guitar. Spectral loss <5.0

Cello E3 — Adjoint

Exact gradients, cosine LR. Rich low-end convergence.

Violin G3 — SPSA

Stiffness map captures bow-string interaction.

Morph: Cello → Guitar

Continuous stiffness interpolation. No artifacts.

Morph: Bell → Violin

Cross-domain physics morphing. Metallic to organic.

Chain: 5 Instruments

Sequential morph across guitar, cello, sitar, bell, drum.

Progress

Roadmap

✓

KAN → FDTD differentiable pipeline — Full forward + backward pass through GPU physics

✓

SPSA + Adjoint + Hybrid + Progressive training — Four convergence profiles

✓

RAVE latent space loss — Pure Rust RAVE encoder via Burn framework

✓

Modal synthesis + learned body IR — Commuted synthesis pipeline

✓

Physics morphing — Continuous instrument interpolation via stiffness map blending

✓

VST3/CLAP plugin scaffold — neirosynth-plugin via nih-plug

→

Multi-resolution STFT loss — 512 + 4096 window sizes

→

PML absorbing boundaries — Replace cubic damping at mesh edges

→

Web demo — WASM + WebGPU live synthesis in the browser