In the physical world, almost all information is transmitted through traveling waves -- why should it be any different in your neural network? Super excited to share recent work with the brilliant @mozesjacobs.bsky.social: "Traveling Waves Integrate Spatial Information Through Time" 1/14

Just as ripples in water carry information across a pond, traveling waves of activity in the brain have long been hypothesized to carry information from one region of cortex to another (Sato 2012)*; but how can a neural network actually leverage this information? * www.cell.com/neuron/fullt... 2/14

Inspired by Mark Kac’s famous question, "Can one hear the shape of a drum?" we thought: Maybe a neural network can use wave dynamics to integrate spatial information and effectively "hear" visual shapes... To test this, we tried feeding images of squares to a wave-based RNN: 3/14

We found that, in-line with theory, we could reliably predict the area of the drum analytically by looking at the fundamental frequency of oscillations of each neuron in our hidden state. But is this too simple? How much further can we take it if we add learnable parameters? 4/14

To test this, we needed a task; so we opted for semantic segmentation on large images, but crucially with neurons having very small one-step receptive fields. Thus, if we were able to decode global shape information from each neuron, it must be coming from recurrent dynamics. 5/14

We made wave dynamics flexible by adding learned damping and natural frequency encoders, allowing hidden state dynamics to adapt based on the input stimulus. On simple polygon images, we found the model learned to use these parameters to produce shape-specific wave dynamics: 6/14

Looking at the Fourier transform of the resulting neural oscillations at each point in the hidden state, we then saw that the model learned to produce different frequency spectra for each shape, meaning each neuron really was able to 'hear' which shape it was a part of! 7/14

Was this just due to using Fourier transforms for semantic readouts, or wave-biased architectures? No! The same models with LSTM dynamics and a linear readout of the hidden-state timeseries still learned waves when trying to semantically segment images of Tetris-like blocks! 8/14

We were super excited about these results—they aligned with the long-standing hypothesis that traveling waves integrate spatial information in the brain*. But does this hold any practical implications for modern machine learning? pubmed.ncbi.nlm.nih.gov/7947408 www.science.org/doi/abs/10.1... 9/14

As a first step towards the answer, we used the Tetris-like dataset and variants of MNIST to compare the semantic segmentation ability of these wave-based models (seen below) with two relevant baselines: Deep CNNs w/ large (full-image) receptive fields, and small U-Nets. 10/14

We found that wave-based models converged much more reliably than deep CNNs, and even outperformed U-Nets with similar numbers parameter when pushed to their limits. We hypothesize that this is due to the parallel processing ability that wave-dynamics confer and other CNNs lack. 11/14

Overall, we believe this is the first step of many towards creating neural networks with alternative methods of information integration, beyond those that we have currently such as network depth, bottlenecks, or all-to-all connectivity, like in Transformer self-attention. 12/14

If you want more visualizations, a bit more depth, and even some audio of what different images 'sound' like to our models, please check out our @kempnerinstitute.bsky.social blog-post! kempnerinstitute.harvard.edu/research/dee... 13/14

For all the technical details and more ablations, please see our paper recently accepted in workshop-form at ICLR Re-Align, and full-version preprint on ArXiv! Paper: arxiv.org/abs/2502.06034 Code: github.com/KempnerInsti... Hope to see you in Singapore! Fin/