It's Raw! Audio Generation with State-Space Models

Understand

Developing architectures suitable for modeling raw audio is a challenging problem due to the high sampling rates of audio waveforms.

• Standard sequence modeling approaches like RNNs and CNNs have previously been tailored to fit the demands of audio, but the resultant architectures make undesirable computational tradeoffs and struggle to model waveforms effectively.
• We propose SaShiMi, a new multi-scale architecture for waveform modeling built around the recently introduced S4 model for long sequence modeling.
• We identify that S4 can be unstable during autoregressive generation, and provide a simple improvement to its parameterization by drawing connections to Hurwitz matrices.