WaveNet: A Generative Model for Raw Audio
📜 Abstract
This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-of-the-art performance, with human listeners rating it as significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. A single WaveNet can capture the characteristics of many different speakers with equal fidelity, and can switch between them by conditioning on the speaker identity. When trained to model music, we find that it generates novel and often highly realistic musical fragments. We also show that it can be employed as a discriminative model, returning promising results for phoneme recognition.
✨ Summary
WaveNet introduced an autoregressive generative model that operates directly on raw audio waveforms. It uses causal and dilated convolutions to model long-range temporal dependencies while preserving efficient parallel computation during training. The model demonstrated strong results in speech synthesis for English and Mandarin, supported speaker conditioning for multi-voice generation, generated musical audio, and produced promising phoneme-recognition results. (arxiv.org)
Its subsequent influence is documented in both research and industry. Parallel WaveNet used probability-density distillation to convert the sequential WaveNet generator into a substantially faster feed-forward model and was deployed in Google Assistant for English and Japanese voices. (research.google) Tacotron 2 incorporated a modified WaveNet as its neural vocoder, conditioning waveform generation on predicted mel spectrograms. (arxiv.org) Google later reported that WaveNet and related WaveRNN systems were used in products including Google Assistant, Maps Navigation, Voice Search, and Cloud Text-to-Speech. (deepmind.google)