A Universal Music Translation Network
📜 Abstract
We present a method for translating music across musical instruments, genres, and styles. This method is based on a multi-domain wavenet autoencoder, with a shared encoder and a disentangled latent space that is trained end-to-end on waveforms. Employing a diverse training dataset and large net capacity, the domain-independent encoder allows us to translate even from musical domains that were not seen during training. The method is unsupervised and does not rely on supervision in the form of matched samples between domains or musical transcriptions. We evaluate our method on NSynth, as well as on a dataset collected from professional musicians, and achieve convincing translations, even when translating from whistling, potentially enabling the creation of instrumental music by untrained humans.
✨ Summary
Paper summary
The paper introduces an unsupervised neural architecture for translating audio between musical instruments, genres, and styles without parallel recordings or musical transcriptions. It uses one shared WaveNet-like convolutional encoder and multiple autoregressive WaveNet decoders, with one decoder associated with each output musical domain. The encoder maps inputs into a domain-independent latent representation, while the selected decoder reconstructs the audio in the desired target domain.
Two mechanisms are central to the method. First, locally perturbing the pitch of randomly selected portions of one-second training segments discourages the encoder from memorizing waveform details and encourages higher-level musical representations. Second, a domain-confusion classifier is trained adversarially against the encoder so that the latent representation contains less information about the source domain. During inference, the input is passed through the shared encoder and directly into the decoder for the selected target domain.
The model was trained on six classical-music domains, including orchestral, string-quartet, organ, piano, harpsichord, and vocal-orchestral recordings. Evaluation compared translations into piano against conversions made by three professional musicians. Human evaluators rated the system below the musicians in audio quality and translation success, but automatic pitch-based measures showed strong preservation of musical content and timing. Additional experiments indicated that the encoder could process previously unseen domains, including swing jazz, metal guitar, instrumental Chinese music, NSynth instruments, and whistling. The learned embeddings also preserved pitch information across instruments, and interpolation between embeddings produced reportedly seamless transitions.
The main technical contribution is the combination of a universal encoder, domain-specific autoregressive decoders, domain-confusion training, and pitch-based augmentation for cross-domain raw-audio translation. Important limitations include relatively low generated-audio quality, the computational cost of autoregressive WaveNet inference, dependence on 8-bit μ-law audio quantization, and the inability of the ablation models to train successfully without augmentation or domain confusion.
Subsequent research and industry uptake
The work was published as an ICLR 2019 conference paper, establishing it as a peer-reviewed contribution rather than only an arXiv report. (openreview.net) Facebook Research subsequently released an implementation, pretrained models, training instructions, inference tools, and audio samples, providing a concrete software artifact for reproduction and experimentation. (github.com)
A quick search found evidence of continued use primarily as a reference implementation and research baseline: later academic work and project reports cite or reimplement the method for music-style and instrument translation, including an extension targeting Indian instruments. (github.com) The available evidence does not establish widespread commercial deployment or a distinct industrial product directly derived from this paper; its clearest documented impact is methodological and educational, through reuse of its architecture and open-source implementation. (github.com)