WaveNet: A Generative Model for Raw Audio

some sense a compromise: a few years ago people were obsessed with the idea of forcing RNNs/LSTMs to summarize the inputs they've seen to date and learning long-range dependencies through a hidden layer that would hopefully be interpretable. Mostly, though, models seem to be very happy staring at the last few inputs...recent paper showed they basically seem to mostly work as n-gram models with relatively small n [0, 1]. The compromise is Wavenet, which can only act on a context of about ~300 ms at a time [2], which more or less precludes learning long-range structure, but doubles down on this inferential bias and runs this tiny audio context through many layers and tens of millions parameters to outdo state-of-the-art models that need to "lose information" as they process it. To your point, I would argue that most real-world applications are more interested in "global" interactions and an ability to "understand" signals rather than expending tremendous resources on every tiny detail that is observed. I'm not sure that this is the typical solution neural networks are going to converge to. Partly I think this is motivated by hardware: GPUs are unbelievably powerful computing machines and they make convolutions look unbelievably attractive. Some researchers have 8 or more of them so you don't have to worry about obsession over detail. The other part of the tiramisu models, DenseNets, basically glue together layer after deep layer after deep layer... I think it's an architecture that is an obvious idea but from my understanding of GPUs, layer concatentation is an expensive operation, and people wouldn't have bothered designing the architecture a few years ago because they wouldn't have been able to run it on anything other than a Titan X from the future. Probabilistic models have been floated as a way to increase global coherence of the information models are learning. In my experience, however, when trying to combine models with probabilistic and convolutional components (like [3]) the neural network's first order optimization promotes obsession over details vs understanding data well enough to be able to handle any uncertainty. To some degree I think this is also what we see in the deep learning community: why do second-order optimization and move toward new paradigms when you have 8 GPUs and can get a 0.1% improvement on the latest image benchmark? [0] Blog post on "Frustratingly Short Attention": https://martiansideofthemoon.github.io/2017/06/28/short-atte... [1] From https://arxiv.org/pdf/1703.08864.pdf : "It shows that the memory acquired by complex LSTM models on language tasks does correlate strongly with simple weighted bags-of-words. This demystifies the abilities of the LSTM model to a degree: while some authors have suggested that the LSTM understands the language and even the thoughts being expressed in sentences (Choudhury, 2015), it is arguable whether this could be said about a model that performs equally well and is based on representations that are essentially equivalent to a bag of words." [2] https://arxiv.org/pdf/1609.03499.pdf [3] "PixelVAE": https://arxiv.org/pdf/1611.05013.pdf

problem. This is partly why there are so many different submodules and parts in the text piece of DeepVoice, and a whole appendix dedicated to that in WaveNet. It is still surprising to me how good the audio models from WaveNet and DeepVoice are (such as the sample you show) - it seems that given good enough text features e.g. groundtruth the synthesis is nearly perfect. So it seems (IMO) that the next research papers will be focused on the text/f0 part. I will also plug the work we have been doing at MILA, which is related but tries to directly go from text -> speech with attention based RNNs [0]. A longer version of our paper should be on arxiv soon, but for now we have a short teaser which was submitted as an ICLR workshop [1]. One fun feature is that we have the ability to handle multi-speaker synthesis, and multi-speaker datasets as well. The primary differences from a high level are: we don't need a pronunciation dictionary for training, but they (DeepVoice or WaveNet) do. However we need a vocoder (currently WORLD) to get intermediate targets for training, while they do not. For training we need a directory of text files, and a directory of audio files, along with a script to run WORLD on the audio. This means char2wav can go to new languages without extra lexical information (e.g. pronunciation dictionaries), since the WORLD assumptions are broad, but we find we have a difficult time on English without using phonemes (from a pronunciation dict), since the character -> audio mapping is pretty difficult for English. We have better outputs (compared to char2wav English) on Spanish, German, and Romanian [2] though we are still improving every day. Note that the Romanian is still using fixed synthesis with WORLD - this is a demo only of the text -> WORLD features part of the model, the "reader". The end result of both char2wav and DeepVoice (be able to go from text -> speech directly in generation) is the same. WaveNet also has this - you can see most of the TTS details in the appendix of the paper [3]. There is also a good discussion on reddit with one of the DeepVoice authors that has quite a bit of detail about their approach, I found it quite helpful [4]. Heiga Zen has a great overview talk on youtube for people who are interested in TTS [5]. There are also some interesting extensions on WaveNet [6][7], though I have not translated enough to determine what exact methods are being used. [0] http://www.josesotelo.com/speechsynthesis/ [1] https://openreview.net/pdf?id=B1VWyySKx [2] https://www.youtube.com/watch?v=cwnDjq33uMs [3] https://arxiv.org/abs/1609.03499 [4] https://www.reddit.com/r/MachineLearning/comments/5wosbm/r_d... [5] https://www.youtube.com/watch?v=nsrSrYtKkT8 [6] https://twitter.com/ballforest/status/838759080621568002 [7] https://twitter.com/ballforest/status/836803448435789828