Alex Graves
2013 · arXiv
This paper shows how Long Short-term Memory recurrent neural networks can be used to generate complex sequences with long-range structure, simply by predicting one data point at a time. The approach is demonstrated for text (where the data are discrete) and online handwriting (where the data are real-valued). It is then extended to handwriting synthesis by allowing the network to condition its predictions on a text sequence. The resulting system is able to generate highly realistic cursive handwriting in a wide variety of styles.
Metadata: arXiv. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
15 selected excerpts · 15 total mentions
Script fonts don’t really look like handwriting - too regular. But one of the early deep learning papers from Alex Graves does this really well with LSTMs - https://arxiv.org/abs/1308.0850 Implementation - https://www.calligrapher.ai/
A large number of handwriting synthesis demos are derivations of/inspired by Alex Graves's paper "Generating Sequences With Recurrent Neural Networks". Alex's code is available under GPL-3.0 here, https://sourceforge.net/projects/rnnl/. One could also port the techniques described in the paper, https://arxiv.org/abs/1308.0850 using any modern machine learning framework.
Looks like it's based on this paper [1]. It's a recurrent neural network trained on online handwriting data. It means you record how the position of the tip of the pen changes as you write. The training data comes from the IAM On-Line Handwriting Database which has only handwritten English text. If you want to support a complete Latin script, you would have to generate a lot of training data yourself, preferably in multiple languages. Quite the effort indeed. 1 - https://arxiv.org/pdf/1308.0850.pdf
This problem was more or less solved with neural networks in 2013, see https://arxiv.org/abs/1308.0850. I've implemented a similar model in javascript that you can interact with: https://seanvasquez.com/handwriting-generation/
This type of concatenation I first saw in Alex Graves' work on "Generating Sequences With Recurrent Neural Networks", including his unpublished TTS demo [1]. Biasing with part of another sentence (as in handwriting) can possibly improve style in TTS as well. We followed this approach in char2wav [2], but "voice cloning" has come much farther in my opinion [3][4][5]. There's a lot of relevant research on techniques for this beyond concatenating indicators or embeddings, if people are interested in the research side of this technology. [0] https://arxiv.org/abs/1308.0850 [1] https://www.youtube.com/watch?v=-yX1SYeDHbg&t=38m30s [2] http://josesotelo.com/speechsynthesis/ [3] https://twitter.com/Jeanne_Heo/status/972089715225542657 (lyrebird.ai) [4] https://google.github.io/tacotron/publications/gmvae_control... [5] https://google.github.io/tacotron/publications/speaker_adapt...
Hi, thanks for the feedback! Honestly we didn't intend to over-hype the results. We took the terms from existing works that we knew: 1) Alex Graves on Hallucination with Recurrent Neural Networks, a 2015 lecture at the University of Oxford from a course by Nando de Freitas (highly recommended). http://www.creativeai.net/posts/kp4bTG993JTQcqy2d/alex-grave... 2) Generating Sequences With Recurrent Neural Networks https://arxiv.org/abs/1308.0850 "Assuming the predictions are probabilistic, novel sequences can be generated from a trained network by iteratively sampling from the network’s output distribution, then feeding in the sample as input at the next step. In other words…
by making the network treat its inventions as if they were real, much like a person dreaming." There are other terms, such as Imagination, also used in the literature: 3) Imagination-Augmented Agents for Deep Reinforcement Learning https://arxiv.org/abs/1707.06203 4) Uncertainty-driven Imagination for Continuous Deep Reinforcement Learning http://proceedings.mlr.press/v78/kalweit17a/kalweit17a.pdf In our work, the procedure is closer to the approaches in (1) and (2), rather than the "Imagination" approach in (3) and (4) where there are more subtle differences (i.e. planning), so we followed the terms in (1) and (2).
As far as I can tell LSTM's are still using a combination of logistic and tanh transfer functions, at least through 2014 and some of the most impressive deep learning results have been obtained with these. For example: http://arxiv.org/abs/1308.0850
Man, the RNN-generated Wikipedia data around page 14 of http://arxiv.org/pdf/1308.0850v5.pdf is fascinating. Some things you'd expect from a modern language model--like, your phone keyboard's predictor already generates some grammatical-looking phrases, and there's only room to do so much in a phone. I'm more impressed that (as the author notes) it gets syntax right, including matched pairs over long distances, occasionally makes words up using plausible components, suffixes, etc. ("quanting"), and learns other quirks (e.g. there's Cyrillic after the [[ru:]] link and some Hangul after the [[ko:]]). Those aren't intuitively things you'd expect from applying arithmetic to a bunch of characters.
It's been done before. Alex Graves achieved 1.33 bits per character using stacked LSTMs.[1] The current Hutter Prize record is 1.28 bpc. Note, however, that the 1.33 bpc value was calculated on a held-out validation set and doesn't include the network weights. [1] http://arxiv.org/abs/1308.0850
I've been trying to read a paper a day since midsummer. These are a few of the interesting, for me personally, since then: Generating Sequences With Recurrent Neural Networks - http://arxiv.org/abs/1308.0850 Older one, but important to understand deeply since other recent ideas have come from this! Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks - http://arxiv.org/abs/1511.06434 Unitary Evolution Recurrent Neural Networks - http://arxiv.org/abs/1511.06464 State of the Art Control of Atari Games Using Shallow Reinforcement Learning - http://arxiv.org/abs/1512.01563 Interesting discussion in section 6.1 on the…
shortcomings/issues of DQN done by Deepmind Spectral Representations for Convolutional Neural Networks - http://arxiv.org/abs/1506.03767 Deep Residual Learning for Image Recognition - http://arxiv.org/abs/1512.03385 Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) - http://arxiv.org/abs/1511.07289 I wish they did more comparisons between similar network architecture with only the units swapped out. Eg. Alexnet, Relu vs Alexnet, Elu. On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models - http://arxiv.org/abs/1511.09249 Just a few from my list :)
Generating Sequences With Recurrent Neural Networks http://arxiv.org/abs/1308.0850 Generating Text with Recurrent Neural Networks http://www.cs.utoronto.ca/~ilya/pubs/2011/LANG-RNN.pdf Bitcoin whitepaper https://bitcoin.org/bitcoin.pdf Ethereum paper http://gavwood.com/Paper.pdf
There is an older paper [0] and demo from [1] Alex Graves that inspired a ton of work around handwriting, and then speech. Previous work from Jose Sotelo et. al. (including me) called char2wav [2] is a close neighbor to Graves' approach, though he (Graves) never published the approach for speech so we don't really know. Google's recent Tacotron paper [3] is also a relative to these approaches. WaveNet certainly changed the game in many ways, but approaches to TTS using RNNs have different roots. WaveNet and friends (incl. DeepVoice and NPSS linked elsewhere in this thread) are largely focused on audio modeling, and generally use something closely related to the "classic"…
TTS pipeline for text in the frontend. The audio modeling results are stellar, and really blew me away personally - basically changing my perspective on what is possible in audio modeling overnight. RNN models try to tackle the whole problem (text + audio modeling) at once, though currently (all?) RNN and attention style models need intermediate / high level hints or pretraining from things like vocoder representations or spectrograms, versus WaveNet's approach using the waveform directly. So they are complimentary in many ways, and I am sure we will see people trying to combine them soon - char2wav has this flavor by using SampleRNN, our lab's take on raw waveform generation though we are still working on the fully end-to-end from scratch training, the inference path is truly end-to-end. Though there are still many details to work out as far as output quality, it seems possible that this will be a productive approach (though I am quite biased). We see similar directions in neural machine translation (NMT) moving from word level representations to word parts or characters directly - one of the big reasons deep learning has come so far, so fast is that a lot of techniques from other subfields can be utilized for new domains, and I think there is a lot more fertile ground for crossover in both directions. Heiga Zen has a great overview talk about how speech synthesis, as a field, overlaps between different approaches and factorizations [4]. His work on parametric synthesis and TTS generally has laid the foundation for a lot of recent advances, and he was also a co-author on WaveNet! [0] https://arxiv.org/abs/1308.0850 [1] https://www.youtube.com/watch?v=-yX1SYeDHbg&t=38m0s [2] http://josesotelo.com/speechsynthesis/ [3] https://google.github.io/tacotron/ [4] https://www.youtube.com/watch?v=nsrSrYtKkT8
It is a dense read, but you might have a look at [1]. This is how attention is implemented in Theano. Basically the key is going "3D" per timestep (where 2D per timestep is the norm when doing minibatch training), then taking a weighted sum over the correct axis to get the right size to combine with the RNN state. Short summary: I: input length M: minibatch size (same for input and output) H_in: input hidden size (arbitrary/user selected) H_out: output hidden size (arbitrary/user selected) C: attention feature size (arbitrary/user selected) Looking at the decode/generator RNN, "context" comes in at every timestep as every hidden state from the BiRNN (I, M, H_in) projected…
to (I, M, C). We do the normal RNN thing (tanh, LSTM, GRU) for the generator at first to get a decode hidden state (M, H_out). Next the "first" output hidden state gets projected to the attention size C (so now (M, C)), and using numpy broadcasting this (1, M, C) size thing gets summed with the (I, M, C) size "context" which is a projection of the input RNN hiddens. Now we have something that is looking at both what the output RNN has previously done (and seen), and some context information from the input RNN. A nonlinearity (tanh) is applied to the (I, M, C) sized piece, partly in order to bound the activation to (-1, 1) and partly just because we like nonlinearity. This (I, M, C) size thing then gets projected to (I, M, 1), or "alpha", then the useless dimension is dropped via reshape so we now have (I, M), then a softmax is applied over the first dimension I. This means that the network has effectively "weighted" each of the I timesteps of the input, though at the start of training it can't know what is relevant, at the end of training this is the thing you visualize. This (I, M) is the actual "attention" - to apply it you simply take the original hidden states from the input BiRNN and use broadcasting to multiply again. (I, M, 1) * (I, M, H_in) gives a weighting over every timestep of the input, and finally we sum over the 0 axis (I) to get the final "context" of just (M, H_in) size. This can then be projected to a new size (H_out) and combined with the output hiddens to get an RNN that has "seen" a weighted sum of the input, so it can generate conditioned on the whole input sequence. Note that this whole messy procedure I described happens per timesetep - so the neural network (given enough data) can learn what to look at in the whole input in order to generate the best output. Other attentions such as [2],[3] are constrained either to only move forward or to be a hybrid of forward movement and (pooled) global lookup. [4] is a good summary of the different possibilities in use today. [1] https://github.com/kyunghyuncho/dl4mt-material/blob/master/s... [2] http://arxiv.org/abs/1308.0850 [3] http://arxiv.org/abs/1508.04395 [4] PPT, http://www.thespermwhale.com/jaseweston/ram/slides/session2/...
The literature is mostly image based, but as other users have said voice, text, and video have also seen substantial gains. It has also been used to great effect for taxi destination prediction (heterogenous information) [1], drug discovery [2], and general operations on graphs [3]. There are also lots of examples beyond classification - structured prediction such as image segmentation [4], predicting depth, surface normals, and pixels from only RGB [5], text to text translation [6], image captioning [7], Turing machines [8], speech synthesis [9], question answering (Q/A) [10][11], handwriting generation [12], playing Go [13], playing Atari [14], speech recognition [15],…
drawing things [16] and many more. [1] http://blog.kaggle.com/2015/07/27/taxi-trajectory-winners-in... [2] http://blog.kaggle.com/2012/11/01/deep-learning-how-i-did-it... [3] http://arxiv.org/abs/1403.6652 [4] http://arxiv.org/abs/1502.03240 [5] http://arxiv.org/abs/1411.4734 [6] http://arxiv.org/abs/1409.0473 [7] http://arxiv.org/abs/1502.03044 [8] http://arxiv.org/abs/1410.5401 [9] http://arxiv.org/abs/1506.02216 [10] http://arxiv.org/abs/1507.04808 [11] http://arxiv.org/abs/1503.08895 [12] http://arxiv.org/abs/1308.0850 [13] http://arxiv.org/abs/1412.3409 [14] https://www.cs.toronto.edu/~vmnih/docs/dqn.pdf [15] http://www.sciencedirect.com/science/article/pii/S0893608014... [16] https://www.youtube.com/watch?v=Zt-7MI9eKEo
If you are into this kind of stuff (time series density/probability modeling), there are several places to go beyond this. The simplest example is the RNN-RBM (restricted boltzmann machine) from the Theano deep learning tutorials [1]. There are follow-up papers by Kratarth Goel extending this to LSTM-DBN, and RNN-NADE (neural autoregressive density estimator) has also been explored in this space. The results for MIDI are pretty good. Ales Graves [2] has done a huge amount of work in sequence modeling for classification and generation. His work with LSTM-GMM for online handwriting modeling can also be applied to speech features (vocoded speech), and seems to work well. No…
official publication to date on the speech experiments themselves, but the procedure is well documented in his paper [3] The most recent thing is our paper "A Recurrent Latent Variable Model for Sequential Data" [4]. It operates on raw timeseries (!) even though all intuition would say that an RNN shouldn't handle this well. Adding a recurrent prior seems to preserve style in the handwriting generations at least - I may try on this "Let It Go!" task and see what happens! We are also exploring conditional controls and higher-level speech features, which should result in samples as good or better than LSTM-GMM. [1] http://deeplearning.net/tutorial/rnnrbm.html#rnnrbm [2] https://www.youtube.com/watch?v=-yX1SYeDHbg#t=36m50s [3] http://arxiv.org/abs/1308.0850 [4] http://arxiv.org/abs/1506.02216
More from the same topics.
Tom B. Brown, Benjamin Mann, Nick Ryder +28
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Chiyuan Zhang, Samy Bengio, Moritz Hardt +2
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski +10
We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands.
David Manheim, Scott Garrabrant
There are several distinct failure modes for overoptimization of systems on the basis of metrics.