Deep Learning in Neural Networks: An Overview

> but it would be impractical to make it book-sized Not really: History: https://arxiv.org/abs/2212.11279 (75 pp.) Survey: https://arxiv.org/abs/1404.7828 (88 pp.) Conveniently skim-read over the course of the four weekends on one month.

Ah good question. I think I have read that statement by some other people. But the limit is kind of arbitrary. And of course, this limit will be higher and higher over time, that's why I put the year. I think the limit should also not be much lower. We already have language models three order of magnitude larger (>1T params), and we also call them "large", so in this context, all those single-digit billion parameter models feel quite small. Similarly, when is a network "deep"? It used to mean more than 2 or 3 layers. And then there was a definition for "very deep", starting with more than 10 layers (I think Schmidhuber introduced that definition many years ago, https://arxiv.org/abs/1404.7828). Obviously, that's totally outdated now. Networks are often very deep, e.g. those large language models often 96 layers.

I mean it's self-promotional yes, but I read this as more of a blog post about advances specifically in his own group. For the Schmidhuberian take on the broader history of deep learning, this other one's the go-to article (though it's much longer): https://arxiv.org/abs/1404.7828 Not everyone likes that article either, but it does at least extensively cite prior work, i.e. accounts for "how his contributions were (like everyone else's) evolutions of existing methods". In particular, sections 5.1–5.4 credit a large amount of work from the 1960s-80s that he considers foundational.

If you want a more nuanced research on the history on deep learning in neural networks, here is an excellent historical survey paper: https://arxiv.org/abs/1404.7828

For more details, refer to Schmidhuber's own website: http://people.idsia.ch/~juergen/ and his excellent review paper, "Deep Learning in Neural Networks: An Overview": https://arxiv.org/pdf/1404.7828.pdf

https://arxiv.org/abs/1404.7828 (Disclaimer: I don't know that much about ML/Deep Learning, the above was recommended by deeplearning.net and is what I'm slogging through ATM. I do have some background in "classical" statistics & optimization theory, and the above article is not that math-heavy)

So, for anyone not aware: Schmidhuber is _obsessed_ with this. He wrote an enormous literature review of deep learning [0] basically because he felt that people weren't crediting ideas enough. This isn't a one-off essay, for him, he's been banging this drum for quite a while. Not saying he's wrong, just FYI. [0] http://arxiv.org/abs/1404.7828

Machine Learning is a fast moving field, and many papers are incremental improvements, so an always up-to-date list will be unlikely. None the less, there are quite a few lists out there if you look. Here are some I've seen posted recently: https://news.ycombinator.com/item?id=8728112 http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Recomm... http://www.cs.toronto.edu/~hinton/deeprefs.html http://arxiv.org/abs/1404.7828

There are some survey papers like this: http://arxiv.org/abs/1404.7828 The point isn't to read every paper it references, but to just get a summary of what the current state of research is. If you are interested in an area then you can go read the papers there.