Learning to learn by gradient descent by gradient descent

> So, in case of (stochastic) gradient descent to train a neural network, that would be the neural network that you want to use AD on, not the gradient descent algorithm. Obviously. The specific proposal of differential programming as a paradigm that goes beyond simple applications of gradient descent for optimising NNs is exactly to apply gradient descent to optimise learning (and other) algorithms themselves. This may be to select hyperparameters, or to select among a family of algorithms, or to include differentiable constraints in the solver, etc. I wonder, are you familiar with now classic paper, Learning to learn by gradient descent by gradient descent? [0] https://arxiv.org/abs/1606.04474

Your algorithm description sounds like a variant of greedy local search with a NN heuristic. While I personaly do not know of a paper that does this kind of local search (I have not looked), I do want to add that there is a rich literature on "learning to learning", including "learning to optimize" approaches, e.g., https://arxiv.org/abs/1606.04474.

There has been some work related to what you are describing: Learning to learn by gradient descent by gradient descent https://arxiv.org/abs/1606.04474 Related (but less so), there are also some papers about learning neural network architectures: Designing Neural Network Architectures using Reinforcement Learning https://arxiv.org/abs/1611.02167 Neural Architecture Search with Reinforcement Learning https://arxiv.org/abs/1611.01578

Besides Googling for tricks, another way of learning LaTeX is to download the source code for nice-looking papers that you find on the arXiv. For example, for "Learning to learn by gradient descent by gradient descent" arXiv:1606.04474 (https://arxiv.org/abs/1606.04474), if you click on Download->Other formats, you get the latex source code and in this case also the Omnigraffle files for the diagrams.

I think it's important to note that all of these methods have been largely superseded by deep learning techniques. For example, we can now directly learn algorithms such as gradient descent[1] and classical inverse problems like superresolution are solvable with deep networks [2]. While there still may be a role for tools like CVX, I anticipate all future progress will come from end-to-end differentiable systems. [1] https://arxiv.org/abs/1606.04474 [2] https://arxiv.org/abs/1501.00092

GANs (and InfoGAN[0] in particular) has shown there are still pretty huge breakthoughs out there. Some other things you may be interested in: Learning to learn by gradient descent by gradient descent[1] Neural Architecture Search with Reinforcement Learning[2] [0] https://arxiv.org/abs/1606.03657, https://arxiv.org/pdf/1701.00160.pdf [1] https://arxiv.org/abs/1606.04474 [2] https://openreview.net/pdf?id=r1Ue8Hcxg

The obvious citation is the wonderfully titled Learning to learn by gradient descent by gradient descent https://arxiv.org/abs/1606.04474

Disclaimer: Interested in NN, but no real knowledge yet. A ML-learning group at my university recently discussed Learning to learn by gradient descent by gradient descent [1], the topic seems related. This indicates to me that at least some progress was made. [1] https://arxiv.org/abs/1606.04474

Blog spam pass through to the original: http://sebastianruder.com/optimizing-gradient-descent/ Useful reference, 6 months old. Side note, anybody aware of any implementations of the "learning to learn gradient descent by gradient descent" work [0]? I'd love to boost my training time, but I'm worried implementing it myself will just result in an additional set of hyperparameters to tune. [0] https://arxiv.org/abs/1606.04474

> Humans can design good features for the ML algorithm to use though For now. Articles like [1] make me wonder whether this current generation of programmers might be the last. [1] https://arxiv.org/abs/1606.04474