Skip to content
The methodOpen data
Back to the collection

Adam: A Method for Stochastic Optimization

Diederik P. Kingma, Jimmy Ba

2014 · arXiv

AlgorithmsStatistics & probabilityMachine learning
Read the paper
HN score
0
Mentions
7
Readers
7
Months
7
What’s in a score?

From the abstract

We introduce Adam, an algorithm for first-order gradient-based optimization of stochastic objective functions, based on adaptive estimates of lower-order moments. The method is straightforward to implement, is computationally efficient, has little memory requirements, is invariant to diagonal rescaling of the gradients, and is well suited for problems that are large in terms of data and/or parameters. The method is also appropriate for non-stationary objectives and problems with very noisy and/or sparse gradients. The hyper-parameters have intuitive interpretations and typically require little tuning. Some connections to related algorithms, on which Adam was inspired, are discussed. We also analyze the theoretical convergence properties of the algorithm and provide a regret bound on the convergence rate that is comparable to the best known results under the online convex optimization framework. Empirical results demonstrate that Adam works well in practice and compares favorably to other stochastic optimization methods. Finally, we discuss AdaMax, a variant of Adam based on the infinity norm.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

7 selected excerpts · 7 total mentions

brosco
Neutral mention · rule
One of OpenAI's founding team members developed Adam [0] well before it was flashy and profitable. It's not like nobody is out there trying to develop new algorithms. The reality is that there are some great, mature solvers out there that work well enough for most cases. And while it might be possible to eke out more performance in specific problems, it would be very hard to beat existing solvers in general. Theoretical developments like this, while interesting on their own, don't really contribute much to day-to-day users of linear programming. A lot of smart people have worked very hard to "optimize the optimizers" from a practical standpoint. [0] https://arxiv.org/abs/1412.6980
Original comment Researchers Discover the Optimal Way to Optimize
cherryteastain
Neutral mention · rule
As a (former) reviewer at 5 journals, I disagree first and foremost with the notion that > We need some process to gate-keep. Journals, when print was the medium through which academic research was disseminated, had to gatekeep because there were practical considerations regarding how many articles they could put in each issue. With online repositories like arxiv, this is hardly a concern anymore. Someone putting a crap article on arxiv does not hurt anyone else, and I'm saying this as a person who recommended tons of articles to be rejected because they had atrocious grammar/spelling issues. Worst case, it gets 0 attention and is ignored by the research…
Read the full comment
community. Something not being published in a journal/conference proceedings clearly does not prevent it from drawing tons of research attention, as we saw in numerous cases like the Adam optimizer [1]. Which brings us to the second point: what even is the purpose of a journal now? The answer is that the sole function of a journal now is gatekeeping, with the presupposition that, as you observed > The venue of publication is a good signal, whether the time to read a paper is well-spent Except, well, top journals have tons of articles that get 0 citations too. Clearly the filter fails at this purpose as well. So, why gatekeep at all then? Because if we did not have some exclusive prestigious journal, the plebs would not be separated from the esteemed titans of academia with the biggest grants, most prestigious scholarships and diplomas from the most famous universities. The only reason we need to gatekeep today is to feed the academic prestige and politics machine. If you care about the science, upload the goddamn PDF to arxiv , tell your colleagues about your research at a conference and let the scientific community decide on whether your idea is interesting. [1] https://arxiv.org/abs/1412.6980
Original comment The PhD Metagame: Don't try to reform science – not yet
modeless
Neutral mention · rule
Kingma is most notable for writing one of the most cited papers in AI. Actually one of the most cited scientific papers ever published, right up there with the transformers paper if not higher. "Adam: A Method for Stochastic Optimization" https://arxiv.org/abs/1412.6980 I remain astonished that Adam continues to be the most widely used optimizer in AI 10 years later. So many contenders have failed to replace it.
Original comment Anthropic hires OpenAI co-founder Durk Kingma
GistNoesis
Neutral mention · rule
Acquisitions, like for DeepMind is usually a way to hire talent. It can make sense when the technology is new and getting a few year of lead time on what is going to be a growing market may make some financial sense. In this specific xLSTM case, the industry has matured, they are just one among many (Mamba, S3Ms, transformers-variants... ), they have already been sitting on it for at least 6 months, I don't see what their play is. An other case study that's probably interesting, are the authors of the Adam Paper, https://arxiv.org/abs/1412.6980 , (Awarded "2020: The Adam optimization paper is the world's #1 most cited scientific paper of the past five years"). Probably a…
Read the full comment
few (10?,100?) billions worth value created. You can find the authors bios http://dpkingma.com/ https://jimmylba.github.io/ I think there is a huge problem with the capture and sharing of value in the whole deep-learning industry. Academia's naivety plays a role in it, Generational Shift technologies are badly rewarded. Incremental Shift technologies aren't rewarded at all. Powerful technologies into many hands with low rewards for their creators while the value they generate keeps going to the same pockets. That's a recipe for disaster. Will be a fun thing to come back in a few years to see how it had unfold.
Original comment xLSTM: Extended Long Short-Term Memory
brandonb
Neutral mention · rule
This seems like a sincere comment, so in case it's helpful, here's my interpretation of why others may be downvoting it. Deep learning practitioners are aware that they're using non-convex optimization, stochastic gradient descent, and so on. In fact, those topics are core to modern deep learning research and explicitly acknowledged in most published research: LSTMs were invented to solve the vanishing gradient problem, Restricted Boltzmann Machines were used as a pre-training step to avoid local minima, and optimizers like ADAM have explicit guarantees about things like convergence. You may know all this stuff already--not sure, based on your comment above. If not, here…
Read the full comment
are a few example papers from mainstream AI researchers explicitly talking about deep learning as an optimization problem or function approximator: [1] Why Does Unsupervised Pre-training Help Deep Learning?http://www.jmlr.org/papers/volume11/erhan10a/erhan10a.pdf [2] Multilayer feedforward networks are universal approximators: http://www.sciencedirect.com/science/article/pii/08936080899... [3] Deep Learning, Nature. http://www.cs.toronto.edu/~hinton/absps/NatureDeepReview.pdf [4] Adam: A Method for Stochastic Optimization. https://arxiv.org/abs/1412.6980
Original comment A New Batch of Deep Learning Startups to Watch for in 2017

Keep following the thread.

More in Algorithms

More from the same topics.