Skip to content
The methodOpen data
Back to the collection

Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches

Maurizio Ferrari Dacrema, Paolo Cremonesi, Dietmar Jannach

2019 · arXiv

Machine learningScience & researchAlgorithms
Read the paper
HN score
0
Mentions
4
Readers
3
Months
4
What’s in a score?

From the abstract

Deep learning techniques have become the method of choice for researchers working on algorithmic aspects of recommender systems. With the strongly increased interest in machine learning in general, it has, as a result, become difficult to keep track of what represents the state-of-the-art at the moment, e.g., for top-n recommendation tasks. At the same time, several recent publications point out problems in today's research practice in applied machine learning, e.g., in terms of the reproducibility of the results or the choice of the baselines when proposing new models. In this work, we report the results of a systematic analysis of algorithmic proposals for top-n recommendation tasks. Specifically, we considered 18 algorithms that were presented at top-level research conferences in the last years. Only 7 of them could be reproduced with reasonable effort. For these methods, it however turned out that 6 of them can often be outperformed with comparably simple heuristic methods, e.g., based on nearest-neighbor or graph-based techniques. The remaining one clearly outperformed the baselines but did not consistently outperform a well-tuned non-neural linear ranking method. Overall, our work sheds light on a number of potential problems in today's machine learning scholarship and calls for improved scientific practices in this area. Source code of our experiments and full results are available at: https://github.com/MaurizioFD/RecSys2019_DeepLearning_Evaluation.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

timkam
Neutral mention · rule
I think when looking at least at large scale applications in the context of games (and not just super-expensive showcases), like in the Stockfish chess engine, we see that it's not primarily about depth, it's about architecture design. Reference: start here https://stockfishchess.org/blog/2021/stockfish-14/ and go down the rabbit hole... In the broadly useful domain of recommender systems (which typically make use of some type of RL-like feedback loop, but can be implemented using simple clustering approaches), at least in 2019, neural network-based approaches didn't seem to fair too well, either: https://arxiv.org/pdf/1907.06902.pdf (arXiv pre-print, but this is an award-winning paper). Since then, it seems that researchers are moving away from getting deeper and deeper (the low-hanging fruit), and try to be more creative instead: new architectures, combining symbolic (logic-based) and sub-symbolic (ML-based) AI, etc.
Original comment Deep reinforcement learning is a waste of time (2019)
tomkat0789
Neutral mention · rule
+1 to this from someone who learned the math behind ML in a PhD and was looking forward to being a gatekeeper :) My favorite academic paper ever [0] was a comparison against a bunch of dimensionality reduction algorithms and 100 year old PCA was tough to beat! Glad I was able to pivot my career out of AI and ML. My PhD wasn't at Stanford, MIT, et al so I couldn't find any jobs doing the "actual research" - if they existed at all outside academia. EDIT to add another funny "frustration" paper more directly related to ML [1]. I consider DR is more of a data analysis thing. [0]: van der Maaten, et al. Dimensionality Reduction: A Comparative Review https://members.loria.fr/moberger/Enseignement/AVR/Exposes/T... [1]: Dacrema, et al. Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches https://arxiv.org/pdf/1907.06902.pdf
Original comment AI skills are worth less than you think (2019)
mkasu
Neutral mention · rule
Yes, the seeming performance of (especially) neural models compared to traditional models is probably the main factor. Although, some voices[1] argue that traditional or much simpler approaches still often do a similar job compared to super over-engineered models, especially when going even slightly beyond an existing target-dataset or task. I'd argue, that improving the ML models is really the job of ML researchers and should be mainly targeting ML conferences like AAAI (Adv. of AI). In other conferences (directly targeting NLP, CV, Comp. Biology, etc.) it should be the main job to combine those models with the domain-specific characteristics (e.g., language information…
Read the full comment
for NLP) or "traditional" methods to make it an interesting discussion. I was recently doing reviewing for a multimedia conference and quite a lot of the papers I reviewed were basically pure ML papers. A colleague had the same experience. 1: https://arxiv.org/abs/1907.06902
Original comment Language-Agnostic Bert Sentence Embedding
timkam
Neutral mention · rule
There are recent works that question whether "deep" learning algorithms, and in particular neural networks, are really advancing the state of the art of recommender systems: https://arxiv.org/pdf/1907.06902.pdf (Best Paper Award, RecSys 2019). It is strange that the blog post fails to mention this.
Original comment Categories of deep recommendation systems

Keep following the thread.

More in Machine learning

More from the same topics.