Zhiying Jiang, Matthew Yang, Mikhail Tsirlin
Raphael Tang, Yiqin Dai, Jimmy Lin
2023 · Findings of the Association for Computational Linguistics: ACL 2023
Zhiying Jiang, Matthew Yang, Mikhail Tsirlin, Raphael Tang, Yiqin Dai, Jimmy Lin. Findings of the Association for Computational Linguistics: ACL 2023. 2023.
Metadata: original publisher. Abstracts retain their original authors’ rights.
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
8 selected excerpts · 8 total mentions
And here's the opposite - using gzip as a (merely adequate) "large" language model: https://aclanthology.org/2023.findings-acl.426.pdf By the way, if you want to see how well gzip actually models language, take any gzipped file, flip a few bits, and unzip it. If it gives you a checksum error, ignore that. You might have to unzip in a streaming way so that it can't tell the checksum is wrong until it's already printed the wrong data that you want to see.
Interesting, I'd love to see a link to that if you know of it. Here's the original paper: https://aclanthology.org/2023.findings-acl.426.pdf In my own work I've successfully classified emergent behavior in Cellular Automata using a similar technique, and the technique has also been used elsewhere with success: https://www.nature.com/articles/s41598-022-12826-w
I'm not sure what is the state-of-the-art in sentiment analysis. However, GPT-3.5 and GPT-4 work great for it with few shot learning. You give a representative set of examples in the prompt and it works pretty well. On the topic of classification, large language models such as BERT are the state-of-the-art. But, there was a paper recently at the ACL [1] which showed that a simple approach of using gzip compression + k-nearest neighbour achieved similar level of performance. Later on, their KNN evaluation methodology came under question because instead of doing KNN with k=2 they reported accuracy based on top-2. Even then, what it shows is that we don't need LLMs for…
simple tasks such as classification, where traditional techniques still work well and are much cheaper to run. I'm sure some people might say that we should just discard that paper because it had that error. But Sebastian Raschka ran it on the IMDB movie reviews dataset and reported 71% accuracy [3]. I wrote an explainer article on it [2]. [1] https://aclanthology.org/2023.findings-acl.426.pdf [PDF] [2] https://codeconfessions.substack.com/p/decoding-the-acl-pape... [3] https://magazine.sebastianraschka.com/p/large-language-model...
this paper by the university Waterloo go in similar direction Text Classification: A Parameter-Free Classification Method with Compressors https://aclanthology.org/2023.findings-acl.426.pdf i post also this short video because correct few numbers who where miss calculated in the paper and show little less optimize and simpler implementations who still works well https://youtu.be/jkdWzvMOPuo?si=K1VRtJ5BtqREa2mz source code implementations of the video https://github.com/Sentdex/Simple-kNN-Gzip
Yeah it was published at ACL ( https://aclanthology.org/2023.findings-acl.426/ ) which is one of the most prestigious conferences in NLP. So kinda disappointing. But paper reviewers are usually not supposed to look at the actual source code of the papers, and definitely don't try to reproduce the results. They just read the paper itself, which of course doesn't talk about the error. Not sure what the best solution is, other than having the most "hyped" papers double verified by researchers on Twitter.
This is interesting, but why don’t they link to the paper? Or at least give its title? The paper is «“Low-Resource” Text Classification: A Parameter-Free Classification Method with Compressors»: ACL Anthology, has PDF link: https://aclanthology.org/2023.findings-acl.426/ Arxiv, has PDF link: https://arxiv.org/abs/2212.09410 arXiv Vanity, HTML render of it: https://www.arxiv-vanity.com/papers/2212.09410/ ResearchGate, has PDF link: https://www.researchgate.net/publication/366423949_Less_is_M... This is Listing 1: ”Python Code for Text Classification with gzip.” import gzip import numpy as np for (x1, _) in test_set: Cx1 = len(gzip.compress(x1.encode())) …
distance_from_x1 = [] for (x2, _) in training_set: Cx2 = len(gzip.compress(x2.encode()) x1x2 = " ".join([x1, x2]) Cx1x2 = len(gzip.compress(x1x2.encode()) ncd = (Cx1x2 - min(Cx1,Cx2)) / max(Cx1, Cx2) distance_from_x1.append(ncd) sorted_idx = np.argsort(np.array(distance_from_x1)) top_k_class = training_set[sorted_idx[:k], 1] predict_class = max(set(top_k_class), key=top_k_class.count) Abstract: In this paper, we propose a non-parametric alternative to DNNs that’s easy, lightweight, and universal in text classification: a combi- nation of a simple compressor like gzip with a k-nearest-neighbor classifier. Without any training parameters, our method achieves re- sults that are competitive with non-pretrained deep learning methods on six in-distribution datasets. It even outperforms BERT on all five OOD datasets, including four low-resource lan- guages. Our method also excels in the few-shot setting, where labeled data are too scarce to train DNNs effectively. Code is available at https://github.com/bazingagin/npc_gzip
Maybe this was the inspiration for your project but I just saw a paper that applied compressors as part of a text classification system getting accuracy competitive with BERT. 14 lines of Python. https://aclanthology.org/2023.findings-acl.426/
Direct link to paper: https://aclanthology.org/2023.findings-acl.426.pdf Intuitively, the key idea is that if you have two documents, say, x1 and x2, and a target document y, if x1's statistical regularities are more similar to y's than to x2's, then len(compress(x1+y)) - len(compress(y)) < len(compress(x2+y)) - len(compress(y)), where "+" means concatenation and "compress" is a compression program like gzip. Keep in mind that len(compress(x1+y)) - len(compress(y)) is, quite literally, the number of additional bytes we need to compress the statistical regularities in x1 given the statistical regularities in y. The more similar the statistical regularities of x1 and y, the…
fewer bytes we need to compress them together. The authors use kNN using a distance function called normalized compression distance (NCD), based on the above idea. Remarkably, this simple, intuitive method outperforms BERT (but not other, larger models) on a variety of zero-shot classification tasks!
More from the same topics.
Dan Tong, David J. Farnham, Lei Duan +4
Abstract If future net-zero emissions energy systems rely heavily on solar and wind resources, spatial and temporal mismatches between resource availability and electricity demand may challenge system reliability.
Steven M. Paul, Daniel S. Mytelka, Christopher T. Dunwiddie +4