Learning Transferable Visual Models From Natural Language Supervision

The author asks: > Has the model ever seen cosine similarity? Yes - most of the time, at least for deep learning based semantic search. E.g. for semantic search of text, the majority are using, SentenceTransformers [1], models which have been trained to use cosine similarity. Or e.g. for vector representations of images, people are using models like CLIP [2], which has again been trained to use cosine similarity. (Cosine similarity being used in the training loss, so the whole model is fundamentally "tuned" for cosine similarity.) Articles like these cause confusion, e.g. I've come across people saying: "You shouldn't use cosine similarity", when they've seen SentenceTransformers being used, and linking articles like these, when in fact you very much should be using cosine similarity with those models. [1] https://sbert.net [2] https://arxiv.org/abs/2103.00020

Where are you getting 2 billion from? The original CLIP paper says: > We demonstrate that the simple pre-training task of predicting which caption goes with which image is an efficient and scalable way to learn SOTA image representations from scratch on a dataset of 400 million (image, text) pairs collected from the internet. [1] OpenCLIP was trained on more images, but the datasets like LAION-2B are kind of low-quality in terms of labeling; I find it plausible that a better dataset could outperform it. I'm pretty sure that the stock images Adobe is drawing from have better labeling already. I agree that this is likely to backfire on artists, but part of that is that I expect the outcome to be that large corporations will license private datasets and open research will starve. [1] https://arxiv.org/abs/2103.00020

From the paper on the CLIP embedder it appears that they use a form of contrastive loss that maximizes the cosine similarity between related images and prompts, but also minimizes the same between unrelated prompts & images. See section 2.3 of the CLIP paper: https://arxiv.org/pdf/2103.00020.pdf Also, the writeup on OpenAI's blog: https://openai.com/blog/clip/

> What is the content of the training set like, that DALL·E 2 has all this info? Web crawling. (https://arxiv.org/pdf/2103.00020.pdf 2.2) > Are there people out there spending thousands of hours just tagging or writing descriptions of training images? "Yes", Google Image Search, Wikipedia captions, Danbooru are all that in some sense. The biases should be obvious if you think about it for a minute and CLIP has even more biases since they removed all kinds of NSFW content, etc. > And does DALL·E just receive the image as-is with its description, or does it get more info (e.g. "this part of the image is the dog")? Only that, but segmentation is a big research area too and it looks like FB has a reproduction with some of that mixed in. https://arxiv.org/abs/2203.13131

Cosine-similarity is a useful metric. The cases where it is useful are models that have been trained specifically to produce a meaningful cosine distance, (e.g. OpenAI's CLIP [1], Sentence Tranformers [2]) - but these are the types of models that the majority of people are using when they use cosine distances. > It's my understanding that the delta between two word embeddings, gives a direction, and the magic is from using those directions to get to new words... it's the directions and distances to nearby objects that matters most Cosine similarity is a kind of "delta" / inverse distance between the represenation of two entities, in the case of these models. [1] https://arxiv.org/abs/2103.00020 [2] https://www.sbert.net/docs/training/overview.html

As far as I know, OpenAI has not made this dataset publicly available. IIRC the dataset is images scraped from instagram and their corresponding captions. Check out the CLIP paper for more details: https://arxiv.org/abs/2103.00020 Theoretically, you could build a web-scraping tool to do something like this, but even storing that data would take an absolutely insane amount of storage. I would assume OpenAI has some deal with Meta to make the creation of datasets like this easier.