No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

These diminishing returns are exponentially diminishing returns [1]. [1] https://arxiv.org/abs/2404.04125 " We consistently find that, far from exhibiting "zero-shot" generalization, multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance, following a sample inefficient log-linear scaling trend. This trend persists even when controlling for sample-level similarity between pretraining and downstream datasets, and testing on purely synthetic data distributions. Furthermore, upon benchmarking models on long-tailed data sampled based on our analysis, we demonstrate that multimodal models across the board perform poorly."

I thought I saw some paper suggesting that accuracy grows linearly with exponential data. If that's the case it's not a mystery why we'd be hitting a training wall. Not sure I got the right takeaway from that study, though. EDIT: here's the paper https://arxiv.org/abs/2404.04125

Research seems to suggest we need exponential training data volume increases to see meaningful performance gains: https://arxiv.org/abs/2404.04125 Personally I think we've already hit a ceiling.

EDIT: cite source I have yet to see any evidence of this. To the contrary, we have seen for years that advances in LLMs require orders of magnitude more parameters/power for each generation. Neural architecture search has been underwhelming. RLHF seems to have regressed models like GPT3.5+, rather than improving them. And recently, researchers concluded that multi-modal models require exponentially more data for each extra "mode" being added to the model.[0] Even in my own experimentation, I've tried to get some AGI-like behavior, and it just isn't there. I have convinced GPT4, for instance, to generate XML source for an SVG, but it looks nothing like what I describe. I'd argue that these models don't generalize well at all, and I'd bet that, like with Moore's Law, advances in AI will require continual discovery of incremental new innovations and occasionally new architectures. [0]https://arxiv.org/abs/2404.04125

Some research to the contrary [1] - tldr is that they didn't find evidence that generative models really do zero shot well at all yet, if you show it something it literally hasn't seen before, it isn't "generally intelligent" enough to do it well. This isn't an issue for a lot of use-cases, but does seem to add some weight to the "giga-scale memorization" hypothesis. [1] https://arxiv.org/html/2404.04125v2

> So, you're betting on no significant cost reduction of compute hardware? Seems implausible to me. This is not about compute, but about data. https://arxiv.org/abs/2404.04125 "...our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found."