Qwen3 Technical Report

recombinations of existing data with random mutation. Where else would the building blocks for each invention come from, divine insight? We build on the shoulders of giants etc etc Worth noting, as a sidebar, that we’re having this discussion on a post mentioning a novel breakthrough made by AI over a topic that many brilliant human mathematicians including Erdos himself failed to do. > Regardless of whether something's been done before people still come up with them on their own without directly copying or amalgamating several copies. I’m not even saying it in the “there’s nothing new under the sun” sense. If you follow an average person’s day from beginning to end. Let’s say in Bangkok or NYC or Paris, at which part of the day are they not simply repeating a variation of something they’ve done many times before, or seen others around them do before, or read about others doing before, or heard about others doing before, watched others do before on TV etc etc What you have left, how is it distinguishable, without reasoning backwards from the desired conclusion of human exceptionalism, from turning up the temperature on an LLM query? How many data points does a human parse when they attempt to stand up as a toddler? Sight, sound, sensation from every limb and body part, inner ear, internal thought processes at the time conscious and unconscious related to the moment and attempting to interpret it in relation to all that it’s experienced to this point, including all prior attempts and whatever retained associated data, a hard to even comprehend stream of data, coming in continuously over however many minutes, hours, etc of attempts. The stream of data the brain is processing from both external and internal sources from birth is incredibly rich, and if we attempted to represent the full depth of it it would far outweigh the size of any corpus models are being trained on now. I think what may be genuinely missing from AI is the type of data that doesn’t translate completely into text. The audio and images/video we feed in are a totally incomplete slice of the POV of say even a single average human through their lifetime, and bereft of all the associated data a human has access to in the moment (sensory etc). I think this tends more towards the world models that Yann Lecun et al are promoting as the key to more capable AI.

doubled the pre-training token count, and tripled the number of languages. It's been shown that training on more languages can actually help LLMs generalize better. They used Qwen2.5 VL and Qwen 2.5 to generate additional training data by parsing a large number PDFs and turning them into high quality training tokens. They improved their annotation so they could more effectively provide diverse training tokens to the model, improving training efficiency. They continued this trend with Qwen3.5, where even more and better training data[3] made their Qwen3.5-397B-A17B model match the 1T-parameter Qwen3-Max-Base. That said there's also been a lot of work on model architecture[4], getting more speed and quality per parameter. In the case of Qwen3-Next architecture which 3.5 is based on, that means such things as hybrid attention for faster long-context operation, and sparse MoE and multi-token prediction for less compute per output token. I used Qwen as an example here, from what I gather they're just an example of the general trend. [1]: https://arxiv.org/abs/2505.09388 [2]: https://arxiv.org/abs/2412.15115 [3]: https://qwen.ai/blog?id=qwen3.5 [4]: https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d...