We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were thoroughly validated in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free strategy for load balancing and sets a multi-token prediction training objective for stronger performance. We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to fully harness its capabilities. Comprehensive evaluations reveal that DeepSeek-V3 outperforms other open-source models and achieves performance comparable to leading closed-source models. Despite its excellent performance, DeepSeek-V3 requires only 2.788M H800 GPU hours for its full training. In addition, its training process is remarkably stable. Throughout the entire training process, we did not experience any irrecoverable loss spikes or perform any rollbacks. The model checkpoints are available at https://github.com/deepseek-ai/DeepSeek-V3.
Metadata: arXiv. Abstracts retain their original authors’ rights.
In the conversation
Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels
> MTP in Inference. Our MTP strategy mainly aims to improve the performance of the main model, so during inference, we can directly discard the MTP modules and the main model can function independently and normally. *Additionally, we can also repurpose these MTP modules for speculative decoding to further improve the generation latency.*[1]
(emphasis mine)
> Instead of predicting just the next single token, DeepSeek-V3 predicts the next 2 tokens through the MTP technique. Combined with the framework of speculative decoding (Leviathan et al., 2023; Xia et al., 2023), it can significantly accelerate the decoding speed of the model.[2]
> As DeepSeek-V3, DeepSeek-V4 series…
Read the full comment
also set MTP modules and
objectives. Given that the MTP strategy has been validated in DeepSeek-V3, we adopt the same strategy for DeepSeek-V4 series without modification.[3]
[1]: https://arxiv.org/pdf/2412.19437#subsection.2.2
[2]: https://arxiv.org/pdf/2412.19437#subsubsection.5.4.3
[3]: https://arxiv.org/pdf/2606.19348v1#subsection.2.1
Side comment: I feel you may be too cynical towards your fellow commenters.
DeepSeek research:
- V3 https://arxiv.org/abs/2412.19437
- V2 https://arxiv.org/abs/2405.04434
- R1 https://arxiv.org/abs/2501.12948 (RL applied to ML models was well-known beforehand, but they show it in the open, at scale, on big models)
Then, there's the incentive analysis. If you can see that these models empirically get better with scale, why would you swap the main architecture? Those events will be pretty rare. I'm not saying there's noone cooking a new architecture, just that it is a pretty rare event. And it would have to come from some researchers that would be happy to not publish their findings, which is not really what a sizable portion of elite researchers…
Read the full comment
(obviously not all) are incentivized to do.
Of course, it's a bit of a verbal compression to claim simply 'scaled up'. They are recognisable scaled up transformers, but most new models come with a few tricks, but we're at the point where those usually are not an architectural rewrite and added to solve an explicit problem, like hallucination, not for big new capability gains.
> Wikipedia is cheap compared to creating and training models.
DeepSeek said it spent $5.6M [1] on training V3, which doesn't look too much for a near-SOTA model. An open source entity can come up with a hybrid business model (such as requiring a small fee from those who want to host the model as a business for the first n months following the release of a new model, but making it fully free for individuals).
[1] https://arxiv.org/pdf/2412.19437
Multi-token prediction is a refined form of speculative decoding.
Researchers at Google came up with Speculative decoding in 2022: https://research.google/blog/looking-back-at-speculative-dec...
(Fast Inference from Transformers via Speculative Decoding -
Yaniv Leviathan, Matan Kalman, Yossi Matias)
Researchers at Meta came up with MTP, a smarter way of doing speculative decoding in 2024: https://arxiv.org/abs/2404.19737 (Better & Faster Large Language Models via Multi-token Prediction
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve)
DeepSeek V3 shipped MTP in a product first, in 2024: https://arxiv.org/abs/2412.19437 (DeepSeek-V3 Technical Report, 100+ authors)
Correct, but note that's exactly what inference providers do.
https://arxiv.org/pdf/2412.19437
> The minimum deployment unit of the decoding stage consists of 40 nodes with 320 GPUs. The attention part employs TP4 with SP, combined with DP80, while the MoE part uses EP320.
EP320 means expert parallelism, each on 320 GPUs.
Some of the big LLM labs have written about their training hardware.
DeepSeek v3 was trained on 2,048 NVIDIA H800s. https://arxiv.org/abs/2412.19437
MiniMax M1 used 512 H800s. https://arxiv.org/abs/2506.13585
The H800 wasn't banned in the first round of export controls - but was after October 2023: https://www.cnbc.com/2023/10/17/us-bans-export-of-more-ai-ch...
Z.ai say they used Huawei hardware: https://www.theregister.com/2026/01/15/zhipu_glm_image_huawe...
Qwen and Kimi haven't disclosed their hardware as far as I can tell.
I'm confused, that doesn't make sense to me:
> They largely come from hyperscalers who want hard drives for their AI data centers, for example to store training data on them.
What type of training data? LLMs need relatively little of that. For example, DeepSeek-V3 [1], still a relatively large model:
> We pre-train DeepSeek-V3 on 14.8 trillion diverse and high-quality tokens
At 2 bytes per token, that's 29.6 terabytes. That's basically nothing compared to the amount of 4K content that is uploaded to YouTube every day.
1: https://arxiv.org/html/2412.19437v1
> Has any of these outfits ever publicly stated they used Nvidia chips? As in the non-officially obtained 1s. No.
Uh yes? Deepseek explicitly said they used H800s [1]. Those were not banned btw, at the time. Then US banned them too. Then US was like 'uhh okay maybe you can have the H200', but then China said not interested.
[1] https://arxiv.org/pdf/2412.19437
Very interesting. It sounds like tuning at the PTX level can increase workload efficiencies, such as quote "Specifically, we employ customized PTX (Parallel Thread Execution) instructions" from the DeepSeek folks https://arxiv.org/abs/2412.19437.
The explanation for Multi-head Latent Attention https://www.stephendiehl.com/posts/post_transformers/#multi-... does not match the definition in the DeepSeek-V2 paper https://arxiv.org/pdf/2405.04434#subsection.2.1
MLA as developed by DeepSeek is a technique to reduce the memory footprint of the KV cache by storing only two vectors of size latent_dim and rope_dim per token and layer, instead of 2 * num_heads vectors of size head_dim. (DeepSeek-V3 has num_heads = 128 and head_dim = 128 vs latent_dim = 512 and rope_dim = 64, so a significant reduction https://arxiv.org/pdf/2412.19437#subsection.4.2 )
What this article describes instead is some kind of two-step attention…
Read the full comment
scheme I haven't seen before and that I think wouldn't work with causal masking (despite mask appearing in the example code) because either you allow an earlier token to attend to a latent that attended to a later token (creating backwards information flow) or the latents can only attend to a limited prefix of the sequence, after which they're frozen and useless. I wonder whether the author dreamed it up himself or whether someone else is actually using this somewhere.
The Deepseek v3 paper details a quantisation method of scaling after matmul but before accumulation to improve precision, this is different than normal GEMM as operations are left till the end, can read more in chapter 3.3 of the paper below.
https://arxiv.org/html/2412.19437v2#S3
DeepSeek V3/R1 are MoE with 256 experts per layer, actively using 1 shared expert and 8 routed experts per layer https://arxiv.org/html/2412.19437v1#S2:~:text=with%20MoE%20l... so you can't just take the active parameters and assume that's close to the size of a single expert (ignoring experts are per layer anyways and that there are still dense parameters to count).
Despite connotations of specialized intelligences the term "expert" provokes it's really mostly about scalability/efficiency of running large models. By splitting up sections of the layers and not activating all of them for each pass a single query takes less bandwidth, can be distributed across compute, and can be parallelized with other queries on the same nodes.
> if a llm will run with usable performance at that scale?
Yes.
The reason: MoE. They are able to run at a good speed because they don't load all of the weights into the GPU cores.
For instance, DeepSeek R1 uses 404 GB in Q4 quantization[0], containing 256 experts of which 8 are routed to[1] (very roughly 13 GB per forward pass). With a memory bandwidth of 800 GB/s[3], the Mac Studio will be able to output 800/13 = 62 tokens per second.
[0]: https://ollama.com/library/deepseek-r1
[1]: https://arxiv.org/pdf/2412.19437
[2]: https://www.apple.com/newsroom/2025/03/apple-unveils-new-mac...
Are you sure they ditched CUDA? I keep hearing this, but it seems odd because that would be a ton of extra work to entirely ditch it vs selectively employing some ptx in CUDA kernels which is fairly straightforward.
Their paper [1] only mentions using PTX in a few areas to optimize data transfer operations so they don't blow up the L2 cache. This makes intuitive sense to me, since the main limitation of the H800 vs H100 is reduced nvlink bandwidth, which would necessitate doing stuff like this that may not be a common thing for others who have access to H100s.
1. https://arxiv.org/abs/2412.19437
Thus you get Open AI spending billions while DeepSeek comes long with, shocking, actual understanding of the hardware and how to optimize for it[1] and spends $6 Million[2]
1. https://arxiv.org/abs/2412.19437v1
2. Quibble over the exact figure. Far less than Open AI, doing more with less.
This articles that distills a report which doesn't bring anything new to the table. I can't load their site right now, but in their release I recall they explicitly excluded experimentation and R&D costs. And have $1.6b worth of hardware is not the same as spending $1.6b on costs to build the model.
The article says:
> despite the company's claims that DeepSeek only cost $6 million and 2,048 GPUs to train. However, industry analyst firm SemiAnalysis reports that the company behind DeepSeek incurred $1.6 billion in hardware costs and has a fleet of 50,000 Nvidia Hopper GPUs, a finding that undermines the idea that DeepSeek reinvented AI training and inference with…
Read the full comment
dramatically lower investments than the leaders of the AI industry.
The underlying report states:
> The $6M cost in the paper is attributed to just the GPU cost of the pre-training run, which is only a portion of the total cost of the model. Excluded are important pieces of the puzzle like R&D and TCO of the hardware itself. > > https://semianalysis.com/2025/01/31/deepseek-debates/
But we already know this. The DeepSeek paper even gives a breakdown:
> Pre-Training: 2664K > Context Extension: $0.238M > Post-Training: $0.01M > Total: $5.576M > > https://arxiv.org/pdf/2412.19437v1
>To build DeepSeek probably requires at least a $1B+ budget.
Zero evidence that the above statement is true, and weak evidence (authors' claims) that it is false. Have you read their papers even?
https://arxiv.org/html/2412.19437v1#abstract https://arxiv.org/pdf/2501.12948
> If the magic values are some kind of microcode or firmware, or something else that is executed in some way, then no, it is not really open source.
To my understanding, the contents of a .safetensors file is purely numerical weights - used by the model defined in MIT-licensed code[0] and described in a technical report[1]. The weights are arguably only really "executed" to the same extent kernel weights of a gaussian blur filter would be, though there is a large difference in scale and effect.
[0]: https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inferen...
[1]: https://arxiv.org/html/2412.19437v1
There's a lot of conflicting information here.
There are two papers: https://arxiv.org/pdf/2412.19437v1 for DeepSeek-V3-Base, which makes claims about 2.8M H800 GPU hours, and https://arxiv.org/pdf/2501.12948 for DeepSeek-R1, which was based on V3-Base but doesn't seem to quote the cost of its RL steps.
https://newsletter.languagemodels.co/p/the-illustrated-deeps... is a really fascinating overview of what those new steps were! The innovation here seems to be that the creation of a cutting-edge reasoning model could be made from a relatively inexpensive base model, regardless of how much RL is needed. Part of this seems to be a novel methodology for generating large…
Read the full comment
volumes of chain-of-thought training data; it's left ambiguous exactly how the bulk of this data was generated, though, and it could have been a long process of manual curation.
I'd also take the assertion that they started R1 with V3-Base in its release form with a grain of salt. We'll see whether people can reproduce similar results. Either way, there's a new chain-of-thought model with open weights, and that's an incredible achievement regardless of the context - and one that will be meaningful to future research.
The DeepSeek v3 model had a net training cost of >$5m for the final training run, the paper lists over 100 authors[1], meaning highly-paid engineers. This is also one of a sequence of models (v1, v2, math, coder) trained in order to build the institutional knowledge necessary to get to the frontier , and this ends up still far above the $10m mark. It's hardly a "trio of super-smart engineers".
[1] https://arxiv.org/abs/2412.19437v1
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition.
Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task.
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small difference between training and test performance.