Original ·1884440764677251515· Alexandr Wang @alexandr_wangWhat does DeepSeek R1 & v3 mean for LLM data?
Contrary to some lazy takes I’ve seen, DeepSeek R1 was trained on a shit ton of human-generated data—in fact, the DeepSeek models are setting records for the disclosed amount of post-training data for open-source models:
- 600,000 reasoning data [1]
- 200,000 non-reasoning SFT data [2]
- human preference (RLHF) dataset of undisclosed size [3]
- human-processed synthetic data for cold-start data [4]
According to Chinese AI engineers, DeepSeek actually values data annotation even more than other Chinese labs, with the CEO personally labeling data for the model [5] (This reminds me of @karpathy who used to spend a quarter of his time labeling at Tesla). The DeepSeek-v3 paper even has a dedicated acknowledgement section for Data Annotation [6].
DeepSeek-V3, which was distilled from DeepSeek-R1, was also trained on an instruction-tuning dataset of 1.5M samples. [7]
These SFT datasets are even larger than other open-source models:
- Qwen-2.5 was trained on 1M SFT samples [8]
- the last time Meta disclosed was for Llama 2, which was trained on only 30k SFT samples and 3M RLHF samples [9]
- Kimi k1.5 was trained on roughly 1M SFT, 1M multi-modal SFT, 800k samples for classic reward modeling, and another 800k CoT labeled examples for reasoning [10]
It’s interesting that the size of the RLHF dataset was undisclosed, while they disclosed the size of the SFT and reasoning datasets. This could be because it is much larger than one would expect, or it reveals some interesting technical detail they don’t care to share. Human preference datasets are often much larger than SFT datasets in most models, so a reasonable estimate would be that DeepSeek’s models are probably trained on at least 3-5M samples, which is quite a large preference dataset!
The main technical breakthrough of DeepSeek-R1 is that for reasoning, you can forgo SFT data in favor of reasoning data—but reasoning data is still human data of difficult problems&answers in a variety of domains.
The reasoning dataset is actually quite large—600k reasoning samples is a LOT. This is in line with a broader trend we’ve seen from SFT data towards other data types like human preference/RLHF data and reasoning data. This is for technical reasons—SFT caps the performance of the model at a certain level, whereas RLHF or other methods enable the models to continue improving without bound beyond the limits of the dataset.
DeepSeek R1 is a very exciting model, and it’s great to see o1 reasoning capabilities replicated in the wild. In terms of training data, however, the DeepSeek models are actually setting open-source records in terms of the amount of human data used.
[1]
[2]
[3]
[4] https://arxiv.org/pdf/2501.12948
[5] https://www.chinatalk.media/p/deepseek-the-view-from-china
[6] https://arxiv.org/html/2412.19437v1
[7] https://arxiv.org/html/2412.19437v1
[8] https://arxiv.org/pdf/2412.15115
[9] https://arxiv.org/pdf/2307.09288
[10] https://arxiv.org/html/2501.12599v1
RT @alexandr_wang: What does DeepSeek R1 & v3 mean for LLM data?
Contrary to some lazy takes I’ve seen, DeepSeek R1 was trained on a shit…