🌐 English entry page. Deep pages are in Chinese. 中文版 / Chinese: 中文
title: LLM Technology Evolution Research (GPT-3 → 2026-06) · Main Summary type: source tags: [llm, survey, pretraining, architecture, ai-infra, post-training, agentic] created: 2026-06-18 updated: 2026-06-18
LLM Technology Evolution Research · Main Summary (GPT-3 2020-05 → 2026-06)
Primary official sources only: original arXiv papers, official technical reports/system cards, official blogs, official GitHub/model cards/HF organization pages. No third-party interpretations/translations/secondary summaries/benchmark aggregations. Five categories:
pretraining data·architecture·AI infra·post-training·agentic training(each entry hits at least one; may be multi-tagged).
1. Corpus Statistics
- Sources after deduplication: 532 entries (parsed from 667 structured pages, deduplicated by original URL)
- Downloaded primary sources: about 760 files / ~2.6 GB —
*.html/jsonship with the repo (sources/llm/<scope>/);*.pdflives on the HF private bucketjaczhao/ai-research-sources(storage-compute separation, not in git) - Open-source model deep-dive archives: 14 files (
deep-dive/, including a “2026 post-research incremental addendum” section) - Categorical summary chapters: 6 (
sections/, with a 2026 incremental addendum appended)
By year (deduplicated): 2018→1 · 2019→4 · 2020→36 · 2021→32 · 2022→58 · 2023→102 · 2024→136 · 2025→110 · 2026(H1)→53
By country (deduplicated): US/West 303 · China 196 · Europe 20 · Other ~13 — all top Chinese and US companies covered
By category (deduplicated, multi-tag): architecture ~298 · post-training ~283 · AI infra ~180 · pretraining data ~178 · agentic training ~143
2. How to Read This Research
Four layers, by granularity from “conclusion” to “primary source”:
- Five categorical summaries (core, read these first) —
sections/ - Open-source model deep-dive archives (for training/data/mixture-ratio/RL/architecture details) —
deep-dive/(see Section 4) - Full source index (by year/month, every entry clickable) — 01-INDEX
- Per-entry structured pages —
2020/…2026/,themes/; downloaded primary sources —sources/llm/<scope>/(html/json with the repo, pdf on HF bucket)
3. Quick Overview of the Five Categories
1) Pretraining Data → full chapter (173 entries)
From GPT-3’s coarse-filtered CommonCrawl (~300B tokens) → quality classifiers and deduplication becoming standard (The Pile, C4, RefinedWeb) → recipes going public (LLaMA mixture ratios, Dolma/RedPajama fully open) → data quality > data quantity (Phi “textbooks”, FineWeb-Edu edu classifier, DCLM) → multi-stage/annealing and synthetic data (MiniCPM WSD, Nemotron synthetic, Llama 3 15T+annealing) → trillion-to-tens-of-trillions tokens, long-context stages, strong decontamination.
- Milestones: Scaling Laws · Chinchilla compute-optimal · [The Pile] · [LLaMA] · Dolma · FineWeb-Edu · DCLM · [Phi Textbooks]
2) Architecture → full chapter (291 entries)
Decoder-only Transformer settling in → positional encodings RoPE/ALiBi/YaRN → memory-saving attention MQA/GQA→MLA (DeepSeek-V2) → MoE going mainstream: GShard/Switch/GLaM → DeepSeekMoE fine-grained + shared experts → auxiliary-loss-free load balancing → trillion-scale sparse (Kimi K2, Ling/Ring-1T, Pangu Ultra-MoE 718B) → linear/hybrid architectures (Mamba, RWKV, RetNet, Jamba, Nemotron-H, MiniMax Lightning/Sparse, Kimi Linear, Falcon-H1) → native multimodal/omni and native sparse attention.
- Milestones: [GShard]·[Switch]·[GLaM]·DeepSeekMoE·[Mamba]·DeepSeek-V2·[aux-loss-free MoE]·Native Sparse Attention·[Jamba]·Nemotron-H
3) AI infra → full chapter (175 entries)
Training side: Megatron-LM tensor parallelism + DeepSpeed/ZeRO memory sharding → 3D/sequence parallelism, activation recomputation → FP8 training, MegaScale ten-thousand-card → DeepSeek-V3 FP8+DualPipe open-sourcing the full stack (open-source week FlashMLA/DeepEP/DeepGEMM/3FS). Inference side: FlashAttention 1/2/3 → PagedAttention/vLLM → SGLang/RadixAttention → quantization (GPTQ/AWQ/SmoothQuant) → PD disaggregation/Mooncake/MegaScale-Infer. RL training systems: DeepSpeed-Chat→OpenRLHF→veRL/HybridFlow→DAPO/AReaL/ProRL.
- Milestones: DeepSpeed·[Megatron activation recomputation]·[FlashAttention]·[vLLM/PagedAttention]·DeepSeek-V3 hardware reflections·DeepSeek open-source week·[veRL/HybridFlow]
4) Post-training → full chapter (276 entries)
RLHF foundations (Learning to summarize→InstructGPT) → Constitutional AI/RLAIF → offline preference optimization DPO and variants (IPO/KTO/ORPO/SimPO) → verifiable-reward RLVR + GRPO (DeepSeekMath) → reasoning-model paradigm (o1 → DeepSeek-R1 pure-RL “aha moment”, Kimi k1.5) → process rewards PRM, long CoT, on-policy distillation, prolonged RL (ProRL) → fully open pipelines (Tülu 3, OLMo 2, MiMo, Nemotron Cascade).
- Milestones: [InstructGPT]·DPO·[Constitutional AI]·DeepSeekMath GRPO·DeepSeek-R1·Kimi k1.5·Tülu 3·on-policy distillation
5) Agentic Training → full chapter (136 entries)
Prompt era (WebGPT/ReAct/Toolformer/Reflexion/Voyager/Generative Agents) → tool-use fine-tuning (ToolLLM/AgentTuning/CodeAct) → evaluation-driven (SWE-bench/WebArena/OSWorld/tau-bench) → computer/browser use (Claude computer use, Operator/CUA, UI-TARS) → multi-turn agent RL explosion (2025) (Search-R1/ReTool/RAGEN/ToolRL/DeepResearcher/WebSailor) → agentic-native models (Kimi K2, GLM-4.5 ARC, Qwen3-Coder, Tongyi DeepResearch, Seed2.0).
- Milestones: [ReAct]·[Toolformer]·[SWE-bench/SWE-agent]·UI-TARS·Search-R1·CUA·Tongyi DeepResearch
4. Open-Source Model Training Recipe Deep Dives (deep-dive/)
Each gives, model by model: exact architecture config · pretraining data sources/tokens/mixture ratios · data processing pipeline (deduplication/quality filtering) · training (compute/parallelism/precision/stages) · SFT · RL/alignment · infra. For cross-comparison see open-model-recipes.md.
- DeepSeek — LLM/MoE/V2(MLA)/V3(FP8+DualPipe)/R1(RLVR)/Math(GRPO)/Coder/Prover/V3.1-3.2
- Tongyi Qianwen — Qwen1→3, 2.5/3-Coder/Math, Qwen3-Next, QwQ, VL/Omni
- Llama — LLaMA1/2/3 (Herd full details), Code Llama, Llama Guard
- Tülu — the most transparent, fully open (data + code + recipe + RL all public)
- ChatGLM · Mixtral · Gemma
- MiniCPM — richest detail on WSD scheduling + data annealing
- InternLM · 01.AI
- Nemotron(NV)+Phi(MS) — synthetic data + reward models + textbook data
- Early and fully open — BLOOM/Pythia/GPT-NeoX/Falcon/MAP-Neo/OpenCoder/StarCoder/RedPajama/DCLM
- New-generation large open MoE — Kimi K2 / MiniMax-01 / Hunyuan-Large / Skywork-MoE / Step / dots / Ling / Pangu MoE
- Pretraining dataset & processing pipeline topic — deduplication · quality filtering · mixture ratios · ablations of FineWeb/DCLM/The Stack/RedPajama/Dolma/Nemotron-CC
5. Annual Timeline (selected milestones; full entries in 01-INDEX)
2020 — Scaling laws and the dawn of RLHF
- GPT-3 175B (few-shot in-context learning) · Scaling Laws
- DeepSpeed + ZeRO-2 · Turing-NLG 17B
- Learning to summarize from HF (RLHF foundations) · GShard(MoE) · DeBERTa · Meena · CPM(Chinese) · OpenAI API
2021 — Scaling up, MoE, instruction tuning takes off
- [Switch Transformer] · GLaM · [Gopher] · Megatron-Turing NLG 530B
- FLAN / T0 (instruction tuning) · [Codex] · [WebGPT] · Anthropic “A General Language Assistant”
- China: ERNIE 3.0 · PanGu-α · Yuan 1.0 · Wudao 2.0 · RoPE · ALiBi · ZeRO-Infinity
2022 — Chinchilla, RLHF productization, CoT, open-source wave
- Chinchilla compute-optimal · PaLM · OPT · BLOOM · GLM-130B
- InstructGPT(RLHF) · Chain-of-Thought · Self-Consistency · Emergent Abilities
- FlashAttention · ReAct · Anthropic HH-RLHF + Constitutional AI · ChatGPT launch
2023 — GPT-4, explosion of open-source bases, alignment and agent tool-use
- US/West: [GPT-4] · LLaMA / Llama 2 · PaLM 2 · Claude 2 · Gemini · Mistral 7B · DPO · QLoRA · [vLLM/PagedAttention] · Mamba · Toolformer/Reflexion/Voyager · SWE-bench
- China: Qwen · Baichuan 2 · InternLM · ChatGLM2/3 · Yi · DeepSeek LLM/Coder/MoE · Qwen-VL
2024 — Multimodal flagships, reasoning takes off, MoE engineering, fully open data
- US/West: Llama 3 Herd · [GPT-4o] · o1 reasoning · Claude 3/3.5 · Gemini 1.5 · Gemma 2 · [OLMo+Dolma] · [FineWeb] · Tülu 3 · Nemotron-4 340B · SimPO/KTO/ORPO
- China: [DeepSeek-V2(MLA)] · [DeepSeek-V3] · [DeepSeekMath GRPO] · Qwen2/2.5 · GLM-4 · MiniCPM · InternLM2 · Hunyuan-Large · Skywork-MoE
2025 — Reasoning-model scaling + agentic RL explosion + trillion-scale open MoE
- US/West: DeepSeek-R1 shockwave (Chinese, but ignited the whole industry) · o4-mini · GPT-4.5 · GPT-5 · gpt-oss open source · Claude 3.7/Claude 4 · Gemini 2.5 · Llama 4 · Grok 3/4 · Gemma 3 · Nemotron-H
- China: Kimi k1.5/K2 · Qwen3/Coder/Next · QwQ-32B · GLM-4.5/4.6 · MiniMax-01/M1 · DeepSeek-V3.1/V3.2 sparse attention · Hunyuan-TurboS · Ultra-MoE · MiMo (Xiaomi) · LongCat (Meituan) · Ring (Ant) · Step-3 · Seed1.5-Thinking
- infra/RL: DeepSeek open-source week (FlashMLA/DeepEP/DeepGEMM/[DualPipe]/[3FS]) · Native Sparse Attention · DAPO/AReaL/ProRL · agent RL: Search-R1/ReTool/RAGEN/Tongyi DeepResearch
2026 H1 — Million-token context, trillion-scale MoE going mainstream, native agentic/omni
- US/West: DeepSeek-V4 million-token context · GPT-5.5/GPT-Rosalind · Gemini 3 Pro/3.5 Flash · Claude Opus 4.6 · Llama Muse Spark(MSL) · Ultra · Mistral Medium 3.5
- China: ERNIE 5.0 · GLM-5 · Kimi K2.5 · Qwen3.5/Omni · MiniMax-M2/Sparse Attention · Ring 2.6 trillion · openPangu-Ultra-MoE-718B · Hunyuan 3 preview · MiMo-V2.5-Pro · Intern-S1-Pro · LongCat-Next
- Incremental addendum (post-2026-05 gap-filling + full-paper six-dimension deep dives): GLM-5.2 (IndexShare/1M) · Kimi-K2.6 (1T MoE/agent swarm) · MiniMax-M3 (MSA) · Qwen-AgentWorld (first language world model) · Intern-S2-Preview (task scaling) · MiniCPM5-1B (on-device 1B SOTA) · Mistral-Small-4 (three families unified, 2026-03)