Core Mechanics
- High-performance GPU clusters (A100s, then H100s) with dense NVLink interconnections
- Massive VRAM requirements (16GB+ per GPU, thousands of GPUs)
- Training-focused architecture optimized for parallel matrix operations
- Capital-intensive: billions of dollars for single frontier model training runs
- Chinchilla scaling laws (DeepMind, 2022) showed optimal data-to-parameter ratios
Key Players (2018-2023)
The Constraint
Diminishing returns. By 2023, doubling compute yielded only 0.5-1% performance gains on benchmarks. The cost-benefit curve bent sharply. High-quality training data was exhausted. Simply making models bigger stopped being the winning strategy.
Transition to Phase 2
When marginal gains from pre-training plateaued, labs pivoted to post-training refinement. Instead of training from scratch with more data, they began fine-tuning existing foundations with human feedback and curated datasets.