Karpathy Demonstrates Massive GPT-2 Training Cost Reduction

Andrej Karpathy reports that training GPT-2 originally in 2019 used 32 TPU v3 chips for 168 hours (~$43,000) to reach a 0.256525 CORE score. He says recent improvements merged into nanochat (from modded-nanogpt) now achieve a higher CORE score in 3.04 hours (~$73) on a single 8x H100 node, representing about a 600× cost reduction and an estimated 2.5× annual decline in cost over seven years.