Anthropic CEO Dario Amudi said on the In Good Company podcast that training AI models in development today can cost up to $1 billion. Current models like ChatGPT-4o cost only $100 million, but he expects the cost of training those models to jump to $10 or even $100 billion three years from now.
“Right now, 100 million. There are models in training today that are over a billion.” “I think if we go to 10 or 100 billion, and I think it will be in 2025, 2026, maybe 2027, and the algorithmic improvements will continue to accelerate, and the chip improvements will continue to accelerate,” Amodi added. If the momentum continues, then I think there's a good chance in my mind that by then we'll be able to get models that are better than most humans at most things.”
The Anthropic CEO mentioned these numbers when he discussed the evolution of AI from Artificial Intelligence (like ChatGPT) to Artificial General Intelligence (AGI). “There won't be a single point where we suddenly reach AGI,” he said. Instead, it will be a gradual development where models build on the development of past models, much like a human child learns.
So, if AI models get ten times more powerful every year, we can logically expect the hardware needed to train them to also get at least ten times more powerful. As such, hardware can be the biggest cost driver in AI training. In 2023, it was reported that ChatGPT would require more than 30,000 GPUs, with Sam Altman confirming that training ChatGPT-4 would cost $100 million.
Last year, more than 3.8 million GPUs were delivered to data centers. With Nvidia's latest B200 AI chip priced around $30,000 to $40,000, we can guess that Dario's billion-dollar valuation is on track for 2024. If advances in model/quantization research continue at the current pace, we expect hardware requirements to remain the same as more efficient technologies such as the Sohu AI chip become more popular.
We can already see this growing rapidly. Elon Musk wants to buy 300,000 B200 AI chips, while OpenAI and Microsoft are reportedly planning a $100 billion AI data center. With all this demand, if Nvidia and other suppliers can keep up with the market we could see GPU data center shipments reach 38 million next year.
However, in addition to providing the actual chip hardware, these AI firms also need to deal with the power supply and related infrastructure. The total estimated power consumption of all data center GPUs sold last year could power 1.3 million homes. If data center power requirements continue to increase rapidly, it is possible that we may run out of economically priced power. Additionally, while these data centers require power plants, they also require a fully upgraded grid that can handle all the electrons needed to power power-hungry AI chips. Is. For this reason, many tech companies, including Microsoft, are now considering modular nuclear power for their data centers.
Artificial intelligence is rapidly gathering steam, and hardware innovation seems to keep up. So, Anthropic's $100 billion valuation is on track, especially if manufacturers like Nvidia, AMD, and Intel can deliver. However, as our AI technologies improve exponentially with each new generation, one big question still remains: How will it affect the future of our society?