Digitale Opulenz: GPU-rich

dall-e%203%3A%20%22gpt4%20was%20trained%20on%2025000%20A100%20GPUs%20over%20a%20period%20of%20100%20days%22

dall-e 3: "gpt4 was trained on 25000 A100 GPUs over a period of 100 days"

"There are a handful of firms with 20k+ A/H100 GPUs, and individual researchers can access 100s or 1,000s of GPUs for pet projects. The chief among these are researchers at OpenAI, Google, Anthropic, Inflection, X, and Meta, who will have the highest ratios of compute resources to researchers. A few of the firms above as well as multiple Chinese firms will 100k+ by the end of next year...."

(Patel & Nishball, 2023, Google Gemini Eats The World)

GPU-poors

dall-e%203%3A%20%22alan%20turing%20inventing%20turing%20machine%2C%20cracking%20the%20enigma%2C%20conceiving%20turing%20test%20and%20biting%20into%20the%20poisoned%20apple%22

dall-e 3: "alan turing inventing turing machine, cracking the enigma, conceiving turing test and biting into the poisoned apple"

"Then there are a whole host of startups and open-source researchers who are struggling with far fewer GPUs. They are spending significant time and effort attempting to do things that simply don’t help, or frankly, matter...What can be done commercially in GPU-poor environment is mostly irrelevant to a world that will be flooded by more than 3.5 million H100s by the end of next year...The underdogs should be focusing on tradeoffs that improve model performance or token to token latency...Europe will fall behind in this race due to the lack of ability to make big investments and choosing to stay GPU-poor..."

(Patel & Nishball, 2023, Google Gemini Eats The World)