What was the power supply on a 1984 Mac? Or a 1984 PC? For the Mac, complete with built-in CRT monitor, it was about 60 watts. For the PC – a heavy 135 watts. A power supply for a reasonable gaming computer now is 600-800 watts. The graphics card alone can chew through a couple of hundred watts. A high-end gaming PC can have a 1 kilowatt power supply. But that’s nothing. A single rack of modern servers can be 30-40 kilowatts. That’s still nothing. Modern AI racks are in the 100-200 kilowatt range and moving to 500 kilowatts.
People just throw around a gigawatt of computing capacity as if we’re talking about something we regularly buy. But we don’t really have any history of building gigawatt data centers. We’ve come close to that by adding on to existing data center complexes. But if a complex of regular 40 kilowatt racks is replaced by a complex of 500 kilowatt racks, you will need to feed it a lot of power.
What do I mean by kilowatt? Well, it’s the amount of electrical power that needs to be fed to the system for it to do work. If I say the machine takes 150 kilowatts, what I’m saying is at *peak* consumption, I expect to see 150 kilowatts consumed by the machine. Below that, it won’t get enough sustained electrical energy to keep computing correctly and producing results. If it draws about 150 kilowatts (which it might for a very short time), I expect it to start slowing down to keep from dangerously overheating. Because these computers cost a lot of money, we want to make sure they are 100% busy all the time, so peak workload is synonymous with actual requirements. The machine could have multiple $50,000 chips in it, and several machines per rack.
Where does this electricity come from? Ideally it comes from the local grid, but more data centers are using on-site turbines to generate power because the local grid can’t sustain their usage. A 150 kilowatt rack (which measures a square meter, give or take) consumes as much as 15 houses. The average power plant in the US produces 50,000 kilowatts. Some are bigger, with the largest ones in the 2,600,000 kilowatt range. That means most power plants in the US cannot hope to power even one gigawatt (1,000,000 kilowatts) data center. You need half the output of our largest power plants or several power plants to feed just one data center. 1,000,000 kilowatts is 100,000 homes. If there is a gigawatt of new demand coming to your local utility, I can almost guarantee they’re not ready for it. The compensate by spinning up excess capacity that’s more expensive to run to supplement their current output. Or they buy power from another part of the grid, which is often more expensive. That’s why your electric bill goes up.
What happens to all that power? Where does it go? The answer is it’s expended as heat. A kilowatt of input power has to be expended as a kilowatt of heat. That’s why data centers have massive cooling requirements. The most efficient way to move heat is to use water to absorb the heat. Air is also used, but as you move beyond a certain point, you need liquid cooling to cool the chips. But is the water run directly over the chips? No, a closed loop liquid cooling system transfers heat from the chips. That goes through a heat exchanger which is cooled by water. A standard 30-40 kilowatt rack produced about 5 times as much heat as one large residential heat pump unit. An AI rack in the 150 kilowatt range produces about 25 times the heat as a large, residential heat pump. A newer 500 kilowatt rack produces just under 90 times that heat pump. A 1,000,000 data center would produce as much heat as 170,000 homes on a cold winter day. (Even more if you go with an average heat pump size).
As you can see, just venting out that much heat is a serious problem and requires a lot of water. In the space of about 2 square meters, a 500 kilowatt AI rack produces the same heat as 180,000 square feet of single family housing on a cold winter day.
The exercise above was to give you an idea of exactly how much energy and heat we’re talking about with a gigawatt of data center capacity. Now let’s look at what the resources are for a 10 trillion parameter model, like one of the frontier models. Let’s say each AI chip has, attached to it, 128 gigabytes of memory. A 10 trillion parameter model needs on the order 80 of these chips. Each chip costs around 50,000 USD and consumes on the order of 1 kilowatt. Let’s say the task requires 60 seconds of compute. The rough cost of that query is $4.50, without considering electricity, cooling, water, or anything else but the amortized hardware cost. That’s 80 machines, at 50,000 a machine (4,000,000) divided by the number of seconds in a 5 year life-span (about 160,000,000) times 60 seconds. What should Open AI charge you for that? I don’t know but $20 a month should only buy you a couple of minutes total usage.
When I look at AI, it’s not that I don’t think it doesn’t work. It does, although it’s hard to say if it actually translates into productivity. For that, each minute of compute time on a frontier model should cost about $5. That does not include training time. And the argument is the execution (inference) is much cheaper than training. The $4.20 above does not include other costs beyond training a new model. Maybe, it should cost $10-20 a minute or more for a profitable AI company to answer questions. And it’s not at all clear that using a smaller model (let’s say one 1/100 the size) doesn’t just require you to ask many more questions, as you tweak and re-tweak the question. Or just abandon it entirely. Or just run it for 100 iterations.
It’s not that I worry that AI doesn’t work. Although there is an argument its effect on productivity is a wash, it’s that it doesn’t work on the economics. At least with frontier models. And some of the smaller models require frontier models, with their many trillions of parameters, to be trained in order to ‘distill’ the smaller model. And if everyone is just using the distilled models, no one will want to work on the big frontier models. Which presents its own catch-22 for progress. But imagine this, the company that screams bloody murder to get you a second monitor, will now gladly spend that much a day to make as productive as you are today?
I wish I was like you/easily amused 