The Cheapest Millisecond
Notes from Chicago on why inference infrastructure stopped being a compute problem I gave a keynote at The Connected World Live in Chicago on September 9th, and I enjoyed it more than I expected to. Twelve minutes is a punishing format. It is long enough to need a real argument and short enough that a single tangent costs you a fifth of it. Constraints like that are clarifying. You find out quickly which parts of your thesis you actually believe, because there is no room to carry the parts you are only fond of. Afterward I joined a fireside chat and then a panel with NVIDIA, Edgecore, and Interglobix. More on that at the end. Here is the argument I made. The question the building has to answer has changed From 2021 to 2024 we built data centers to answer one question. How large a model can we build? That question has a shape. It wants the largest coherent cluster you can power and cool. It tolerates a bad night, because you checkpoint and resume. It books capacity in GPU hours an...