Groq Builds the World’s Fastest AI Inference Technology | The Six Five Summit

Hey everyone. Welcome back to the six five Summit. Daniel Newman here, CEO of the future of group.

It's day two. We are in one of my favorite tracks. Chips weren't always cool.

Semiconductors had a period of time where no one wanted to talk about 'em, but yet I said Silicon would eat the world. This goes back to 2019. Didn't even know that we would see the AI trend in the exact form, but I was pretty sure accelerated computing was gonna change the world.

Today. I'm joined here during day two by Jonathan Ross, CEO of Grok. Jonathan is no stranger to the six five.

He is been on for several years because it was several years ago that I began working and we began doing advisory relationship with, uh, grok. And also it was at that time that I said, this company is doing something pretty fantastic. And that's when I decided to join the cap table, join the ranks of the grok and, uh, of Grok and its leaders.

And I had to put that out there because that disclosure was important. But from the time I met you, Jonathan, I knew you were onto something special. I'm glad you're here back at the six five.

Congratulations on all the success. Well, thanks for betting on us and hello everyone. And, uh, really excited to talk a little bit about LPs and what we're doing.

Yeah, no, and, and, and listen, you, you made it easy. Um, the technology is hard, but the way you described it and the way you have committed to doing something special has been really easy to get behind. Um, you know, it's been incredible.

Um, I won't spoil your riches, but you've seen your community grow a little bit. Tell me a little bit about like, you know, gimme the last six months or so, what's been going on at Croc? 'cause all I can say is it feels like the momentum is building Actually.

Yeah, six months ago, I think is roughly around the time when we first demoed what we had. And then of course, demoing is one thing, getting it on our website so everyone could use it was another. And we start, we, I think we made it available to developers for the first time.

About 11 months ago. We had a closed beta. We had fewer than 10 developers, or sorry, 11 weeks ago.

And then in 11 weeks we've gone from fewer than 10 developers to over 208,000 developers. That's a, that's a massive number. And I've watched it online.

I've shared a few of these demos. Sometimes I'm blown away, you know, I gotta step out of the role as advisor, step out of the role of investor. And I just watched this from afar, and I see you demonstrating these tokens, you know, these side by side inference demos.

And I'm just like, wow. It's not even Us though. That's the beautiful part.

It's the community. Absolutely. You've basically said, here's the here's the race car, um, by the way, bring your own steering wheel, bring your own arrow, bring, you know, and you put it on the track.

You put the, you know, and, and I'm watching this and I, it is just incredible. And I mean, look, generative AI is like, it's definitely a megatron, but grok is taking a bit of a different approach, Jonathan. I mean, you are basically looking at all, you know, yes, GPUs are a big thing and, and there's some different, accelerators are a big thing.

And by the way, you can do some inference on CPUs. It's actually probably still where most inference is done today that's changing. But like, talk a bit about rock's approach to generative AI and how you're thinking about infrastructure to enable this trend to really continue to grow and, and, and be consumed.

And the experience to be good, Be good. I think one of the big bets that we made early on was that speed was going to matter. I think most of the in, uh, sort of the inference architectures out there, infrastructure that's being built, were all targeted on, uh, what's called a batch, which is let's do a single memory read and do as much computation as we can.

But that slows things down. Now because inference requires a low cost, you have to come up with something very different. If you wanna have both speed and cost, we did that.

And so by the end of this year, we're actually going to be deploying over 25 million tokens per second. Everyone at Rock carries one of these challenge coins to remind us. And when we get into debate about anything, we just plunk that down on the table and we say, what is, what helps get this number?

And so that's an amount of tokens per second that a hyperscaler would have. In fact, uh, this is where one of the hyperscalers ended last year. So that's where we're gonna be the end of the year.

We also focus a lot on quality. Um, the reason is with language, it's not just about, uh, unlike an image where it pixels off a little bit. Yep.

Um, You know, you have to get the exact answer. The difference between should and shall and legal contract is all the difference in the world. You can't have a slight difference.

So we actually came up with this technology we call True Point, which is, uh, an FP 16 numeric, but it actually gives you the correct answer unlike normal floating point. And so our quality is higher, our speed is higher, and our cost is lower. We're also lower energy.

So all these things at the same time typically aren't possible. And we had to trade something off. And what we did was we didn't focus on training, we focused on inference.

And that's how we were able to get all of these things at the same time. And I remember you took a pause there, 'cause I do remember early on we had some conversations and I think, and I, and I'm sure deep down the engineer in you still sort of believes like you could take the training challenge on, but I think somewhere along the lines, you came to the recognition. I'm not putting words in your mouth, I'm just saying I, having spent some time with you, I'm sure you have ideas of how we could make that better too.

But at some point in time you kind of came to the recognition, Jonathan, that you're like, look, we've got something really special here that could handle and all these generative outputs. And I'll tell you, you said two things that three or four, but two or three really important things there. The first thing you said is accuracy matters, right?

So these, these tokens, these responses that you get, um, you know, a lot of people get really excited, but then they actually verify what they got. And you realize you got a lot of junk, you got a lot of crap in there. It's hallucination is sometimes the word that's used, but accuracy has a pretty wide con continuum.

Um, the second thing is you do talk about like the fact that when you're generating things for business that the words matter. You know, um, you should not use a LLM that's gonna give you bad results and you shall pay the piper if you do. Um, every word matters.

Yes. And every word that I get, by the way right now is not particularly good. Third thing I'd like to double click on, I'd like to get your take on this as energy.

Mm-Hmm. So I've been reviewing a lot of, of, of analysis in the market, and one of the things that has come to a, a very evident conclusion to me is we are going to run out of power very, very quickly is if we continue to deploy ai, the way we are deploying ai, these racks, they're not 15 kilowatt, they're 20, they're 25, and it's growing every single day. And, and not 1525, it's way higher.

Now we're seeing 80 120 kilowatts per rack for GPUs. Well, what I was gonna say is when you actually get to these 72 GPU black, I mean it go, but what I'm thinking it was 15, that was 20 and it was, so we're progressing now, okay, to your point 75 80, how do we, how do we deal with this? People are building to half a megawatt racks now.

That's what's being designed in preparation for where GPUs are going and it's unsustainable. So That's the question, what do we do? What do you think?

Also, A single GPUA GB 200 uses twice as much power as your house. Think about that. That means that if you wanna deploy A GPU, you have to choose between that GPU or two houses.

Um, yeah. So that's, You like your house cold in the summer. I, I like to keep it nice and cool.

Well, you can keep it nice and warm if you put the GPU in it. Um, so, so, um, that's not sustainable. And one of the the things is for training, you need this thing called HBM High Bandwidth Memory.

Yep. And, um, in terms of financial analysts, uh, you know, what you're looking at in terms of the market, every single bit of HBM that's made is gonna be sold and it's gonna be sold for GPUs that are gonna be used for training. People are trying to repurpose these for inference.

The problem is your bottleneck on your performance based on that HBM, either because you can't read the weights in fast enough or because you can't read the sequence length in. So if you have a really long context length for an LLM, it slows down if you're reading it from external memory, if it's already inside the chips, it's instant. And that memory burns power, we burn, uh, or A GPU burns almost as much energy just reading a single, just the, the weights in the context for a model as we burn in total for the entire end-to-end computation.

But then you've also got the system overhead, the networking overhead. So we're looking, I, we've heard people talking about hundreds of gigawatts over the next couple years. It's insane amounts of power, But it's like you said, it's not sustainable.

There's gonna have to be an answer for it. Rock it obviously still requires some power. It's not zero, but it is substantially less.

Yeah. Or 14 nanometer chip is between one third and one 10th of the power versus the, um, uh, latest GPUs and the GPUs that are coming out in 2025 versus what we're gonna have in 2025, watch this space. Um, we're gonna pull ahead further to at least five x better on energy, uh, but potentially much more So for everyone out there.

I mean, effectively what he is saying to you is you could have either one fifth as much power used in to get the same amount of inference that you would require. Right. Is that, And this is important, I I don't want to ding GPUs just because Yeah, no, no.

It's like the amount of power that they use. It's high, but that's a little misleading. What matters is what is the energy per token?

How many joules of power are burned to get that word? Because if you are burning, um, uh, something like, uh, you know, 3000 joules per second of power, that's actually enough energy to, um, power, uh, uh, you know, the amount of force that it takes to lift a adult male off the ground, that's an insane amount of energy. So if you're producing 3000 tokens a second, that's like what an airplane needs to keep you alt.

That's insane. So, but the amount per per GPU doesn't matter as much as that per token, and that's what you should be asking people about. Yeah, well, it's, it's a big consideration.

It's probably underused, I've said this on a few different conversations here and elsewhere, uh, Jonathan, is that it was interesting when we were pre generative ai, every technology company was kind of waving this big sustainability flag. All of a sudden we see the gold rush, we see the opportunity, and it's kind of like these are diametrically opposed ideas because in order to scale ai, everybody's gonna burn more carbon until we fi unless we figure out some really, you know, magical solution that right now is not evident. The only thing that is evident is trying to use the most efficient compute architecture for each workload is a good idea.

Because when you use really inefficient ones, you burn a lot more power unnecessarily, which is, I think where your point is, it's not a bad, like for the training workloads, GPUs are the right architecture For Inference though. When you're using a this, this certain architecture, it can be very inefficient for power for what you actually would've needed to do. That same thing For every token that you train on, it is more efficient by a multiple to do that on A GPU.

So we strongly recommend that when you were doing training, you do that on A GPU, if you're doing fine tuning, if you're doing inference, that works much better on an LPU and that's where those multiples come in. Um, but I wouldn't be surprised if at some point you see someone pull a GPU from a rack somewhere and insert an LPU, but they're not gonna then store that GPU, they're gonna move it somewhere else where someone's doing training. Alright, Jonathan, I wanna put you in the CEO chair here.

You've built quite a team. You've made some great, uh, expansion ex, you've grown your developer ecosystem. There is clearly a ton of demand, 208,000 developers to, you know, today as we talk, who knows, by the time we actually talk again where this is gonna go, what does the future look like for Rock?

What are you focused on? What are you thinking about? What do you want your community thinking about when they think?

Well, um, there's been over 80,000 different API keys that have been generated that are active. So that means that that's roughly how many applications and people have been developing things that need speed. And the engagement has been going up.

And this is something that not everyone is prepared for. When they built something on grok, we had this one person tweet that they had this story writing app. And when they switched from GPT-4 to LAMA three 70 billion running on grok, the average, uh, engagement screen time or whatever the time while, uh, user was using it went from 18 minutes to 31 minutes.

So, LAMA three 70 billion is a great model, but it's not quite at the quality of GPT four that tells you that speed was the, um, thing that increased that engagement. Well, what does that mean? It actually means you end up needing more compute.

So the faster the compute is, the more people want tokens, and the more important it becomes to, to have that cost be lower, otherwise you're not gonna be able to afford it. So as you're building these apps, keep in mind that you first build the app that you, you want working, but then once you get that speed, you're gonna be using way more compute than you ever imagined before. And we're here to provide it 25 million tokens per second One.

That's a bull by the end of the year. Here's A one 25 million token pass from Jonathan to all of you. Free.

I'm just Kidding. That'll be One second, one Second, y'all. Listen, this is happening so fast.

And Jonathan, I wanna congratulate you on the tremendous progress. You know, I always, something I always say to people is slow at first, then all at once. You know, and sometimes when you're building, you know, it can never happen fast enough, but you never quite see exactly when that inflection happens until you're looking back at it.

And right now the question is, are we looking back at the inflection or is that just the beginning? We're still slow and all at once is still yet ahead. But either way, it's been amazing to watch the ride.

I'm proud to be part of it. I'm, I'm, I'm telling people out there that this is definitely a technology that you need to put your eyes and your hands on if you have not yet. And Jonathan Ross, CEO of gr thanks so much for joining Six five Summit.

Thanks for having me. Alright, everyone. Jonathan Ross was back not his first time.

Hopefully not his last time, but Grok is definitely on fire. Appreciate the chance to talk to all of you. It's day two here at the six five Summit.

Now I'm gonna kick it back to the studio. Stay tuned. Plenty more coming your way.