Actors Give AI Agents Their Own Runtime on Kubernetes
Agent Substrate is Google’s answer to a problem Kubernetes was never designed for. Kubernetes was built for web servers that start once and run for months. AI agents wake up, work for milliseconds and then wait.
Google’s Dmitry Berkovich and Drake Williams examine that shift with Alan Shimel at KubeCon + CloudNativeCon in Salt Lake City. Berkovich is a senior staff software engineer, and Williams is a senior product manager.
Why One Agent per Pod Breaks
Many teams start by running a single agent in a single pod. Tools like Agent Sandbox make that workable at large scale. Problems appear as the numbers climb.
Securing every pod individually adds complexity and creates single points of failure. Cost becomes a wall too. There are not enough VMs in the world for a million always-on agents.
Meet the Actor
The project decouples the agent from the pod life cycle. Each agent becomes an actor, a new construct designed from the ground up for agents.
Every actor gets a cryptographic identity, a secure sandbox and default-deny network policies. No credentials live inside the agent. Teams can still plug in their own gateways and guardrails.
Kubernetes still handles hardware provisioning and metrics. Where it cannot reach billions of records, a dedicated control plane takes over.
Snapshots That Keep Agents Lossless
Williams says common productivity and coding agents sit idle more than 80% of the time. Keeping them running wastes money and capacity.
Agent Substrate snapshots idle actors in three layers: volumes, container writes and memory. A router holds incoming requests, resumes the actor and passes the request through. If the original hardware is unavailable, a cold start still restores files and storage.
Google measures success by agents per unit of compute and cost per agent. Integrations with Gemini and Anthropic’s Claude show it is not tied to one model.
Google has donated Agent Substrate and gVisor to the CNCF. The goal is an open layer at every level of the agentic stack.
Transcript
Hi, everyone. We're back here live for our day two coverage of KubeCon in Salt Lake City. I hope you enjoyed the Techstrong Gang episode we did here with Mike Bizard and Guy Currier and myself, and a few special friends from Mesmo and SUSE.
Our next panel, because I guess there's three of us, features two friends from Google Cloud. I want to introduce you to my immediate left here. Actually, I always worry it gets mirrored on TV, it might be right.
But the guy standing over here is Dmitry Berkovich, right? Yeah. Dmitry, welcome to Techstrong Gang.
Yeah. It's great to have you here. Well, welcome to Techstrong TV.
Gang was last show. Dmitry, tell people a little bit about you, your present position, and maybe a little bit of your story. So I joined Google, I think seven years and a half ago.
I was on GK from day one. Mm-hmm. And most of the time I work on the networking part of control plane and Kubernetes.
And last year, actually, I switched my role and I started working fully AI. Really? Yeah.
I started with Agent Sandbox last year, and the beginning of this year, we started this amazing project, Substrate. We're going to talk today about it. And yeah, it's AI.
Very cool. Yeah. You know, control plane is probably one of the most overused words I've been hearing over the last couple of months.
Oh, yeah, but this, I work on control plane of the Kubernetes. Okay, so... A little different.
A little. It's called, in the past, called master. Now nobody call it master, now it's called control plane.
Yeah. Exactly. Yeah.
Next to Dmitry, I want to introduce you to Drake Williams. Drake, welcome to Techstrong TV. It's great to have you here, man.
Thank you. Give people a little bit about you. Yeah.
Happy to be here. I've been on the GK team for over five years now. Yeah.
I'm absolutely flies in the product side of things. Worn a lot of different hats, but for the last six months, I'm focusing on our agentic offerings. Ultimately, how do we make Kubernetes the best place for our agents, full stop?
I really need to be- Wasn't it already? Yeah, exactly. How do we make it even better for agents, right?
The evolving agentic landscape. Well, I will tell you, we were talking off camera, look, the fact of the matter is, whatever's running AI, when you peel the onion back, you'll find Kubernetes there usually. More than usually, almost every time.
But that doesn't mean that we should stop evolving Kubernetes- Mm-hmm ... stop developing more functionality, better Kubernetes, especially as we move into more agentics and everything else in here. And in that regard, a lot of people think, certainly Kubernetes emerged as the winner, let's call it, of the first phase- Mm-hmm ...
of cloud native, act one. I remember, I guess it was 2016 or 2017. I'm at DockerCon in Austin, Austin, Texas, and we hadn't launched cloud native yet, Cloud Native Now, our site dedicated just to cloud native.
And I went down there looking to see what was this cloud native thing everyone... Or actually, truth be told, I went down to see what was this container thing everyone was talking about. It was all Docker then.
8. 1. 0.
" But I got to tell you the truth, I don't know if that's going to catch on either. It's hard as heck to use. It was really hard.
I can't imagine that people are going to adopt that over Docker Swarm or Mesos or some of the other ones. But clearly, Kubernetes took the market by storm and very quickly established the dominant position. But I don't even know if Mesos is around anymore, any...
I know Rancher is, but they're really built around Kubernetes. Yeah. But now we go into act two, right?
And that's my question to you, right. What happens when we move beyond just microservices? We get to model serving, we get to stateful AI agents- Mm-hmm ...
right? Do we run them on just our regular old pods, or do we really need to change it up? You want to take that?
So yeah, so agentic workloads, they are special. They're slightly different. When Kubernetes was built, it was built towards running web servers.
And web server usually, you create a pod, you spend a couple of seconds, and it can run for months, and nothing going to disturb it. While agentic workload is completely different. We have billions of agents.
Each one of them is like, it has its own life cycle, super busy. It's waking up, run for 200 milliseconds. It can run for a couple of minutes, but majority of them running super fast, like- Yeah ...
one second and they're done, and they wait. And they wait for the next request will come maybe in next minute, next hour, or maybe next day. And it's not exactly what Kubernetes is built for- Right ...
for this kind of really, really fast, come up, down, up, down, and- But, and Drake, I'll throw it to you. Isn't that part of the beauty of Kubernetes, that though it wasn't built for this, it's the best we have, right? What else do we have to put in its place?
And many customers are using Kubernetes at scale today using just pods, using things like Agent Sandbox, one of our open source tools, to run agents in a single pod at hundreds of thousands of scale, if not millions of scale. Problem is, how do you then think about things like security? How do you think about zero trust security for agents?
And then as you start scaling and scaling out, how do we now think about the cost, right? Having one agent running in one VM, sure, that's $30 a month. But if I have 100,000 agents or a million agents running in one VM, there's just not enough VMs in the world to handle that, right?
So how do you start being more efficient with how we run this? I think that's the issue. I think it's a question of optimization.
Yeah. So one of the problem, I'm sorry, interrupting you. Oh, go.
Problem, we talk about the security, it's a tricky one, right? Because security is called I blame the humans. I don't blame the agents.
I don't think we have figured out... Actually, I was reading an article about this this morning. I don't think we figured out how to program these agents in terms of the grinding of mission versus doing what's right.
Yeah. And I'm hoping, especially smart guys like you from Google, we get to figure this out, right? Is there some sort of overriding ethics, for lack of a better word, that overrides mission?
I don't know. I think there's a lot here, and- Good ... one thing I've been thinking about recently is that we're hearing these stories about agents doing bad agent things.
I want to describe it. It's not a random startup that doesn't know what they're doing. These are the frontier labs, the leading edge of knowing how to use agents.
With billions of dollars. And they're still having these problems. Yeah.
And so one thing we've been thinking about is how do we take what these frontier labs are learning, how they're building these primitives and concepts to protect their systems and their users, and make it available for all of our customers to use those same patterns as easy as possible. How do we take those learnings, just make them reusable, right, for our entire customer base? Agreed.
So this is a good time for me to introduce this concept of substrate. Yeah. Right?
You guys, Dmitry, you were on the keynote stage yesterday, and then you- Yes ... did another, I don't know if I'd call it a workshop, but another session around it. Assume our audience doesn't know anything about substrate, though.
How would you describe it? I will describe, so when we started substrate, we just realized we set ourselves three goals. Security was number one, right?
So we don't trust the agent. What we did say, every agent has its own identity. It must have, because it needs to be identified, but it doesn't have any credentials inside the agent.
It doesn't have it, no way escape from the sandbox, and we need to control the ingress and egress. It's goal number one. Goal number two, we defined we need to run for the density, because we have billions of agent.
Majority of them, as we mentioned previously, they sleep most of the time. So even I can create billions of them, only maybe like hundreds, thousands will run at the same time. The rest of them are supposed to be suspended.
The state is safe, and they can be ready to wake up any second, and the speed. Speed is the most important part because in most of the cases, the human is in the loop. " Like, this is a human waiting for the input, and humans don't have patience.
They want answer immediately. Like when I work with Gemini and I want to grab it, ask some question, I would like expect answer like immediately, not like tomorrow. This second.
This is what we built, this is our model to build substrate. And we also believe Kubernetes is great infrastructure, right? It's already proven it can scale, and we say let's try to use from Kubernetes the best what we can do, where it's good, like all the infrastructure, how to create, provision the hardware where agents can run or collecting the metrics we will use from Kubernetes.
Where Kubernetes cannot scale to billions, this is where we actually build our own substrate control plane that actually can scale, like we have our own persistent layer optimized for billions of record. Mm-hmm. And, yeah, we completely built our mini control plane on top of Kubernetes just to run agent as well.
I mean, it makes sense to build on top of Kubernetes, obviously. Yeah. Exactly.
You know, I've been in security about twenty-five years. I've seen a lot of changes in security. But you want to know what hasn't changed is the perception from developers, and even ops people, that security is the anchor, the weight on the thing that slows us down.
" Zero trust had this problem originally, right? If we make everything zero trust, what are we going to get done, right? And it took a long time for that concept to kind of percolate, you know, into the way we build our infrastructure, our AppSec and so forth.
Are the things you're talking about going to slow us down? You mentioned speed. Yeah.
I would propose that it's not just the speed of the AI getting back to me as a human, but am I the human slowing down that agent from doing their work? And how do you feel about that? Drake, we'll let you go first on that one.
So, I mean, I think even taking a step back, like what is agent substrate? At a high level, how I see it, it's decoupling the pod life cycle from the agent. For all the things we discussed previously, most customers run one agent in one pod.
So when we talk about security, everything you do is securing that pod. Right. That creates a lot of complexity.
Q&A is not really meant for that. It gets- It's a single point ... single point of failure.
And so with agent substrate, instead of managing a pod as an agent, now we have an actor. Wow. We're creating a new concept, a new construct to manage an agent.
Okay. And we can design from the bottoms up how do we make that easy to do. So every single actor gets a native SFEE cryptographic identity.
So every single by default will have a secure identity. Every single one will by default have a secure sandbox. That just works.
We have default deny network policies. So like Dmitry was saying, the agent can't talk to anything. So it's all built in.
And so because we're able to create this new construct, we're able to kind of rethink how to make this as easy as possible so it just works. Also giving the flexibility to extend it and build on. If I want to plug into my own open shell, if I want to plug into my own gateway and my mall armor, things like that, 100% can, but we want to give you that base foundation that just works out of the box.
So this is the first I've heard of this actor term. That's great. Because if you think about it, you know, we were talking about act one, Kubernetes now moving in act two.
Look, for me, a big catalyst for Kubernetes' success was the whole concept of operator. Mm-hmm. Right?
" This made it real for a lot of people. Is actor the operator of act two, if you will, right? So actor is the main part of it, right?
It's everything. Everything is around the actor. And to be honest, we spent about, how did we call this kind of unity?
I think at the beginning of the project, we spent like amount months talking like- Hundreds of discussions ... " Like, is it going to be sandbox? It's confusing.
There's too many sandbox. Please, I beg you, not sandbox. Is it going to be agent?
No, but there's also people running sandbox only, right? So we would spend like months, like we have so many proposals, and we realized, okay, actor is the best name. This is actually- No, I like it.
I think it's great. You know, the same way we think now of operator, like duh, of course. Yeah.
I think actor's a great one, a great way of doing it. And every large customer we talk to have all gone through the same pattern. They started off with one agent in a pod.
They realize that breaks because of security, because of scale, and they start building their own abstractions, which looks a lot like what we've built. And so the idea is let us build it once and give it to the community, build it open, so we can all have a shared layer to build on top of. Just like we do with Kubernetes, want to do the same thing with actors and this act two agents.
I like it. Yeah. Let me bring up the next thing.
You know, if you walk around the booths around here, the concept that agents are persistent- Mm-hmm ... durable versus ephemeral, you know, what's the difference between having an agent versus generative AI kind of things- Yeah ... is that the agents can last.
Mm-hmm. However, that creates its own set of security issues and problems and everything else, right? So, is an agent on 24/7, 365?
Does it sleep? Does it get woken up? When does it get woken up?
How long does it stay awake before it goes back to sleep? The whole thing around what do I do with this agent is based on, is something people I think are still trying to get their heads wrapped around. The hard thing is there's so many different types of agents.
You have your coding agents, you have your, like a Gemini assistant type agents, and internally what we found for like the common productivity agents, we'll call it that, or coding agents, over 80% of the time they're idle Which means the traditional model is to have an agent running 24/7. That means 80% of the time you're paying for resources, it's not doing anything. Yep.
And so one realization we had even last year with Agent Sandbox is the ability to snapshot. So take what that agent's doing, take its state, take its storage, take all the context, save it to storage. That way you free up the resources for other agents, then start using it.
That's one of the core beliefs we have at Substrate, which is every agent needs to be very easily snapshotable, so we can suspend it and let other agents use it, and really drive that multiplex, drive that efficiency and gains of the underlying infrastructure. So let me geek out on you a little bit. Yeah.
So I've been following for a couple of years, you know Dapr? Yeah. I'm sure you guys know Dapr.
You talk to the Dapr people, their community, they'll tell you a snapshot alone is not enough, right? That in order to wake something to a stateful agent and everything, it can't be just a snapshot. There's more to it.
Otherwise, you're liable to lose some context. Yep. So the file system, the memories, and things like that.
I can talk about it. We actually built the entire life cycle around it. So- There you go ...
so when you take a snapshot, yes, well, it depends what the snapshot is. So what we defined, we don't just take a snapshot of everything as is, right? Because can you restore it later?
What if your hardware has been changed? Mm-hmm. You will not be able to restore it as is.
So actually what we did, we took the sandbox of the actor, and we split the snapshot in three different layers. Ah. We have a bottom layer, is all your storage, like all your volumes.
It can be external volume, or it can be volume like you do Home-Dir, for example. We just separate it. My Home-Dir is always going to survive between resumes.
Mm-hmm. On top of it, we say the next layer, we talk about what container has. Container can be OCI image, right?
But user can write to temp folder or install like do apt get and install kind of stuff in the second layer of snapshot. And third layer is memory. So when we detect an actor is idle, it actually takes a snapshot of all three of those, and we slowly, in the background, upload into the durable storage.
Mm-hmm. And when the next request is coming, because actor doesn't have a direct IP. Right.
So your only way to reach the actor, we have an egress gateway, sorry, ingress gateway. Ingress. Router.
Request come into the router, and router check at the beginning, is the actor is alive or not? If it's not alive, it holds the request, it resume from the snapshot, and pass the request. And when we do the resume, we actually look on which hardware we land it.
We try always to match the exact same hardware so we can restore all three layers. It's going to be immediate resume because it just came exactly where you store. Exactly, yeah.
But if hardware doesn't exist and specific O is busy, okay, so we will do cold start, but we still have all your file system and on top of OCI image. But what if you detect one of the OCI image, you detect a bug and you release new version of image? Okay, we can restore it, but now we cannot overlay layer, we cannot restore because it's changed.
But your volume, your home directory, data storage is survived, and we can always restore. So actually, you're always coming to the same state. Right.
And this is how we promising you can resume. Almost lossless, if you will, right? Yeah.
If you build it correctly, it's going to be lossless, yeah. So, and there's a couple reasons for doing this. One, I think, Drake, you hit on is the money.
Mm-hmm. Economics of it, right? Why have all this stuff stood up there sitting idle 80% of the time?
The other piece of it, though, is scale, right? If I don't have to run everything or every time, all the time, I'm able to scale. It makes logical sense to me, right?
That I should be able to scale much easier, much better, much more efficiently by running what needs to run when I need to run it. And that is something that if each agent is running in its own pod, you'll never get. Yeah.
You'll just never get to that point, right? Because each one is its own world for- Think about two metrics internal, our benchmarking, we kind of look at our success, it's agents per compute. So if you have one CPU, if you have one unit of compute, how many agents can I stick on there?
And the flip side is what's the cost per agent? Yeah. Two sides of the same coin, but very, very important.
One is how much can I scale? The other one is like, what are the actual products I can build? Right.
How can I actually scale this? And that, we talk about human in the loop. That's a human decision, right?
I could go like this here. Which way am I going with it? I want to ask you guys something else, though.
I got to assume, you're Google, you're Google Cloud, you started optimizing- Mm-hmm ... working with Gemini. It makes sense, no?
Mm-hmm. So my question then is, how does Substrate play with Gemini managed agents? And then, I'm sure you didn't just play Gemini.
You had to make sure it worked on Claude, and Chat, and maybe some of the open weight models, and all of these things. Talk about how that works. So all this kind of pattern that Gemini or other companies, they have something called harness.
They can run the loop, right? Yeah. In the loop, they go to LLM and decide what is the next step.
And the next step is you're supposed to run command somewhere, right? And you need this kind of sandbox. And a lot of customers, they think, "We want to have it like running on my computer, my VPC.
Like I have my environment, I have-" Access to my data, my environment. "To my data. It's super sensitive.
I don't want to host it there," right? This is where come integration between all the Gemini agents, where they execute the sandbox on the Substrate. And they build Gemini to run billions of agents.
Right. And we actually need the same infrastructure, how you can run billions of sandboxes, right? Yesterday, we actually released open source, how we can integrate a Gemini agent integration with the Substrate.
Like we just create a bridge, and the customer just needs to install this bridge on their cluster. Mm-hmm. And it just works.
Works natural. Just naturally works. Is that only with Gemini or with third party agents?
Yesterday on the keynote, we actually showed the similar example how we did it with Anthropic. With Claude? Yeah.
So it was- So it's not, because you know what people say, "Oh, sure, they're Google. They're just working with Google," right? But it works- We released yesterday both of them ...
with third parties. Sorry, third parties. And when you say, oh, you released them open source, were they donated to CNCF or Linux so they could just- Um ...
come out under an open source license? Good question. If you know, know.
I don't know that. It's in a repo anyone can access? I don't know.
Yeah. If donation, don't- Something like- I don't know, don't ... GitHub or whatever?
Yeah. It's on GitHub publicly available. Yeah.
That's all that counts. At some point, people worry about managing it going forward and stuff like that, and you see which way it goes. Yeah.
But that makes sense to me as well. You know what, Dmitry, in my notes here, it says that Agent Substrate and gVisor- Mm-hmm ... were donated to the CNCF alongside launching GA on GKE.
Yes. So maybe, it sounds like they maybe were. So- Or is that not the bridge you're talking about?
No, it's not a bridge. It's something else. Different piece you're talking.
It's actually the sandbox environment around your actor, because this is where you do the sandboxing. And there's multiple technologies on the market. One of them is MicroVM, and we have Cloud Provider, we have Firecracker, and gVisor is same MicroVM.
They work slightly different each others, and gVisor one from the Google that was always like, it was open. It was open, but it was not donated to CNCF. It was managed by Google.
And I think we decided last week we will donate it to CNCF together with Substrate, and it will become fully open to open to community, not just owned by Google. And the broader idea is there's a growing agentic stack. As we talk to all of our customers, there's different layers you need to build.
You need the compute, you need the sandboxing technology, you need substrate, which is the actual runtime. But then even on top of that, you have things like open shell for your agent- Sure ... and your governance, right?
What we want to do is at every layer of that stack, there'll be something in the open. And we want customer to have a full end-to-end experience. They can use that, just works out of the box, but also have the full flexibility to kind of plug and choose, swap out in case they want.
Excellent. Gentlemen, thank you so much. Thank you.
I appreciate your time. Pleasure. It was great.
You know what we didn't... Wait, before we go, people want to get more information. Where should they go for this?
0. Ah. Mm-hmm.
This is, it's very important. Now everyone can just take it out, try it out, fully support it. dev.
dev. Go and try it out. There you go.
I don't know how good the focus is- Yeah ... or if that'll be sharp enough, but it's- Or you just Google Substrate GitHub, and you'll find the... dev.
Yeah. Very cool. Drake, thank you.
Pleasure. Dmitry, a pleasure. Thank you.
Hey, we're live here at KubeCon. We're coming back in a minute.