Micro VMs Secure Agentic AI Workloads

Hey guys, thanks for the throw. We're here with Barry Cooks, who's vice president of Compute Abstractions at Amazon Web Services, and we're having a little chat about, well, the rise of micro VMs. There's a new animal in the cloud computing zoo.

Barry, welcome to the show. Hey, thanks for having me. I really appreciate it.

The concept of micro VMs has been around for a little while, but the nature of the workloads seems to be shifting, and what used to be maybe something that was esoteric is moving towards mainstream. What's going on here, my friend? Yeah, I think that's exactly right.

It's not a new concept. It's not new technology. In fact, it's the thing that powers Lambda at the back end.

It has for the last decade. So what we've observed is the real shift is this agentic AI flow, and you have a lot more folks who have two fundamental concerns. One is always latency and how fast can I fire something up and get it executing.

Two is there state associated with these new workflows that, say, in the Lambda world, you wouldn't have had state. And I think the third, and probably most important, is trust. These are untrusted execution environments versus trusted execution environments, and so you really want to be able to have some tight controls over security boundaries for these workloads because it is running code that you did not write yourself, and you may not want to have running loose on systems next to code that you do trust.

So to your point about that, one of the things that I've seen to observe with these AI agents is it's very hard to predict what kind of workloads they're going to kick off, how much data that they're going to need. So is that driving us towards serverless because we need infrastructure that can kind of spin up and down to support these AI agents? I do think that's part of it.

I think, if it's an unpredictable workload, that is the sort of great use case for serverless because I just spin it up on demand, I execute for as long as I need to execute, and then I let it go. And the trade-off would be I'm going to go run a persistent VM. Well, if you don't know what your workload characteristics are, you're going to be paying for idle time.

And one of the nice things about the micro VM model is you don't do that. You can burst up. In fact, we allow you to do a 4X level at burst, because we recognize that it's difficult to understand exactly how large of a workload you're going to land, and you tune that over time, and that way, you're only paying for what you're actually using.

And it's a much better economic model for most of these use cases. The second thing is, and you touched on this as well, but AI agents, let's put it this way, they're aggressive. They are designed to accomplish their mission by any means necessary.

And we talk a lot about guardrails and controls, and I'm sure that's all good for its own sake, but is there a way to think about securing these AI agents almost by design at the infrastructure level, and that's where we need to get to? Yeah, it is a very complicated topic because there's lots of layers to the security models associated with these, of course. And I think that what we're trying to do with micro VMs is say, look, one of the most foundational ones is going to be to basically lock it in a room, and it's not allowed out of that room.

And that's what we're doing. And so micro VMs gives you these hardware-level security protections for running the agent within that domain. We think that's a really good model for these untrusted environments they run in.

It doesn't solve the fact that if you're trusting that AI to write code, that you're then going to take this without looking at it, go run it in some other production environment, the code that it's generating may not be what you expected. And so you still own the business logic that's getting generated in there. But we're going to give you strong security by default, so that in that runtime environment, you reduce the likelihood of these agents kind of running amok, especially with side-by-side workloads.

There are multiple serverless computing frameworks these days, so what ultimately will drive organizations towards one versus another in this agentic AI era? I think that the classic problems still apply. Am I going to get the scale that I want?

Am I going to get the speed I want? And do I get the security that I want? And those three things are sort of our bread and butter at AWS.

So one of the things that I really like about our micro VMs in this world is that it's running 10 years of incredibly aggressively scaled, secured, and tested code. That's how we run Lambda today. When you run a Lambda function in today's world, you are running on the same back-end technology that we've been pushing over the years to get it to scale and performance may be as secure as we need it to be to run those workloads.

So you take full advantage of that capability set, and I still think back to basics, those three things really matter for a lot of these workloads. What is the relationship between cloud native and containers and Kubernetes and serverless computing frameworks is going to be because we've been on this kind of journey, and then agentic AI came along, and it's a different kind of workload. So I think a lot of folks are kind of confused about what to do, when, and where.

Yeah, I was laughing because my title of Compute Abstractions, that's what that is. It's all of those things wrapped into one group where we're attempting to give people some better clarity, give them the tools that they need. And the classic problem with these abstractions is that you have a very broad domain of problems people are trying to solve.

Even if you just take the agentic use case, there's lots of things I can use agents for. " But it's a really wide range. And so for each of these kinds of abstractions, you're just making different sorts of trade-offs.

And so, at the top level, we're aggressively pursuing simplicity. That's what we started with Lambda. That continues to be something we think is incredibly important.

The easier it is to just run a workload on AWS without having to think as much about AWS, the better for one class of workload. As you get bigger and bigger in scale, and you get into these very large, complicated enterprises, you start to see them wanting to take advantage of underlying things that they don't want you to abstract away because they have enough complexity that they need to have those tweaks, those knobs they want to be able to pull. And that's where you start seeing things like Kubernetes come into play and ECS coming into play where I've got enough scale and enough persistence of workload that now I really do want to be able to tune it in my own ways for my own workload at my own level.

And so you end up with all of them being required. And the key here is helping people to frame the problem they're solving into one of these buckets to understand. And micro VMs, as we've launched, is just a new frame within this space.

It's kind of a tweener in these, but we think it's super important in the agent space in particular, partly because of the characteristics of incredibly fast startup and the fact that this supports persistent workloads. Traditional Lambda would not be for persistent workloads, and this one will. And that's going to be a key differentiator here to get people onto the micro VM strain.

And all these things are somewhat interconnected. So, historically, let's say I had a cloud-native application running on Kubernetes, and it was typically being accessed by a bunch of end users, AKA humans. But if I go forward here and now I've got AI agents that are essentially invoking a bunch of headless back-end services running on Kubernetes, that too is going to drive a level of scale that we hadn't previously considered or imagined.

And so are we prepared to absorb that? Because all these things, they have dependencies on each other, and you push on one, it tends to pull the other. It does.

The complexity of the solutions being built is only increasing. As underlying compute performance improves and you keep layering more and more abstractions to make it easier to build more complicated things, which means people build more complicated things. And so understanding how that is going to play out is a challenging environment for folks.

We spend a lot of our time focused on that scalability parameter and how do we help people to be successful? Because what you don't want to do is come and hit some ceiling and then have to rethink everything you were doing, especially when you introduce AI agents into the framework. At that point, you don't have a person that's traditionally got their head wrapped around the entire problem domain.

You've actually been parceling this out, and understanding how you're parceling it out and how that is going to kind of take you along a journey is an incredibly important thing. So that journey, what does it look like? Because I can also see in my mind that a lot of these AI agents will be running on an edge computing device, some sort of phone, whether on the network edge, wherever that's going to be, and then they're going to be accessing services in the cloud as well.

So how do I think about my architectures and my abstractions in that world? Because it's going to be highly distributed? It is.

And I think if you were to peel the onion on existing problem domains, they're already pre-distributed. And so I think one of the key points in this shift that we're sort of in the middle of is understanding the role that AI is playing in that shift. And so what we've seen, and we talk about quite a bit here inside of my teams, is who's your customer?

Because it used to always be a human. Either they were an operator or a developer, and increasingly, we're seeing that the customer is now being sort of a bifurcated view. There's the human who is overseeing things, but there's also these AI agents that are making critical decisions around APIs and around scalability decision points and coding decisions.

And that's something that is a new vector that we're trying to ensure that we're providing the right guardrails for and the right capabilities for. And so you see a lot of effort around how do we provide the right sort of skills in terms of MCP capabilities and things to help the agents make smart decisions in the environment. That's a key facet of success moving forward, is ensuring that the agents have the right information, that it's current information, that it has our best practices built in.

So that's a big focus of ours. You've been doing this a while, and you see folks playing around with these agentic workloads now. " Yeah.

The temptation with new technology is you get this spectrum of people, some of whom want to run ridiculously fast, and some of whom are like, "I like typing at the keyboard. " So you've got this kind of broad spectrum, and I think both of those ends of the spectrum have their trade-offs. Like you're going to get left behind if you're not attempting to adopt and understand the tools.

They're here to stay. And if you get way out over your skis, you're going to make a pretty sizable mistake. And that balance at that other end of the spectrum is something that we've spent a lot of time on.

And one of the things that we talk about internally is whoever wrote the code, if you checked it in, it's yours. So you need to understand, if you're leveraging these AI tools and you're working through these things, you need to ensure that you understand what you've asked it to build and what it has built before that gets out the door. " And so we're putting a lot of rigor around how do we do that better?

And actually, not surprisingly, we're using AI tools to help us in that as well. And so you kind of have layers of AI there, but you get much better results when you do that. How do I roll something back?

And I'm asking the question because as one wag once said to me, "It's one thing to be wrong. " And when that happens, how do I kind of roll something back that's been occurring at machine speed, and how do I think about that or frame that? Yeah.

So one of our principles, and we have these very large operational meetings every week in AWS to look at these kinds of challenges, to understand how our different teams are solving problems and what problems they didn't hit, that they avoided because we exchanged information across teams. One of the big challenges is exactly that, which is when you're doing a change a day, just to make up a number, rolling that back is pretty easy. I can page someone, they can go look at what happened.

They can quickly figure out, like, clearly I broke something last night. If you're doing 500 changes a day, it's a whole different ballgame. The magnitude of impact and layered impact becomes significant.

And so this notion of improving testing, improving auto rollback, we have to be able to roll back automatically on first detection of an issue because that's the safest thing to do. And we have to ensure that that safety capability is in place, because otherwise you end up in a scenario where 500 changes have gone in and some human operator with AI assistance probably is now trying to unwind that entire stack to figure out which of the 499 are okay and which was the bad one, and how do I get from where I am now back to unwinding that stack? You don't want to do that at 2:00 in the morning.

You want that to be an automated process where the rollbacks happen. So it's a good hygiene. Like you need solid testing.

You need to have the right propagation mechanisms to catch things as quickly as possible. And so we built a lot of those frameworks over time at AWS, and now we're optimizing and leveraging them more than ever. How will these microVMs evolve?

And I'm asking this question because well, we've had VMs, and now we have smaller VMs, and that's great, but the workloads keep getting smaller and more finite. So will there be sub-microVMs? How far does this go?

Yeah, it's a good question, actually. I think you can slice microVMs into really small sizes, and at some point you're correct, you'll hit some kind of limitation. The way that we structure things today with microVMs, which is going to be an interesting thing to see as these applications evolve, is in the reverse direction.

So microVMs are very fast, and they are very secure, and the security comes in layers because as we always talk about defense in depth. So there's a bunch of stuff that is not in a microVM that a general purpose application might need from an operating system perspective. We don't expose those things.

We intentionally don't do that because one, we're much faster execution-wise, and two, we have a much smaller attack surface area by excluding some of these things. But those decisions will get challenged by customers over time as their workloads become more complicated and start to grow and they want to use different things. I suspect, and we'll see how this plays out in the coming months.

I suspect we'll start to get asked like, "Hey, can you include this kernel module in the microVM? " And so actually I suspect you'll see it both directions. You'll see smaller and smaller.

We're pretty well aligned for making that work. But you'll also see larger footprints and larger, more complicated application types. That's one where we'll have some decisions to make on exactly how we want to support those.

And how smart can the microVM get? Because underneath that abstraction are now a wider range of classes of processors, and then I've got these workloads running on it, and at some point, do I just express my intent in the microVM or something else figures out where to run that most optimally based on performance, cost, and all those other good trade-offs we've had forever? Where does this evolve to?

Yeah, that's where I want to be. I talk to my teams about this all the time, and we're all pretty much on the same page. Remember I talked about that spectrum of complexity in the enterprise class workloads.

So when you start out, abstractions are awesome because your focus is the problem you're trying to solve and not the complexity of scale or how AWS may or may not do one thing or another. And so the ideal scenario is like, here's my workload, run it. Like that's it, full stop.

You can optimize, you can make a bunch of decisions, and I'm okay with that. " And so that's kind of how we see things changing, is I think you will always have people with the wrenches that want to get into the system and they want to tweak because it is of value to their business at their scale, for example, to be able to make these tweaks. You also will have a growing class of people, and I think this is part of the big shift with AI, where you now have this new crop of developers who do not write code for a living directly.

They write code for a living indirectly. It's a very different mental model, and so their ability to go in and do fine-tuning, tweaking, and optimizing is not there at the outset. And being able to support their use cases by automatically doing these things on their behalf and exposing how you've done it is a great way for them to both take full advantage of what we can do and learn at the same time.

And that grows them as they start getting more success in their workload, starts to grow in complexity and size. Ultimately, when you put all this together, what's your best advice to folks about how to approach this? Because I think on the one hand, there's a lot of excitement.

On the other hand, I think folks are overwhelmed, and when you get overwhelmed, you tend to do nothing. So where's my point of entry? Yeah, I think my favorite point of entry is to play, to experiment just a bit.

Because to your point, if you try to tackle the whole thing at one time, it is daunting to say the least. There's so many different things have changed over the last couple of years, and so if you just take someone and say, "Here, I need you to go take on this whole world," you're going to have trouble getting off the starting line. But if you can go and do, as we often do, quick POCs, proofs of concept, play with some aspect of something, build something really simple.

And actually it's one of the nice things about microVMs is like it's super easy to just get something spun up. Now I can look at the thing I've spun up, throw a web server in there, just have some really basic stuff, and I've got something to react to, right? I think people generally want to react to something.

And then you kind of evolve that over time as part of your process for learning. Otherwise, you bite off a giant chunk of this new world order, and you can't chew it. It's too much at one time.

All right, folks, you heard it here. Reading is fundamental, but there's no substitute for hands-on experience, so you might as well start getting the keyboard out and start playing with it, because even in the age of AI, somebody has to start this process somewhere. Barry?

Exactly. Barry, thanks for being on the show. Awesome.

Thanks for the time. I appreciate it. All right, and back to you guys in the studio.