Accelerating AI In the Enterprise with Traefik Labs

So ai, um, I'm gonna zip through some of these, but let me just frame kind of, uh, there's a lot of things happening in ai. There are lots of different kind of, uh, places in the AI stack you can play what traffic is doing. In a nutshell, because we are a runtime gateway provider, our value is in being able to address this problem from a runtime perspective on how we can allow customers to run AI workloads in an agnostic way.

So they're not vendor dependent on the, the gateway, uh, part of it. And we'll talk more about that. Uh, but in essence, like what's happening in the AI world, there are lots and lots of models that are coming out, right?

So part of the talk that we did this a while back was think beyond the model. There's gonna be models everywhere, right? Like if you look on hugging fist, I went there today and this number continues to grow.

Like they're adding almost a hundred thousand models every month in hugging face. 2 million models. If you go to the Nvidia kind of curated model set, that number is about 200 plus.

The point of the story is that models are going to continue to evolve and change. Uh, you cannot hardcode and you should not, I, uh, I should say hardcode your application to a particular model. 1 coming out.

Like if you hardcode things to your model, you're constantly gonna have to go back and refactor the application to make the change. So then that question becomes, well, okay, decoupling makes sense. Where do you do the decoupling?

And this is where we believe that we should decouple at the gateway layer. 'cause that gives you the most operational freedom and the most operational leverage to be able to define the routing logic on which model that you wanna route this particular application to at the gateway layer. And it goes beyond just routing, right?

With the, let me kind of zip through some of this stuff. Uh, and I wanna, in the interest, so I'll come back to this. So this is a reference architecture.

You look at what happens when you are routing, uh, AI traffic routing is the last part that you do. You know, whether this goes to, let's say GPT-3 or sonet running in AWS bedrock or running in your local kind of, uh, environment or some other public cloud. But all the things that happen before you route the traffic, that's where the gateway really shines, right?

The first thing you'll see is, uh, the authentication logic, right? Just like we talked about with API gateway, you still wanna authenticate the user. You may need to rate limit them.

Then in the case of ai, like you want to have these guardrails that say, okay, well this is not a, uh, not a accepted conversation with the LLM. Like, your enterprises are putting guardrails in place saying, I'm just not gonna use the AI for everything. Or I have certain policies where let's say you define, uh, a finance agent and that finance agent should only be able to respond to finance type questions if it starts getting legal questions.

You wanna stop that from even getting to the LLM. So all of that type of authentication and security type policies and caching, by the way, 'cause you don't want to get penalized for the same question over and over again. There's no reason to consume tokens for the same question that comes up.

All of that stuff. You have two choices where to put 'em, you can definitely put 'em in the application. Like if you're just hacking something together over the weekend, makes sense.

But if you're putting it in production, this is where you want all of that logic to be at the gateway, because that's what the gateway is known for, is designed to do that is scalable. It performs very well. And you as an organization would get all of that unified control and the observability on everything that's happening.

So you route everything through the gateway, which, uh, sorry for the, maybe it's not coming out clear here, but if you look in the box in the middle that says traffic hub, and you look at from the application standpoint on the left, and let's say it's a rest call first is gonna get authenticated and perhaps rate limited. There's a WAF in there too. If there's something you wanna block there, then you go through this LLM guard, think of this as a composable pipeline, okay.

Of traffic just flowing through and it's just kind of acting, uh, in a chained sequence. Then you would go through the LLM guard. And here we have done the integration with the NVIDIA safety nims.

You know, there are three specific Nvidia safety NIM out there today. One is for topic control, one is for jailbreak detection, and the third one is content safety. It goes through that.

Then you may want to take it through a caching layer. And then you route, and even when you're routing, you have a bunch of different options for routing. Let's say you have only two models throughout.

1 as their workhorse LLM model. 2 comes in. The question is how do they absorb that change?

How do they integrate that change into the environment? You know, if it's baked into the application, they gotta go back to the application and refactor it. But let's say they have decoupled it and it's at the gateway.

What do you do? Then? You can do many things.

You can do a simple one, which is a canary based routing. 2 and let me run that for a few days. Let me take, collect some telemetry.

Let me see how the, how the user experience is. That's one way of doing it. There's two other advanced ways of doing it.

You can do it based on identity based routing or time of day routing. And let's go through those in a little bit more detail. Quick Question, with the routing there, can you inject transforms along the way as well?

So if you are going to one model versus another model, you may wanna manipulate the system prompt in some fashion for that second model. Uh, we don't get into that. So we can do header manipulation, we don't get into the body manipulation.

Uh, and you'll see those are the kind of things that are gonna add more latency, uh, going through the gateway itself. Uh, can it technically be done? Sure.

But we don't get involved in that yet. Okay. Uh, Saddi, can you back up just one bit?

A guy courier, uh, at Futurum. Hi Mitch. Hey, um, so you call this an AI runtime reference architecture.

Um, do you have a distinct one that, uh, includes the MCP gateway in it? Yep. I'm, that's, I'm gonna show that next.

Okay. Because AI's AI means is pretty general. Yep.

Should include agents. Yes. But you're gonna have a new ano a second reference architecture to show us in a Bit.

I'm gonna, I'm, It's called something else. It Should be all one. To your point.

There should be one AI reference architecture that includes MCP, includes AI and also includes API gateway, and I'll show that to you in a second. Yeah, I think there should be either be one or a hundred. So one one is good for, for this venue.

Yes. Alright, cool. There's an extra layer that MCP brings that I wanna introduce before I show you that.

Okay. Okay. Uh, that would make a lot more sense.

Great, Thanks. So I talked about canary routing is one way. Uh, let's look at identity based routing.

You know, this is where, you know, there's a bunch of code here. What really matters is you look on line 20 and line 28. 2.

1. So this is where you can do identity based routing. And everything I'm showing you here is just code, right?

So this is just a CRD or a YAML file. And all of this could be done through A-C-I-C-D-A time of day routing. This could be another one that could be that.

2 into the model, but you don't wanna really risk it during your normal, uh, business hours. So you wanna route it to LAMA 3 0 2 after your prime hours, and that's what time of day routing does line 20. Again, it's saying, okay, during prime hours, in this example, I'm showing you a different model.

I'm saying, okay, send it to mytral, small off hours, mytral medium. So just pick your model, swap it in here. But the concept stays the same.

And there are many other ways to route, but these are, like the three that I showed you are the most common ones that you can route based on identity based routing is becoming more popular. I think canary based routing is the most default. I think that's the most simplest to do.

Identity based is more sophisticated, but it's also much more powerful and much more flexible. Customers have any interest in doing, uh, kind of performance based, like this is taking too long to respond. Let's go.

Yep. Back up. Yeah.

Model for, so that could be your standard kind of failover workflow where you try this, this doesn't work. Then you go to that and what you're saying is, uh, latency based routing. Uh, so we have that in our roadmap, uh, where we can actually do it based on latency.

Okay. And both in a proactive manner and a reactive manner, which means we could be, we could have sensors that always, there's measuring the latency, uh, between point A and point B, and you can set a threshold saying, I'm gonna route to this if I can predictably say the latency is gonna be beyond, below this number. Mm-hmm.

So those are some advanced things that we have in the roadmap. Okay. And latency, I mean, some, I usually hear it referred to also to as performance based, meaning latency could just mean the latency of traffic.

Yep. You know, or packets flowing. Performance usually implies like, this takes too long to get a response or Yeah.

You know, it has three retry and I'm not gonna let the bottle try at third time or whatever. Absolutely. And that, those are exactly the things you want to do at the gateway.

Yeah. Okay. Yeah.

Latency is a, a prerequisite to that, but, but you're absolutely right. You use that as a base knowledge and then you build on top that advanced lo logic. Okay.

Okay. So I zipped through a lot of this stuff, but, uh, I wanted to actually make sure that, uh, I shared this concept of, um, uh, the runtime environment. You know, when you're deploying ai, uh, there's the gateway runtime environment that we all are familiar with, right?

AI gateways. They represent the AI runtime environment, but there's the model runtime environment. And the objective of the model runtime environment is to optimize for the inferencing to make sure that the models have high availability and high resiliency, right?

Traffic doesn't operate in this, this is, you know, the task of kind of the, the other platforms out there that are giving you the ability to kind of stand up a bunch of GPUs and being able to get maximum throughput of the tokens and get maximum throughput of the GPU environment. Uh, and the LLM that is running, okay. Traffic then takes over as soon as the LLM is exposed to the outside world as an API.

That's where traffic comes in. As you guys all know, like all of these models are being exposed as A-P-I-A-P-I has become the interface. And this is where traffic comes in and says, okay, the abstraction of that model as an API is what creates the need for having an API level availability.

An API level resiliency. And this is where we come in. Both environments are equally important.

You run into problems though when you try to embed them as there are some platforms out there in the market, I'm not gonna name them, but that try to embed part of API runtime environment into their model runtime environment. Um, and what you end up doing is, you know, you go with the least common denominator and you end up getting some functionality, but you lack in a, in some major ones. So the three questions you wanna ask yourself to know if this is a limitation that you're running into, number one is can you freely share these endpoints with your developers through a developer portal?

You know, just like you would any other API, you know, does that experience exist? And what you'll realize is no, uh, and if it does, it's very, very archaic. Number two is, can you actually integrate with your, uh, identity provider for doing the API token generation?

And I'm not talking about the token for the LLM, I'm talking about the, the API key or the jot token that allows you to access the LLM as an API. And the third is like, okay, what kind of observability do you provide? Uh, is it vendor specific or is it vendor neutral?

So these are some of the challenges that you'll run into with this Saddi. Can I, can I probe a little on, um, on ai, ai, ai, API on? Mm-hmm.

It's, um, you know, but models as APIs, there's, there's, there's something a little funny sounding that to me. 'cause I, you know, I think of an API as highly structured providing re requiring a a, a structured and compliant or conformant, um, well formed, uh, a request mm-hmm. Um, in order to operate best.

Um, but if you think of a model as I, I remember how you put it, it's pretty clever on the, I think on the next slide, um, if you think of a models, uh, APIs as as, uh, uh, a Abstraction Yeah. Model. Yeah.

I mean, you know, uh, I, I think of an API BA request to a model is unstructured largely, um, you know, plain language, plain English, what have you, you know, a little chat, like it's a prompt rather than something structured. But it's, uh, But it's, is this a semantic issue away for us to shift our thinking as as to what we are doing? So there's parts of it that is highly structured.

There are parts of it that are unstructured. Right? The actual call that you're making to the LLM.

So let's say you ask what is the capital of us? And a response to you when you're sending that question in, it's going inside a highly structured JSON, uh, and, you know, wrapped in inside an API that's going to the LLM and then the LLM is computing through tokens. What the answer is.

And that's coming to you, but it's also coming to you in a structured format. But the answer is gonna be variable depending on your question. So essentially it is an API interaction, but the values are dynamic.

Your question, you said highly structured, JSON, you, you that, you lost me a little bit there in the sense that Yeah, sure. The package might be JSON highly structured. Hmm.

You know, I, I don't know, like, uh, um, you know, highly structured JS os are like big complex, require a lot of validation. There's a whole, what I guess I'm trying to say is that the payload of that JSON is doing a whole lot of work. Yes.

And in conventional API needs to be done in a structured way. Yeah. So there's, so I look at this as what's happening behind the scenes and what's happening at the interface level.

Uh, to me, if I go into, um, so let's say if I do a curl command right from my machine to an L-L-M-A-P-I, I am doing it in a very structured way. It's, I'm always doing it the same way. I'm passing it the same parameters.

Mm-hmm. My question might change, but the way I'm calling it is always staying the same behind the behind the scenes. What it does, what the LLM does with that is a whole bunch of other stuff that I don't see, but the abstraction of it to an agent, to a user, it's a standard API that you're calling.

And that part has to be standard because otherwise Yeah. You don't know how to model against it. You don't know how to code against it.

You don't know how to integrate that into your application. So that part has to be standardized, which it is. And that's what the API is for.

This is a really useful semantic shift in my opinion. It's a really useful semantic shift. I just wanna distinguish that from maybe a more technical question of, Hey, we're gonna transform your prompts into API requests, you're not doing that, you're not really changing that, but you're encouraging us to think about this in a different way.

So I look forward to seeing how What I, what I think is interesting is when you're talking about inspection of the prompts mm-hmm. And the responses that come back from those prompts and checking to make sure that you don't have someone trying to pull financial information when they shouldn't have access to that, or they're asking for something that would violate policies that you have within your organization, being able to check those contents and look for prompt injection attacks and other things that a typical gateway wouldn't know how to search. Right.

Contents that way. 'cause it's looking at headers and other components of the JSON payload. Yeah.

And that's why like, uh, so it's a great point you bring up. So what the gateway is really doing is what it does best. It is orchestrating when that traffic is coming in, it's orchestrating the set of checks that it needs to go through.

And by sending that traffic to endpoints that behave as a well understood API. Mm-hmm. So, you know, in the NVIDIA safety nims, you know, those three that I talked about, topic control, jailbreak detection, content safety, each one of them exposed themselves as an API.

So what the traffic API gateway is doing is orchestrating across all of them, sending them the user information, whatever came in unfiltered to them and saying, Hey, check this for making sure it's safe. Making sure you know it's the right use. Uh, making sure it's not doing any kind of jailbreak detection through the prompt engineering traffic's not getting involved in that.

All it's doing is orchestrating and saying, Hey, check this. Mm-hmm. This, Hey, API X, check for this.

Send me a response. If your response is good, I'm gonna let it through. If your response says bad, I'm gonna stop it.

Is that filtering happening in parallel? So is it checking multiple signals at the same Time? Yes, it can.

So, uh, you can chain them in parallel. You can do linear, uh, you have different options. Okay.

Cool. Yeah, you had a question? I did have a question.

So I'm Gina from Digital Sunshine Solutions. My question is, um, the title of the slide says No, AI without APIs, but everything you've described has been generative ai. So is, are we specifically talking about prompts coming in from externally, perhaps from a, a chat box that's on a window for customer support or whatever, um, going into an LLM?

Or do you also cover, uh, more narrow AI situations, maybe, um, uh, just a plain type of agent that doesn't have the LLM behind it, but it's just doing some basic lookup or, or that kind of thing? Uh, so in all of these examples, there is an LLM, there is an LM component. So an agent has to have the intelligence of an agent comes from the LLM.

Okay. So, so this is just, so this whole presentation or your product is just about generative AI and not about other types of ai? Uh, correct.

I mean, so we are, and also we're not just about ai, but yeah. This part of the presentation. So what we offer is an API gateway, which can be, um, you know, manifested into an AI gateway because an AI gateway has a lot of the API gateway foundations, and it just adds the AI ness on top, like the extra stuff for ai.

But it is still a gateway, which fundamentally means you have an input and an output. Uh, so if I go back to this, uh, this diagram, you have an input and an output. The input is coming from the left, like you said, you know, it could be your chat users, it could be an application where they are trying to interact with an LLM with an AI model.

And before you allow that interaction to happen, you wanna put a set of safety rules, right? And that's what this is, right? It's a bunch of different policies, let's, let's call it, for lack of a better word.

And that policy has routing logic. The policy has safety logic built in all of that stuff. Got it.

But it's not ai, it's just generative ai. Right. Um, I mean, when I think of ai, I think of generative ai.

Yeah. Most people, so for me, like, you know, there, there is in a distinction between the two. Okay.

Uh, but, uh, you guys asked about some questions on MCP, so let's go there, right? Because what's happening with MCP, uh, the, if you focus on the left side, MCP really comes in specifically for agent workflows, which means there is an agent trying to achieve an objective. And that agent now has to do three things.

The agent has to talk to an LLM, the agent has to talk to MCP resources, whatever you expose, and then the agent has to talk to your backend APIs as well. Traditionally, if you just have an API gateway in your architecture, you're not protected because you're only protecting one of the three kind of pathways. Typically, gate number three at the bottom is what most companies would have, which is an API gateway.

So what it can do, it can protect the communication between an agent and the backend APIs, but it has no visibility or no way to control what the agent is doing with the MCP server and the resources behind it. And it also has no knowledge of what the agent is doing with the LLM itself. So this is where we have introduced this notion of a triple gait pattern where you need to have three different gates for your agentic workflows.

Gate number one really is the agent is talking to an LLM. And so you can control that with the AI gateway, you know, which we just talked about. The third one, the agent talks to the backend.

API, you know, that's well understood problem. This is your regular API interactions. The second one in the middle, you know, this is a little mystery, right?

Because MCP is just coming to the, uh, into this, uh, you know, the party here, so to speak. Uh, there's a lot to be understood here. Uh, it doesn't behave like our typical API, it has a new language, a new protocol.

So how does the agent talk to these resources and how do you govern that? How do you put in the set of rules and policies and stuff that allow that interaction? This is where the MCP gateway, this is the job of an MCP gateway.

And so what we are doing at traffic is giving all of these, all three of these kind of capabilities in a single binary, because the last thing you want is a customer. That's what I was deploy. Yeah.

Three different gateways. Yeah. So Guy Ker again, future, I was just thinking this, which is, these are three names for the same thing.

Um, it'd be three names for the same Thing. Yes. If you can come with a better name, like I would love it.

But they are A-I-M-C-P-A-P-I Gateway. Yeah. API Gateway, MCP edition.

They are c they are three distinct capabilities that are typically not in the, if you look in the market today, it's very rare that you'll find all three of them in the same product. I just feel, I feel like the, the scope, let's call it the scope, or maybe there's a better word for it, differs among the three. Absolutely.

Be really helpful for you to point out that that sort of payload inspection, since we're working at the application layer, you know, the prompt is the application in a sense, and payload inspection and, and routing based on that. I mean, that prompt is, that's brilliant. So I extend it to, to, to, you know, um, uh, agent, agent and the agent agent and, uh, model agent interaction and stuff, the sort of thing that's an MCP, that's a little bit more prosaic.

MCP is a protocol, obviously that's in the name. Um, but it has particular elements to it thanks to its connection to agent and everything. So, right.

So all of these kind of need to do the same thing, but, um, what differs is, um, the, uh, the, um, the, the, the boundaries that you're putting on Yeah. Like you said, the scope. Yeah.

The scope is different for each one. I would describe it, I would describe it as like three different use cases you're controlling. Mm-hmm.

What models can agents talk to? What you're talking about, what resources can an agent or a model access through an MCP or MCP servers? And the third is what APIs can an agent use to go to a traditional API?

Yeah. So it's, you know, first one is kind of what models can I use? The other are what resources through either traditional or non-traditional, right?

Yeah. So I think, yeah, no, this is, I would Describe it as like, use cases for, and your point is it's in one product, not two or three different products. It's just that when you put MCP or AI on the name, they can sell it better Always.

You know, and I'm in favor of that. So, and charge twice As much, you know, and, and in agent workflow, you need all three of those, all three use cases come together. I'm not entirely joking that you can sell it better.

You're positioning it for these different use cases in a stronger way than just saying that our a, our API gateway is M-C-B-M-C-P capable AI capable. I I think it's stronger to, I I don't normally advocate for that sort of thing. Um, but you are, you're, you, you are broadening the positioning of what you do, and there's a technical foundation for it and technical proof for it.

Yeah. You know, you know what, you know what I mean? It's, it's, it's beyond a scenario and your scenario.

Well, What's interesting too is that yeah, it's a, it's a, uh, an emerging market because you have a lot of different people going after the same parts of the same thing, right? You've got the Nvidia DGX clouds of the world, that that's more the tied to the specific hardware resources. You've got the vertexes and sage makers and Bedrocks who have, you know, a gateway for, part of a gateway, right?

Mm-hmm. For access to models and MCP servers. Then you have emerging agent control planes, which are also starting to apply some of the rules and guardrails, right?

Of some of what you're doing, not traditional APIs. And you're trying to position it as, yes, you're gonna need all those things. And do you want a different solution in the Google cloud versus the AWS cloud versus ad's hardware versus NVIDIA's, and then you have your traditional applications.

Here's one model or one product. Yeah. But one, one way of doing it that we're work on all Those environments that abstracts it away.

That's your value product. Yep, exactly right. Exactly.

Yeah. It's exactly. Everybody's fighting over this territory right now.

And MCP really pushes the boundary of this because it forces you to think about this holistically, right? You cannot think of this as a silo because in an MCP architecture, like this is a reference kind of a high level reference architecture that you see the triple gate pattern working. Like first you gotta go through the AI gateway, then you gotta go through the MCP gateway checks, then you go to the API gateway and your resources, the data layer where your MCP server and the resources are staying, where also your APIs are there.

So you have to authenticate through these three layers for the different scopes that you're trying to achieve here. I think it's an argument to be made too, that while MCP and other open protocols are adding more security, needing to beef up their enterprise level security, that's gonna vary by implementation and through a gateway or one common gateway that you can now apply security rules and guard guardrails and things like that. Kind of, you can deal with the inadequacies of different implementations of, Especially if SP servers, right?

Especially if there's documentation behind it. So if you have, and I don't know that there is, but if there's a way to, to have that observability where, yes, we actually did have this communication pathway com, this was all that was allowed, and this is right trail. I think that Is a huge deal That that's a big deal too.

Yeah. Yeah. 'cause if you're not having, if you don't have the centralized picture, then you're not gonna have centralized auditing or the observability in it.

Okay. Um, let's go to our last topic. Um, I'll just say a couple things here.

It's really important to think about the runtime environment as being strategic. Like how your customers design runtime environments are very important. Our recommendation is to decouple the API runtime from the model runtime, because the assumption here models are gonna continue to change.

So you need to bake that rate of change into, into your infrastructure and this one model to rule them all. That's a myth, right? As we all know, like, you know, that's not gonna happen.

You may have temporary advantage, but you're not gonna have long-term any, no model's gonna have a long-term advantage given the pace at which we're seeing things move.