Crossing the Production Gap to Agentic AI with Fabrix.ai

Hi, everybody. So my name is Rashed. I'm at Fabrics ai.

I won't, uh, bore you with a long introduction. I, I help make things, I sometimes break them. And I have a, a few minutes to try to go a little bit deeper on what the, uh, uh, what we're doing with the middleware.

Um, uh, first of all, I just want to talk a little bit about motivation. So this, my, my piece is really about trying to convince you that we're very focused on, uh, one specific problem, which is how to take the agents from, um, from the prototype to production in a large enterprise environment. Um, I was, prior to joining Fabrics ai, I, I, I was an independent person, uh, building agents, and I, like everybody else, got extremely excited about what you could build really quickly.

Um, and then like everybody else, I started running into the wall of making this work in a real enterprise environment. So, when I joined fabrics, I was actually surprised to find that there was already something in place, uh, to help deal with that. And so I'm gonna talk to you about that.

Um, so ba basically, one of the, one of the, some of the, the key things that make this easy to, to get off the ground with an agent is you're, the main thing I'm gonna say is that you're working with it all the time. You're sitting there, you're talking to it, you're interacting with it. Every time something goes wrong, you pick up on it, you're able to, uh, iterate very, very quickly, and then you're eventually, without realizing it, kind of paint yourself into a corner where everything works great, but, but you've kind of guard railed yourself.

Um, and then you put it out in the wild, and then the variables change and everything, uh, falls apart. Usually in our experience, they fall apart for, uh, a number of reasons. So, what's on the screen right now are 16 examples of how things go wrong when you try to take a prototype agent and you try to move it into the enterprise.

Um, these are not the only things that go wrong. Uh, a lot of the things that go wrong go wrong with any agent, whether they're prototypes or deployed at scale. And so there's things you have to solve for if you wanna build any agent, you do have to solve for how do I retrieve information in rag?

How do I, you know, do good prompt engineering? How do I, all of these things that we've kind of like as a community have been learning out loud and, and, you know, there's the buzz of the month about how to solve this problem and that problem, whether it's chain of thought and whatnot. I'm talking about things that happen when everything's working great, but then you try to scale it and it just breaks, and it, it basically breaks down into these three kinds of categories.

Most of them are gonna be in this context, um, uh, management bucket. And so when we're talking about context purity, which is something that was mentioned earlier, what we're talking about is we're talking about the fact that the LLM needs to get exactly what it needs to manipulate the statistical probability that it's gonna give you the output you, you're desiring, right? So you are actually playing with a statistical system, and you're trying to game those stats.

That's what you're trying to do. And there's a variety of ways that this breaks down. I'll just pick a couple here.

Um, one of them is gonna be large tool responses. When you're prototyping something, you've got mock data, and it's usually, you know, dozens, hundreds lines long. So anytime you call a tool, you get something, it, the LLM understands it and then proceeds, um, in a production environment, sometimes the tools will, will return enormous data.

Sometimes they'll have to return data from thousands of remote systems, right? These are things that you typically don't play around with when you're prototyping, but you deploy, especially my background's in network automation. I'm, you know, I'm telling you, if you try to get an agent to work on your network and you've got hundreds or thousands of devices out there, and you need to do something on that network, it's going to break unless you have a solution to handle that.

Um, the other ones is gonna be about operations and how do you operationalize this? And a lot of these, uh, gotchas are gonna be around observability. As soon as you set it out there, you lose sight of it.

And when you lose sight of it, that's when it misbehaves. It's kinda like letting your, your dog go, you know, and they're gonna get into trouble if you're not watching them. Um, and then the last one is gonna be, uh, you know, operational infrastructure.

It's gonna be things like the security people, those pesky security people that are gonna come after you because you're not being careful with the data. You're not being mindful of what the LL m's doing with it. Um, you're, uh, not, uh, properly segregating different users.

You don't properly handle the fact that the person using the agent may or may not have the same rights as the agent itself. And how do you handle any kind of differences that arise from that? So these are not corner cases.

They're going to be the types of things that everybody who tries to scale an agent is going to run into. Um, so it just turns, it just, you know, coincidentally, completely coincidentally, turns out we've solved a lot of these things and, um, we've solved it by basically taking three core principles, um, about how to properly, uh, build agents for scale. The first one is don't resist the temptation to give the LLM the whole problem.

Move as much of the problem into the tooling layer as you possibly can. Okay? And fortunately, uh, in a lot of use cases, there's ample tooling in place.

It's actually underutilized. I was at a recent conference where people were bemoaning the fact that network automation, for example, which is near and dear to me, um, is, has been slow to be adopted, but it's not for lack of tools. The tools are there, it's just that people are very resistant to using them.

Agents can be convinced quite easily to use those tools, right? But designing the tools, software practices, how to, where to put the, uh, the right levels of, of abstraction so that the tools are operating effectively. There's an art to it, and we'll talk about that.

The next one is curate the context feed. Try to tee up the problem for the LLM so that it always scores a home run. That means get any of the garbage that isn't gonna help it out of that context window.

Try to use context efficient formats, uh, token efficient formats when presenting, uh, the model that information. Um, try to not make the LLM be the thing that carries the data from one tool to another, if at all possible. That's a waste, right?

And so then the last thing is gonna be all of the operational stuff. You have to not just observe the model. You have to observe the agent.

What does that mean? Well, the agent has a job responsibility like a person does. So manage that and, and look for outcomes.

Is it successful? Not just did it, was there a tooling error or, or did it take too long? Or did it actually do the job successfully?

You have to monitor it at that level, at the business outcome level. So the way do we do this between this, we, we, we call this the middle, uh, the middleware, right? Um, and you've already heard the terms context engine and universal tooling and connectivity engines.

This is the two functional pieces. This is a functional diagram. This is not an architecture diagram.

It's showing you the function that the fabrics AI platform provides that sits right between the age agent layer and the tooling, uh, enterprise data and system layer. Okay? And it provides this three basic, uh, capabilities.

Um, a task relevant context, um, so that we're, we're making sure that the context is gonna help the LLM be successful coordination of tools and, and connectivity of tools. What that means is we let the tools talk to each other, uh, at least to pass data to each other, rather than bouncing through the LLM. The LLM can say, Hey, tool two, I just called tool one, the outputs over here.

Go get it and then run, uh, run yourself, right? Um, and as well as the connectivity. So make it super easy and, and uniform.

Uh, some of the, the, the, the questions earlier, were talking about normalization of data and about, um, making, uh, the, the data digestible, uh, at correlation time. So we want to be able to handle that, and that's what the tooling, uh, and connectivity engine is doing. And then the controls, which I've talked about, and I'm gonna show you a couple of examples.

So the, Yeah, I'm Gina Rosenthal, digital Sunshine Solutions. So the tools, so the middleware actually sits, is the first thing it sits on top of, are the tools. It it collaborates with the tool, collaborates it, it communicates with the tools, and then that is what goes that, that output of that is what goes into the infrastructure and the data.

Yes. Would that be true statement? Yes.

I'm gonna double click on the, on each one of these boxes, and I'll show you exactly what goes on inside each one. Okay, cool. Thank you.

All right. So the first, uh, I, I should take a step back. Uh, the reason that fabrics AI is called fabrics AI is because we're, our platform is, is, is, is a trifa platform.

It's basically what we call a fabric is gonna be anything that connects many to many. The data fabric is about connecting data to data. The automation fabric is about lacing together all of the automation processes and the policy management and all that stuff.

And the AI fabric is gonna be all about the stuff that happens at that level of abstraction where you're talking about things like, um, like, uh, reasoning and, and prompts and, and, and all of that AI stuff, right? And so, two, the two components that I talked to you about, these are functional components that are, that, that can, that constitute, um, the, um, the middleware. They emerge out of that fabric, right?

That, that fabric is what they are made of. Um, the, the context engine is lives inside the AI fabric, and the tooling and connectivity engine lives inside the data fabric. So the context engine in a nutshell is this, it's this box that takes in all of the dirty, dirty context and cleans it up and provides pure signal to the LLM.

Um, there's four main ways that it does this. Uh, one of them is, uh, intelligent caching. So what that means is that when a, when a, when a tool result comes in or when, uh, you know, any kind of data comes into the context, um, it is cached intelligently based on how much data it is, if it needs to be converted, it's converted, uh, in, in, and it, there's basically a, a virtual file system in which it goes, and then that becomes addressable.

It becomes addressable data rather than data that lives inside the context. Okay? Mm-hmm.

Um, the next one is con conversation compaction. So we, we, we, we heard earlier, there's, there's a lot of, there's a lot of, uh, mind share around compacting the context. And a lot of our tools, like Claude Code or whatever, they, they've got that built in.

They, as you go. And as the context grows, it just compacts. It does this though, through summarization and summarization is very lossy.

Um, uh, especially for long operations, you end up losing something. And it's hard to tell at the time when you're making the summary what you need to keep and what you need to throw away. And if you make the wrong decision, you kind of screwed, it's not gonna work.

You've lost the information that you need. So the, the conversation compaction, um, uh, technology that we have in the context, it's, we have a few clever tricks to make sure that the compaction maintains the relevant information for the current, uh, uh, conversation, um, dynamic. Yeah.

Is that easier to do? Because you're specifically working with tooling for enterprise applications, so you know what information you need to maintain when you do the compassion. So what we do is we have a, we have a, a a clever trick that we pull off, which involves understanding.

So usually agents have a multi turn interactions, right? Especially if you're talking to an interactive agent. There's this multi turn things that's happening.

And so you might say, for example, you know, what's the capital of France? And it might give you the capital, and you say, what's the weather in London? And it gives you that.

And you say, and what about Paris? Right? And, and it needs to know, is it asking me about capitals or everybody's asking, right?

So the trick that we pull is that we're always looking at what the current, uh, goal of the user is, and then we go back into the history and we create a summary that's tailored to the current actual goal rather than the past. And it, it gets a little tricky to do that, but that's basically what we do. And so you end up with a series of little tailor made summaries, so when you chain those together, you actually preserve all of the information that's relevant.

Thank you. Right? So I'm not gonna have time to go into this too deeply, but I do wanna point out, you know, that this, the, the context cache sits there in the middle of the context engine.

It's this virtual file system where we put stuff, it lives there. And, and, and there's been a recent, um, paper that was, that was published about, um, recursive language models. I dunno if anybody's read that, but recursive language model this is, is this idea that instead of getting an LLM to work with an gigantic document, it can spawn off, uh, a bunch of an, uh, an arbitrary number of subagents, and then assign bits and pieces of the, of the document to those subagents, um, through what, uh, and our EPL notebook and our, what that actually gives the, the whole basically benefit is that the, uh, the, a credibly large document becomes addressable data rather than data that has to be completely ingested.

You can seek it, you can profile it, you can histogram it, you can do all sorts of things to it to understand and feel around its edges, and then locate, pinpoint the bits and pieces of information that are going to be relevant to the thing that you're doing before you load it all inside your contract. What actually makes intelligent caching intelligent? What, uh, that's, what are you, what are you calling the intelligence behind that caching?

Um, so the, in the, the way that, well, there's different little bits and pieces. One of the pieces gonna be that, um, we're not necessarily gonna cache something that is small. That's a very basic decision that we're making.

If it's small, just throw it in. Um, but if it's large, then we want to cache it. And then, uh, what we want is, as we cache it, we glean certain bits of information from the data that we've cached, and we provide that to the LLM.

So the LLM has a sense of what lives in the cache before it tries to work with it. It's almost like vectorization. I mean, you're, you're extracting some dimensionality out of the data and, and providing some sort of a, a number stream to the, to the LLM, so it can understand that it exists.

And if he wants more data, go for it. Something like That. Some sometimes like that.

Yes. It depends on the, on the type of data. Sometimes it's gonna be very numeric.

So you want to, you want to maybe give it, uh, some sort of a basic profile. What are, they're your mins and maxes and averages. What does the histogram of the data look like?

How many roads Have perspective? Yeah, okay. Okay.

With a pointer to the data in case they Wanna go back with a pointer to the data. And so you, you know, for example, that, you know, there's a, you know, 10,000 lines of data and you have a histogram that says that the first quintile ends at, uh, at, at row. If Quin first quintile of this particular column, value ends at this row, and so you're able to request, you know, maybe the last three items and the first three items of that, uh, data frame, you know, so all of the machine learning type of techniques, we, we give those to the LLM through a, a set of fairly abstract tools.

So I have a question based on this slide. Yeah. Um, which is probably bigger than the question I wanna ask, but more to like, what is actually involved in your solution.

So this context, c is an external source. Uh, so does, does that bring your own or do you guys provide a source, or how is that working? So we do, uh, our platform does have, you know, a, a data persister and data storage, but it could be external just as easily.

Yeah. Are you building a data graph DB underneath? We do have a grave grade, uh, uh, graph db.

Yeah. So we, our platform comes with graph. Okay.

Uh, as well as object storage and, you know, so Your sources are becoming nodes and you're building all of what's building the edges around all of that, Depending on if the data, uh, is structured that way. Yes. Okay.

Gotcha. Yeah. Alright.

So, uh, Before you go on, I think, uh, Marianne had a question. Yes. Can you talk Marianne?

Can you guys hear me? Yeah. Now we can.

Yes. Thank you. Okay.

So I had a question. I see you have on your slides, uh, peer and purity. So my question was around what, what does contaminate your model?

And what are the enforcement controls? Is it isolation or something else? So usually what contaminates the context is going to be, um, uh, a bunch of extraneous information.

So for example, um, take, take, uh, uh, a tool, uh, that it's output has a bunch of introductory text and a bunch of, you know, closing text or a lot of formatting characters. Let's say my tool returns HTML, right? I don't need the HTML tags, so why don't I just strip those out and just keep the text, right?

So that's because each one of those little characters in an HTML document is one token. Um, whereas words are, you know, oftentimes you can get like a long word where that's just two tokens long. Uh, but, but if you have an angle bracket on each side of it, that's another two tokens.

And so keeping the, the, the token count low, uh, not only lets you put more information that's relevant into the context without reaching that context rot threshold where the, the, uh, the effectiveness of the LLM drops off. Uh, but it also, you know, is reduces the cognitive load on the model as well. Does that answer your question?

You don't look like I answered your question. I guess my question was more around the controls. Uh, you talked about data context.

Is there some automated governance controls or security controls that prevent that? Uh, no. So we do have, as part of our platform, like as guardrails, right?

That guardrails is one of the operational features. So anything that goes, uh, prior to going to an LLM will pass through a guardrails model that will do safety checks and compliance checking that. But that is outside of the, uh, that is part of the security infrastructure of the middleware, but it's not specifically part of what we call the context engine.

And so is that auto automatically deployed to do that in seconds, or is that, where does that fit in that process in your middleware? Yeah, it, it does introduce a little bit of latency between the prompt and the, and the LLM receiving it because it has to pass through. But those models are typ typically very fast, and they're, uh, specifically trained to, to, uh, to find, um, you know, uh, not non-compliant, harmful type of instructions.

Okay. Thank you. Yeah.

I have a question, but I don't wanna subvert your presentation. I think what some of us are grappling with is, okay, you have a bunch of slides saying you've been there, done that. What's the customer get as a deliverable code consulting support pilot project?

Yeah. We're a product company. Yeah.

Uh, so our, and so our product, uh, you know, we, we, we deliver that product in a variety of ways. But, uh, I would say that, uh, oftentimes what we're delivering is agents that are going to be, uh, operating inside of a, you know, enterprise and they're gonna be effective and scalable. Uh, but for some customers, what we're doing is we're saying, here's a platform in which you can develop your agents.

And what we're saying is, as opposed to some other platform that's out there, um, that this, this is going to provide the facilities or all of the services that platform based services that agents need in order to be scalable and reliable in, in an enterprise. And I would think, from what I've seen of other products, not in this space necessarily, but folks might want a pilot project where you demonstrate how to use the toolkit on some sample problem, maybe one or two. Yeah.

And then maybe some consulting support as they go along are the Yeah. Yeah. We always help.

We're always, you know, uh, we always get in there. Um, but our business model isn't primarily consultation. We want to give people, enable them to mm-hmm.

To do stuff. So rights to code, but not so much services, which That's right. Doesn't scale.

That's right. Yeah. And we have a extremely low code type of approach.

I'll show you an example of that, uh, in, in a slide that's coming up here. So just before I run outta time, I'm, I, I may not have time to do, uh, the demos here, but I'm, I'm going to, uh, talk to you about the connectivity engine a little bit because it's the other important piece of this. Um, this is the part that, uh, mentioned earlier that's connects to the tooling, right?

So, uh, tools, uh, MCP tools, for example, they sit out there on a server. And MCP was a great innovation because it creates this level of abstraction around APIs. Everybody loves them.

And as we mentioned, however, they're in an enterprise environment, there's still plenty of tooling out there that isn't, uh, wrapped by an MCP uh, server. But even if it is, right, you have this problem, I, let's say I have an MCP server that's a MCP server that stands in front of my SQL database, and I have another MCP server that stands in front of my, uh, my, uh, my inventory system. And I somehow need to update, uh, do something that involves information from my inventory system and information from my SQL database, while great.

There's these two MCP tools. So the LLM can easily call the tools that are sitting there, but if anything needs to happen here based on something there, the LLM has to suck in all of the responses, understand that, construct the data that it then pushes down to the other MCP servers and you just wasted a bunch of tokens, essentially performing a shuttling process from one thing to another. Mm-hmm.

Right? So the fact that we have this MCP, um, the, the, the, our particular, our, our, our engine over here, the, the, the tooling engine is, it creates kind of like a, a, uh, an net, uh, an MCP abstraction layer. It's itself an MCP server that wraps other MCP servers and non MCP based tools.

And because we do that, you get this joint and common execution environment for them. My tool results come back and they're in the tooling engine, and therefore I can move them to the next tool as an intermediary without ever touching the LLM. I just let the LLM know that the tooling, the tooling results are there.

Right? That's one thing. Another thing is that, um, we have this YAML based way of defining tools because we provide some primitives inside of the, the tooling engine that pretty much lets you develop any tool.

And through this, uh, YAML based abstraction, you can easily develop new tools and you can take a, a tool that might be, for example, SQL Query tool. And a lot of people, they love this when they're building a prototype. They say, look, I hooked it up to an MCP server, it has a SQL Query tool, and now the LLM can write SQL queries.

Awesome. Except it breaks on production because building SQL queries, if anything goes wrong, uh, it doesn't work. And the LLM, the bigger the job you give it, the more likely that the SQL query it will build for you, will be wrong somehow.

It's too complicated. It can't come up with it at inference time or whatever. When you have, uh, this, um, this, uh, middleware layer, you're able to define a tool or specific types of queries that are going to be common in your enterprise workflows.

And now it's just simply calling a tool that has a name and a bunch of parameters rather than constructing an expression that's brittle. Right. And we can actually, you know, you can talk to the system and have it generate those for you.

We have a, a thing that makes the tools for you, so you're gonna have to bother coding all that up. It'll do it because we have something called dynamic data discovery, where you can say, I'm interested in doing these things. The system will connect to your MCP server or to your data source, and it will not just consume the entire API, it'll explore that API and figure out exactly where the data lives.

Mm-hmm. And it'll skip the tables that don't have data. It will focus on the tables that do, and, and, and, uh, it just generally means less responsibility for the LLM at inference time.

It doesn't have to query the wrong table and say, oh, I've got zero results. Why is that? Oh, there's another table with the same data every single time it runs.

Right. You mentioned early on that you were doing sort of data science type analysis Yeah. With the intelligent caching.

Yep. Do you do similar type of data science work in this layer as well? Yes.

As you explore sources? Yes. We, because data science is in our DNA, so we do, we, we do, uh, schema discovery and we, we code our agents when we write our prompts and all that stuff to do exploratory data analysis as just a matter that they work, always do EDA every time.

And we give them the tools to do EDA because when you do your data analysis, you create shortcuts to good information that you otherwise would, uh, would miss. Right. Thank you.

Okay. Would you guys, um, like to see the demo, or actually we have a couple case studies. Case studies would help me understand that.

Yeah. All right. So I'll, I'll just, I'll just, uh, I'll skip to a couple of case studies then.

Um, so two, I have two, I have two case studies for you. This is a customer, they came to us. They had, um, uh, a lot of documents.

Uh, this is actually not an IT or a networking use case. It's straight up, we have an enormous cache of documents. Each document is very, very large.

We need to be able to, uh, to query those documents. We currently have a team of 12 people, and it takes them a long time to, uh, find the documents that we're looking for. Can you do this with an agent?

Somebody said, yes, sure. And they built a prototype and they did the demo and they said, this is fantastic. And so they tried to use it and it broke almost immediately, right?

So then they came to us and they said, can you do better? Um, where we said, well, we're certainly gonna try. And so, uh, we built, uh, this using our middleware with all of the features that I've been talking to you about.

And so some of these questions that they had, like, you know, um, were very basic questions. How come one time I tell you to find a matching documents? You gimme 12 results, and sometimes there's only three.

Uh, this is the kind of stuff that drives people crazy when they use agents because it's probabilistic and it's, you know, not always the same. Well, we want to get to the bottom of that. So a lot of the techniques that I've been talking to you about, we developed while learning how to do this right.

Um, and, uh, and, uh, so we were able to, you know, deliver consistent results, um, and convince the customer that basically with the agent in a few minutes, they're able to get the results that are better than what their team of 12 people were usually able to accomplish in the two or three hours. Okay? So that's, that's one use case.

And it involves, like, enormous amounts of large documents. When you're prototyping, you would never use, you know, 50, 60,000 token documents with lists of thousands of them. Where when you try to find matches, you get 150 matches.

Like how do you, how do you handle 150, 60,000 token documents in an LLM unless you've got some sort of a context management strategy. So that's, that's what we're doing. The second case study is, um, is a case study where the, the customer had an enormous number of systems.

This one came up a little earlier in a previous presentation. So it's a very heterogeneous environment. Each one of these tools provides an amazing dashboard that lets you look at their, their information, right?

And, um, and so they're like, don't replace, um, there was a question earlier about data lakes. We can do a data lake, but they didn't want us to do a data lake. They said, no, no, we have, we're very happy with the job that each one of these tools is doing, gathering information, syn synthesizing it.

It's just that in our business, sometimes issues come up that span these domains. And right now I have to go and sit in front of one beautiful dashboard after another and correlate between them. And a person has to do this work.

Can, can an agent do that? So we took that on, uh, and, and we, we, we built, we built an agent. Um, and we basically, we, it's what I was just talking about was the, the, uh, dynamic data discovery is what led us do this a little bit better than just federating all of the data and ingesting it into our platform, which is what a lot of people would say is they say, oh, the data exists in all these places.

Why don't you just pull it all in and, um, and into our system, and then we can do all the analysis. Um, certainly there's times that we do that, but with our platform, you don't have to do that because the, uh, the, uh, middleware and the, the dynamic, uh, schema discovery is gonna be really good at, uh, understanding where the data lives and then going to get it when you need it. There's another thing that I really want to show you guys.

I, I don't have time to, to to do the demo, but I do want to talk to you about the fact that we take a very different approach to, uh, evaluations as well. I, I mentioned this at, at the top right. Um, a lot of times you'll see, uh, observability platforms that are simply, um, evaluating how many tokens, how, what the latency was, uh, whether there are any, uh, tool errors or anything like that.

Um, but we have this really nifty system, um, that, uh, once a agentic session goes idle, we'll go back and review the entire session and perform a qualitative analysis of it, and it'll pull out various themes, what the topics that were covered were any lessons learned, any, uh, optimizations that are possible. It creates these reports. It scores the agents dynamically, it considers any user feedback as well as any interactions.

Uh, and then, uh, it creates, um, an improvement strategy, kinda like a pip for each of the agents that the, uh, agent administrators able to click through and apply on the go to update instructions, uh, over time and, and guide the agents to, uh, better performance And institutes. The first time I've heard putting your ais on a pip. Yeah, well done.

Well, you know, because we're giving them jobs that are human-like jobs. And this is kind of like, um, not only like the, the first point I mentioned where I said, push as much of your job into the tooling layer as possible. Mm-hmm.

But then there's still like, what are you asking the agent to do? And I, I really think what you want to ask it to do is something that you might ask a person to do mm-hmm. That they don't want to do.

Like it's a person like activity that sits low on the person, like enjoyment scale. Right. Get the agent to do that.

Right. And then measure them, And then measure them like people, did you do a good job? Do I need to train you?

Do I need to give you additional information that you didn't have before? Right. Yeah.

Okay. And I think, uh, what, maybe since I have 15 seconds, we do have spend management and cost controls, uh, built into the platform, which is sometimes people forget about that. Agents end up costing you hundreds of dollars.

It's kind of like, uh, back in the nineties when the kids used to call the party line, um, we, we have controls to help, uh, Hundreds. Where are you getting your cheap ais from? Thank you very much for.