Delivering Nokia Enhanced AIOps with the Right Foundations

So kind of the next, uh, the next part of the presentation, we're gonna, we're gonna pivot a little bit and start talking about some of our AIOps capabilities. So I've got two different setups. I'll like introduce the topology again that we're gonna use for this.

We're not gonna configure anything in this next part of the demonstration where it's like operate the network and show some of our, our AIOps capabilities. But there's this concept of agentic AI before we jump in. Um, those that again, will remember last year or have seen edda in prior sessions, know that the platform itself is very generic.

It's more of a framework, more than it is kind of a whole, a whole bunch of hard coded kind of configuration of abstractions. And we showed this last year by introducing you all to our application store or our app store, um, where you could go in and upgrade different intents kind of independently of all the other intents in the system, um, and extend the system kind of in real time without upgrading the underlying e DDA platform. So we're gonna show some of that, uh, throughout today.

And, uh, if I look at it through the lens of AIOps and specifically agentic ai, and, uh, I promise it's during time you'll hear me say that term, the idea that you kind of have the central brain and obviously we have one of these in now and then you have a bunch of, uh, agents that get kind of plugged into it. Now we really wanted to apply that concept of extensibility to these agents as well. So we wanted to be able to, in the field, kind of add new capabilities to our, our AI engine as we call it, um, without you having to rev the underlying platform.

Uh, and we already have a bunch of these agents. I've listed some of them here. We'll go through I think most of these, uh, throughout the next little while.

Um, and, uh, we'll do this all using our app store. So we already were consuming intents using this app store. So we have, you know, an intent that deals with fabrics, for example, uh, we were dealing with workflows.

This was like the ability to kind of like write a method of procedure and code and then kind of roll that out across, uh, a fairly complex topology. I think we showed, uh, image upgrades last year as the example for that. And now we can actually extend, uh, our AI engine and I will do my best to show that over the next little while.

So I'll come back to my topology here and log into my second instance. So quite a different topology. Uh, I've got two spines.

I think I have 20 leafs and two border leafs or just scroll across. So a little bit more of a complex topology, a more real world topology that you'd find in your, in a pretty typical data center. Um, what we showed again last year, and I'll do this again just for emphasis, is we can run queries and we actually introduced our natural language, uh, at the same time.

So you can do things like list me, my SRL interfaces, and obviously this is going to use, uh, rag and a bunch of the, the techniques that I'm sure you've heard ad NASM over the last little while to figure out the correct yang path to go and query in the network. And then to return it back to you. So we showed a, a bunch of different examples of how you can use this to query your network without really understanding what the underlying operating systems are without really understanding YANG or the schema.

You kind of ask questions and kind of human friendly language and we'll do the hard work to figure out the correct data to serve to you. Now, there were a couple of limita uh, limitations with this. The biggest is that you kinda lose your context.

Uh, you notice there's only like a single input box here. And so if I were to ask multiple follow up questions, you can imagine I was debugging a fabric and I was looking at interfaces, but then I was also looking at BGP peers and I wanted to look at the routing table. All queries you can run, but anytime I run the next query, I've lost the previous one, my context is gone.

So if I wanted to compare things or, you know, compare A BPP versus the interfaces that are down, that type of thing, it was pretty difficult to do it through this interface. So this is where we introduced our, uh, our aska. I'll just start a new chat here and I can do much of the same thing.

List my SRL interfaces. This one supports multiple namespaces, so I'll add that too. And, uh, IDA namespace and it's gonna do much of the same as we just saw it in the query.

It's gonna find the best table. It's figured out that it's namespace node SRL interface and uh, it returns us kind of a live table here. Now I can ask follow up questions so I can say, uh, only show me those that are admin enabled.

'cause a bunch of my edge interfaces are not actually enabled here. So I can ask kind of follow up questions that holds onto context. You'll see it's tweaking the query.

I won't go through kind of our query language again since we've, we've beaten that to death in previous sessions. I'm so glad you asked. I was literally just about to click.

So Yes, I've, uh, I've actually like talked to some customers around this and they're like, that's my new ui. This is the only way I want to interact with the network. Um, we can of course, like filter more.

We can say, uh, there's a lot of fields here. Uh, only show me maybe I care about, uh, I index, which I can see and Mt u and it should further filter this down and only give us the stuff that we're interested in. So I've got a question for you.

Yeah. So is this on, is this a closed data set and that it's only able to work with or analyze what is fed in through EA or like for example, you know, let's say on one of my border leaves I've got a public a SN or something like that and I see that in there and say, who is this a SN? What is this a SN?

Can that go out to an RAR and look that data up and return it through the same interface? Yeah, we don't have that capability yet, but yeah, absolutely. This is kind of through the concept of those agents that I was just introducing.

Okay. So I'm only focusing on one agent right now, which is just our query agent, which is go and grab data. We have agents that deal with kind of topology and understanding addresses and adding one that understands ASNs would be, uh, would be pretty trivial, I would say.

Okay. So it's, it's not there today, but for sure that's the type of thing this, this platform would be capable of doing. Yeah.

Um, and I will say the only thing it's really dealing with here is it knows the devices that are underneath management based on the schema. So you kind of, when you onboard a new operating system or a new vendor, you give us the yang models essentially. And that is what it is doing kind of that, that, that augmented retrieval on is it's pouring that into a vector db, it's doing similarity search, all that type of thing to figure out what to query.

So this extends as you add new operating systems using those new yang models, that type of thing. It's based on what is currently in the system versus some kind of closed data set that's fixed in content. Okay.

And that's operational data as well, not just configuration data. So if I wanted to look at drop stats across these different vendors, I could say, okay, show me. Absolutely.

You know, Hey, show me all the interfaces that have drop stats on them right now, and whether it's Nexus or it's Arista or whatever. Yeah, exactly. You could return that back through that same interface.

Exactly. Yeah. So right now I'm querying the node native model that we have been equivalent for kind of, uh, normalized, uh, drop counters, for example, as you were alluding to.

So right now I'm kind of querying a specific set of devices, the Linux devices in the topology, but asking it for drop counters through the normalized model would give you that, where again, there's a single model that is grabbing everything from Okay. So yeah, absolutely. That's the type of stuff this thing can do.

Cool. Uh, one other follow up here. Once I'm this far into this, uh, can I change the, the context in this same prompt and expand now?

Like say I've, I've now seen this and SR Linux, can I go back and say, gimme all of them? Uh, you can, I only have Estro Linux in this topology, but, but yeah, you can, you can also like just create multiple chats similar to like you've seen with chat GPT and other kind of chat clients. You can see I have a, I have a bunch of them over here, so you can create multiple contexts for querying stuff, but you can also change your mind.

Yeah, if you ask it for, I'm dealing with interfaces right now. I'm gonna jump and ask it questions about a bunch of other things and it will follow the context along and hold onto it the same way you, you'd be familiar with, with other kind of jack clients. So, uh, tables are all well and good, but they're a little boring to look at.

Um, so we also support, uh, some of our other visualizations. You saw our dashboard, um, and I think last year I also introduced our dashboard designer. This was the idea that we're streaming all this telemetry in.

Obviously we can put it into tables and refresh that data as it, as it kind of gets refreshed from the nodes. We can also do that for a bunch of other types of visualizations. So I'll just, for the sake of time, actually, I will jump to copying and pasting everything, but, uh, I'll do this where I'm gonna create a pie chart of BGP neighbors on my SRA nodes.

I do have a, a, a fabric, uh, already deployed here. Uh, yes, I do already have a fabric that I deployed earlier, a much simpler one. This one is just using, uh, V four V six.

There's no BFD or anything like that. Uh, but it should create me a pie chart here. And it does.

There's 328 BG p piers, which is kind of cool. We're gonna make this chart a little thicker and like thick charts so you can actually modify and iterate the visualizations that it's creating for you. And it should give us the same thing, but ideally with that being a little thicker, very nice.

Uh, else can we try here I will build a counter chart of the interfaces on SRA again, so a different type of visualization, but visualizing some of that same stuff that we just saw. And then while that's running, I will, uh, pump this back across. Now it's all good to have it in the chat, but what we wanted you to be able to do was actually design your dashboards using this.

So I'm just gonna jump into our dashboard designer here. Now, you could obviously go and create these things yourself. Uh, the dashboard designer lets you do everything that you, you can see on the right here, but it's a bit easier if you can just drag and drop like this.

So I'm just gonna add a couple of visualizations here and we can see that it throws 'em in. You can save that and share it with your friends and colleagues. Um, and another cool thing we can do, I think I actually probably already ran this earlier, is, uh, obviously these LMS are very, very good at, uh, kind of translating between languages.

So I was actually just in Osaka last week. We had our SR experts conference, and I demonstrated this where we can actually ask it questions in Japanese and hopefully we going, well, it will build us a line chart for interface traffic rates in bits per second. That was what they were saying for those who missed my, uh, my chart here.

And we get a live traffic rate in our chat here as well. So rather than having to go and design the stuff yourself and, uh, that being a bit of a, a hurdle to get over. Uh, we tried to make the designer as easy as we could, but obviously it's super powerful.

There's a bunch of different visualizations, there's different schemas you can pull things from. Uh, there need, it needed to be a little bit easier and, uh, you know, AI is here to save the day. So a question for you on, uh, the visualizations.

Um, where all are you going into visualizations beyond just, um, beyond charts? Is is topology, like physical topology? Yep.

Logical topology possible. Yep. So the topology you saw me use earlier, we don't have the dash lit version of this as we call it just yet, but we are actively turning this into a dash lit so that you can, um, for example, visualize packet loss on top of this type of topology.

That was gonna be the question I was gonna ask. If, if I, hey, you know, color all the lines that have, you know, know this percentage of packet loss, red or orange or whatever, or show me down links in red, whatever that is, you could, you know, build yourself a custom map of Yeah, exactly. The operation show just yet.

But absolutely we, That's the, that's the path you're headed towards. That's the path We're towards. Yeah.

Cool. Okay. So we have some, and, And, and after we've designed this, you can automate, say, run this every night or week or Yeah, whatever, and alert me if x, y, or Z limits are.

Yep. Remember the nice thing here is that we are streaming everything. So you could just leave it running, save it as a dashboard and then say it triggers on top of the dashboard triggers.

Yep. Yep. So you don't have to necessarily say, uh, come in and do this at some interval.

We can do that all for you. Yep. Okay.

So we can query the system, we can build these different charts. We can throw them onto our, our dashboards and build dashboards with this. There's some, uh, some other fancy tricks we have as well.

Uh, I have this fabric here, so this is the fabric I deployed earlier. Again, it's a much simpler one than the one we, one we saw before this open this. We can say, what am I looking at?

So this is one thing we can kind of do because of, uh, embedding our AI kind of natively in the UI so it actually understands what you're looking at and you can ask questions about what you're looking at. So you can see it recognizes it's, it's dealing with a fabric here. The US West Sunnyvale one fabric in the Eden name space is BFD enabled for this.

So it's gonna have to actually grab the resource here and it's gonna think and it's gonna tell us that yes, BFD is enabled for this. So you don't really have to understand this kind of schema you see here. Now either you can imagine someone else built this fabric and I just wanted to understand stuff about it.

So we can now, uh, we can now, uh, ask questions about it. I can do things like, is this before RV six? And uh, that was a trick question.

It's actually dual stack Question that now that I saw your, your BFD settings, it, are you far enough along where you could ask it for its opinion? Could I say, do you think my BFD timers are too aggressive? Is it, are you at that point?

That's A great question. Not, not yet. So what we've started to do is embed, usually people jump to embedding documentation straight away.

We're actually doing that kinda last. So we do have a bunch of that stuff in our documentation. Um, and the documentation is already written at markdown, which is nice and easy for the AI to understand.

So yes, you will be able to do that. We're certainly on that path. I can't show that today, but Okay.

Yep. You'll be able to do that. The other thing we are focusing on, you'll notice I'm not gonna make any changes to anything today because one of the things that all of these, uh, LMS do is they hallucinate, uh, and I mean they hallucinate not that all felt like it hasn't done it yet, which is, which is great.

Um, I've run through the demo a bunch of times and it hasn't done it. Um, but when you start thinking about making changes, you need to be able to validate that the change is gonna have the desired effect that you want. So our kind of solution to this is that whole digital twin thing that I talked about.

So you can imagine you have like a, a pipeline of changes that you wanna get into the network and they can either be suggested by humans, they can be suggested by the AI with kind of a human overseeing it. They would run through that digital twin first. We have the ability to run like tests.

You can do reachability tests, you can do ab testing, that type of thing. And that digital twin, and only if that kind of succeeds based on those automated tests, does the change role into production. So we're planning on delivering that in 26 4.

Obviously the digital twin I'm playing around with today. But that whole kind of structured pipeline approach, we're delivering an April next year, and then we'll allow this to start making changes. How Do you deal with problems in a digital twin when you have hard things that are in hardware?

Only if I have, you know, specific QOS settings or things that you really can't replicate in, in a vm Yeah. In your digital twin. How does that happen to be short?

We don't try. Okay. So I mean, even for our simulators, if we were to try and honor the kind of various data paths that we support, and we do a fairly good job of doing that, um, when we apply this to some of those multi-vendor scenarios, we can't guarantee that their data path will function the way you expect.

So the shortest we don't try, the only thing we will do is obviously we do all the schema validation. We can do things like compare routing, um, we can do, uh, all of the control plane obviously. So the, it's a true kind of implementation of the control plane, but we don't try and simulate data path.

We don't try and do hashing or any of that type of stuff. We don't try and do fib compression or tcam compression because software is always going to fail at those types of things. Sure.

Yeah. So the shortest we don't try, um, with configuration and control plane, you get vast, the majority of the way there of the types of changes that humans make that are gonna run into trouble if you kind of make them incorrectly. Got it.

Versus like, yeah. Is is this going to cause contention with new QS settings? That's not the type of thing that we're gonna try and catch in the simulator.

Okay. And, and one, one other thing on on that, uh, if I understood the first part of this correctly, you have a template that's, that's applying these things. Is, is this capable of comparing the, the current state against templates as part of a, a thing to look for config drift and that sort of thing?

Yeah, so we catch config drift using our deviation framework. So we actually subscribe to the configuration of all the devices that we're managing. And if changes are made kind of as they're made, we get in a notification to say that something bad has happened.

Um, we also are vision controlling all of the resources, uh, like this fabric resource, if I were to make changes to it, those changes get saved in GI and uh, I don't think I have, I only have one version of it right now. But we can compare the gap delta between those different versions too. So we cover both the, the EA making changes, we can compare those and someone coming into the node and making changes.

We, and then the follow up question is, I had to drift because customer B requires this drift. Mm-hmm. And I want to approve that.

Are you able to save that? This particular change is allowed to drift? Yeah, you can.

So, uh, we'll alter deviations now. Actually that was gonna be part of my, part of my script anyway. Sorry about that.

No, I mean, you're, you're leading me along. It's great. Uh, actually what I'll do is I'll go and, and shut down a p or something.

So, um, I'll pick leaf one neighbor instinct disabled. How am I doing on time? People speed up.

That's what I heard. So, uh, actually I can show the next part of the note. We'll, we'll, we'll show the, the drift in just a second.

So I just shut down a BG P pier and I have a couple of alarms saying that A BGP pier is down. Obviously that was done outta bend. If I'd done it through Edita, we wouldn't get the alarms 'cause we kind of know that it's meant to be, uh, shut down.

So if I select one of these alarms and we have this, uh, these stars are meant to be the sign for ai, everyone keeps telling me that. I'm not sure I buy into it, but we're gonna ask our next tattoo. Yeah, exactly.

We're gonna ask Edita to try and solve the alarm for me. So you say it's gonna fetch the alarm, it's gonna think it's actually gonna go through and do a bunch of things. Um, and eventually, hopefully it's gonna realize that there's what we call a deviation, which is someone's gone and changed the config.

That's why you have this alarm. Um, we didn't teach it about the deviation, obviously, especially the specific one. We taught it that deviations can exist.

Um, we didn't tell it anything about BGP, so there was no, like, we didn't train a model or anything here. This works kind of generically across all of our alarms and I have really no idea how it's gonna try and reason its way through how this alarm is, is there and what it needs to do to fix it. But it does have a bunch of kind of threads that can follow.

Um, for one, we do this thing where we kind of give you the path that we, uh, we saw this alarm on. So it knows that there is A-B-G-P-P. It's gonna go and fetch that and try and figure out if there's anything wrong.

Uh, at this specific path, it's gonna fitch deviations for this node. 'cause it knows which node this was triggered from. I actually triggered the alarm just by chance from the opposite side, I'm guessing.

Yeah. So I triggered this, I shut down on the leaf, but I triggered the LLM from the spine side. Uh, it has these resources that it can go grab, like we have an interest switch link resource that actually controls the generation of the BGP pairs and everything.

So it can go grab those resources and check those. It can check transactions. Uh, and eventually we usually see it's grabbing a bunch of different js paths on, on the device.

And I'm hoping it will figure out that there is a deviation on this device. Uh, while it's working. I can, uh, it's bigger, actually go to my deviations.

So you'll see this is the deviation we just saw. I can like open this up and it will tell me, found that right away. Yep.

That the intended value is enable, the running value is disabled for this specific P pg PP. So it will actually, I'm, I'm hoping once it finishes reasoning here, give me a direct link to this deviation once it finds it. Mm-hmm.

Uh, and then you have the option of either accepting, let's say if this was actually good configuration so you can accept this and we'll just keep it. That's now part of the golden configuration for this device. Or you can reject it, in which case we will go and eliminate it from the, uh, from from the note.

And if I accept that deviation, it's just for this specific. Just for Specific. It's not changing the overall.

Yep, Yep. Exactly. I've Got a question, quick question for you on the deviation.

So let's say I accept this deviation and let's fast forward a week. Mm-hmm. Do you have the ability to ask it, the, the network, how has the network state changed since I accepted this deviation?

What is the, is it because you're streaming the telemetry? Can you see the change in the operational state based on this change? I mean, that's one of the hardest things to track back and network engineering is things are different.

A change was made. Yeah. You don't really know why.

Yeah. So we are adding, uh, some of kind of the more native time series. Obviously we can stream this all out to Prometheus and influx, your typical time series, uh, including the deviations.

So those can be streamed out as well. What we are adding is kind of exactly what we're talking about, where we can say, here was a configuration change made at this time. 'cause that deviation now just looks like a config change to us.

It happens to be kind of a, a raw one, but it just looks like a config change. And then being able to trend that from the time, like you would tell us like, this is when the deviation occurred, what has changed since we would be able to answer those types of queries. Yeah.

We don't have the native TST B hooked into this just yet. So again, that is, uh, uh, something else we're, we're adding, uh, in the, in the short term. So just briefly, um, my name's Taryn.

Um, I've, I'm curious if you've thought about training the model on, you know, you said you didn't train it on BGP, right? Mm-hmm. I'm curious if you've thought about in the future training and on BGP training and on OSPF and being able to say, you know, have the model come back and say, well, based on what I'm seeing, make recommendations to changes in the network and things like that.

Yeah. You know, make it a little bit more smart about, about, uh, the technology. Yeah, I mean, we are kind of actively looking at building our own kind of networking model that would have a bunch of this understanding.

Our labs team are, are working on that. So short answer is yes, absolutely. The nice thing is like, uh, because we're not tied to any specific LLM, so this can work with on-prem language models, it can work with open ai, it can work with xai, it can work with all of them.

You see, it finally linked me to that deviation here. Um, because we can do that and it's still, I would say like the, the hit rate that it has is ridiculously high, like ridiculously high. Uh, we haven't had featured anything yet where we're like, oh, this is kind of something it struggled with.

Except in cases where there's like a complex task, like, uh, it needs to understand how we're gonna route a packet from one node to another and maybe that's on an overlay and there's indirect next tops, that type of thing. And actually, 'cause I'm running miles over time, I'm being told, uh, I'll, I'll reject this deviation. Um, if I, uh, let's assume that I, I found there was a problem on one of my nodes here, like, uh, my leaf one, in this case, we can do things like, uh, run a tick support on leaf one and it should be able to figure this out for me.

So this is, uh, kind of the way we're teaching the model how to do stuff is by adding workflows or those tools, kind of the, that agentic AI approach. So here it doesn't obviously know how to run a tech support, right? That's something especially on our devices being a, a leaf one.

So when we need to teach it something it doesn't know, instead we kind of expose it as a tool through one of those agents. So in this case, how do we generate a tech support? Um, what I'll show before I get kicked off the stage here is that right now it doesn't know how to turn on locators.

So we have locator locator LEDs on our devices. Uh, I will try to, uh, can you enable the locator LED for LEAF one? And I'm hoping, yes, it says that it doesn't know how to do that.

It gives you a suggestion because obviously it has access to documentation. We're using open AI here. So it has access to a bunch of, of different ways to try and figure out how to do it.

What I will do is come into our app store here and I have this application called support and this has a workflow in it for a locator. So I'm gonna just install this. So you can imagine that, you know, someone released this app last night and you're like, oh, I really want to be able to turn on locators on nodes.

So I'll refresh this. Can you enable the locator on leaf one? And hopefully it knows how to turn on locators.

Now you can see that it realizes there's this new workflow that it has access to and it's turned on the locator. So this is kind of showing the idea of being able to extend the capabilities of the, of the, uh, of the AI kinda in real time. And the last thing I would do before everyone gets very upset with me is, uh, I will, I'm on my Leaf one here.

I'm going to look for our BGP process. This is ESTRO Linux. So obviously each one of the, the processes manages specific parts of the system.

We're going to kill this and I'll come back here and come to my alarms and we should see, I have a bunch, although they're gonna start clearing very, very quickly. So we've got a bunch of different alarms. Uh, I myself come from an operations background.

I used to work in the, I remember kind of that scary thing. It's 1:00 AM you get the flood of red alarms. You're like, oh, so what I'll do here is what just happened.

And what this will do is it will go look at transactions. You can see it's gonna look at no, oh, it found it immediately. That was nice and fast.

So it actually found that between LEAF one, spine one and spine two, there was a bunch of problems. It saw that deviation that we saw on leaf one, which, uh, which is pretty nice. Uh, and it sees that the BGP manager process crashed and was restarted.

Estro will obviously detect that that process failed and it actually spins it back up. So it recovers kind of nicely. But this is kind of the the last thing I think I wanted to show, uh, before my colleagues get very, very upset with me.

Thank You. I'm gonna ask one more question, Mitch. Yep.

Um, you, I think you said it would show side by side configuration changes if something externally changed. Yep. Can I summarize that for you?

Here's the five places that I need to show you where things have changed and what it is. Yeah. Before you go, because you may not know all those configuration details, right?

Yeah, exactly. So one of the other things I was going to show, I, I weren't just for the sake of time, is we have stars in the transactions as well. So I actually went and intentionally failed a transaction here by trying to emit raw configuration to a node that I didn't like.

Uh, I won't run it again just for the sake of time, but Yeah, exactly. Because every change, whether it be through a deviation or someone configuring or directly or someone doing something in Edita gets registered as a transaction and we have the ability for transactions to say like, help me out. I don't understand what this transaction means.

Uh, 'cause sometimes you can see here, like that little tool tip, we are a multi-vendor platform. Now that error is coming directly from the node, so we're not trying to digest it for you or anything. Mm-hmm.

And obviously it's a very qy error there. So we realize that this is a great place to kind of give you recommendations to understand why something failed, uh, and to be able to compare them, uh, against the history of this resource. Okay.

Thank you. And apologies. Yeah.

Multi-vendor example is perfect for that. Yeah.