Juniper Marvis Minis, LEM, GenAl/Customer Troubleshooting, and Marvis Actions 2.0

Marvis Minis. This is Sear's favorite baby, new baby in the block. Uh, if you listen to Sier, there's only two problems.

Life should be simple, right? Either wifi and Connect. I don't know if anyone's familiar with that problem this morning.

Our wifi sucks, right? And as long as we have users on the network, we have plenty of data to understand their post connect pre connection and post connection problems. But what happens, we don't have users on the network.

That is where Marvis Minis comes into play. Marva Minis is the first synthetic user on the market right now that allows us to ensure that all critical network services and critical business applications are actually up and running before the business opens in the morning. So this is what we announced last year at MFD this year.

What we're announcing is Marvis Mini's client to cloud application visibility. This is where Marvis Mini's now has complete visibility from the client application to the application in the public and private cloud. So we have synthetic users now going from everywhere on every missed device, and in the internet pops across the internet.

This will allow us now to get to the root cause of a client, the application problem, and isolate it down to the router or the service writer in between those two aspects. And what you're gonna see coming up here in seds dim is that Marvis Minis, SLE, that is gonna be the industry's first way of actually taking data and getting to the root cause of that client cloud user experience. Sadir.

Alright, thank for razzle Dazzle. Alright, good morning everybody. Thank you so much, uh, for hosting us.

We are excited to be here. Uh, you know, if you haven't, uh, seen the press, we actually do our big press announcements. Uh, like Sujay said, uh, on Mobility Field Day, there's actually a press release that went out today around this Marvis client to cloud story.

We are very, very excited because, you know, as we look at this thing, Marvis Minis has become sort of the new source of truth in networking. What does that mean? What that means is, you know, Marvis Minis doesn't take a weekend off.

They don't have a bank holiday. If they're in Europe, you know, they, uh, they, you know, it doesn't actually work on a, a, a laptop with CPU pegged or is in a a bad wifi area. Marvis minis has become deterministic.

So with that determinism, what we've been able to do is actually at an entire organization level, I want to say, Hey, I wanna understand how network services are doing at the full organization, right? Because very deterministic data is feeding into this to say, okay, which sites are having DHCP issues, which, you know, which locations are having, uh, D-H-C-P-D-N-S radius issues, you know, being able to understand at a global level for an organization. Then we say, okay, this is great, but what about, you know, applications now for the first time?

This is the Marvis Minis application service level. What this is doing is at an end-to-end level, being able to say which sites are having failures, and for an individual site where is an application degrading? So think about, you know, a a million minis around the way, uh, around the world feeding into my application experience.

And basically, you know, for us, you know, most of the other systems in the world focus on, in the industry, focus on observability. For us, it is all about actionable insight. We want to bring you to an actionable data point to say, Hey, for this user from, for, for on this application at this particular site, it is either the wireless, the wan, the lan, it may be some somewhere deep into the internet circuits, uh, you know, maybe it is closer into the public private clouds, you know, so now with minis deployed globally, right?

Uh, we've taken an architectural approach to this. Historically, every other vendor wants you to deploy overlay sensors, agents on laptops, raspberry pies, you know, uh, you know, costs a lot of money built into the subscription of current Marvis subscription for our global customers. Marvis Minis client to cloud means that architecturally no sensors, no agents, no raspberry PIs, no deploying on your, you know, an agent on your router minis is running across the global internet across all the pops.

And so what does that mean? What that means is now there are minis running in CDN pops around the world. And so we are establishing this last mile baseline for every one of our customers.

And so now that we know this, a million minis around the world are contributing to the answer on if one individual application is slowed down at one site at several sites, multiple applications at a site, multiple applications across the entire organization, we're able to actually pinpoint to be able to get to, you know, ease it an application issue, ease it a a point in the circuit, where in the circuit, uh, you know, is the application degraded This entire picture. We've now appended to Marvis Minis. Last year what we launched was on the lan, uh, wireless side, uh, with an agent sitting on the, uh, on the SD van circuit as well, right?

So this is, yeah, Peter, good. Um, You've talked about across multiple sites within an organization, but are you doing anything across organizations that there's a cloud based application that's gone down at to cloud level that's used by multiple organizations? How are you looking at that?

Yeah, so, so actually, uh, the, the reason we keep saying a million minis around the world is the, the millions of, you know, Juniper aps deployed out there all contributing to the health of advanced circuit, right? And so if if everything's pointing towards, hey, you know, my SAP application or my Salesforce application is degraded, yeah, that's a global thing that we're learning, but we want to localize the insight. I'll give you an example.

We had a, a customer open a ticket saying, Hey, I have 10,000 aps down, right? And, you know, we step back and say, 10,000 apes don't commit suicide at the same time, you know, what really is going on? Look back at other customers.

And they've had outages as well. And turns out, you know, um, a, a regional pop in Western Canada was down. So now with this, this insight, Peter, what, uh, what Marvis actions is gonna show is not 10,000 aps are down.

Hey, you, you know, uh, um, you know, for your own network or someone else's network, it's going to say this regional ISP invest in Canada is actually down. And, and that's a global learning, right? So that, that's the whole point of this thing is the global learning.

Yeah. You said you're getting the data on the pops. Is that because you've got agents or business partnership within the pop, or is that coming from the sensors or the organizations that are connected to that pop, Um, uh, within the pops as well?

Yeah, and, and we are, we are managing that globally. Customers don't have to, uh, lift a finger for that. And if a customer did want to deploy one of their own agents at a POP that you did not get service correct, how Would they do that?

So, uh, that is part of the future roadmap. It's not there in, uh, on our first release, but we do want to enable customers to be able to deploy Marvis workers, uh, in their own data centers in, in, in A VPC, all the, all that kind of stuff. Yeah.

Where that's not part of, you know, this, this initial, uh, realm of the launch. So you're talking about taking pop data, Marvins minis there and using every other customer that has minis enabled to be like this, distributed sensors. Mm-hmm.

And you're then using your own learning based on all that information to tell everybody where, Yeah. So, so again, if it is within the network, wireless LAN van, obviously that's your own data. If it is on the wide internet, and we are realizing, hey, a Western Canada POP is down now for every organization in their own Marvis actions, it's gonna say, Hey, it's this ISP that's contributing.

Right? How they got the learning is through their own sensors or through other people's sensors. Yeah.

Okay. Super. Very good.

Uh, with that, Bob, uh, you know, we'll, uh, we'll actually maybe a couple quick stories. Marvis minis, because it has become truly the source of truth in networking, uh, has become very deterministic in, in, in, uh, delivering some amazing customer outcomes. Uh, this is an example of a a a, you know, a a co a company where we were detecting application reachability issues.

Uh, they were saying, well, nobody really opened a ticket yet. Uh, again, uh, Mar every Marvelous Mini's failure has a dynamic packet capture waiting for you. And that basically, actually, it's, it's, imagine having the video footage of every crime committed in the world, and this is not even a real crime.

It's a simulated crime committed in the world. We have the video footage waiting for it. So they went back and, you know, a good old MTU problem, uh, actually turned out to be, uh, what was causing this.

And, and so again, uh, without users having to experience failure, we can get that, get them ahead of this. Similarly, we find a lot of scenarios where, hey, uh, all my SD van is deployed. All the hubs should be active and up and running, uh, active, active scenarios.

And sometimes one of the nodes is actually not, uh, a real customer, uh, in a large retailer, we found. Uh, so we are finding all kinds of interesting scenarios with Marvis Minis. With that, Bob, over to you.

So Bob, Bob or Sadir, can I ask, um, so it sounds like Marvis minis is enabled on the access point. Yes. And so then my question is, you said if, if it's enabled in the organization, what, what does that entail?

And then what kind of telemetry is it spending up and which organizations might choose to do or not do that? Good question, jj. So basically, um, at an organization level, they can, uh, turn it off.

Marvis means if they would like to, right? Uh, so step one, but to enable it means that they will come in, they, there is no configuration. They will auto learn what SSIDs there are, meaning what s what VLANs those SID drop into what the H-C-P-D-N-S radio services are.

We ought to learn all of that. And then we start to validate. And, and from a workload perspective, some customers say, oh, my God is, am I network gonna get overwhelmed with all this synthetic testing at all times?

We're watching that our workload is less than 5% of the overall workload. That's number one. And so, um, but from a configuration perspective, enabling IT perspective is really one checkbox to say, yep, I want this, or Nope, I don't want this.

The only other configuration that we allow at this point is, Hey, I may have a private application that you may not have learned or known, and so, you know, go to this IP address, do an HCTP curl, get me if, if I'm getting a, a reachability response. Uh, so we can actually define in this new context of deep into the network, you know, application visibility, uh, you know, we have a certain set of applications we're gonna let them do, you know, complete visibility into there. We're doing continuous monitoring, literally, uh, you know, every minute, uh, I know lost latency jitter across the entire van path.

Okay. So that makes sense to me for everything inside the organization. Yeah.

But you also kind of mentioned this, the kind of distributed prop pop that everybody participating in this client to cloud Yes. Experience reporting. So I'm assuming it's also reporting up in for that?

That's correct. If, if, but if they turn it off for their organization, there is no last, last mile, mile latency, none of that for them, because they've turned it off. Okay.

If they're enabled, then we are learning that last mile latency across the global CDN pops. Okay. Yeah.

So large experience model, so this is where we're taking Zoom teams data. We're doing it with network data, and we're basically training models now that can actually predict that Zoom teams user experience. And then we're using some fancy math called shapley that actually looks at the contribution margin of every network feature to that user experience minute.

So this is like second generation sles first generation is around trying to get network features, how they correlated and predicted a bad user minute. This is actually looking at how each network feature contributes to that bad user minute. So when we look at latency, we can now tell which features contributing the most latency to the problem.

Now this year, what you're seeing us announcing, we're basically bringing new network features to the model. We're bringing new Zoom teams features to the model, and then we're bringing a new gen AI summarization to the shaft results. But most, more importantly, what we're announcing here today is a generalized large experience model that will now let us to predict the video collaboration user experience on any network, even if they don't have Zoom teams users.

This is what I call the needle in the haystack, right? Most if the signal's big enough, they don't mean fancy math to see the problem. The average IT person can see it.

That's why they value the SL e. But when you have what I call needle in the haystack, this is like where we had a customer that was having a problem, but the model highlighted it was really a radar channel that was causing a problem that we weren't able to see on the radar, narrow band radar. We had another issue where basically they're having problems at a site in India.

They couldn't find out why certain people were having a problem. It turns out it was a misconfigured VAN gateway. People coming there were getting routed to Australia.

We had a professor at a university was having problems at lunch. This model was able to track down. It was basically a bunch of students walking past the office during lunchtime that was causing a problem.

The large experience model is, I would say, is your needle in the haystack problem? Said, dear, you wanna come up? We'll give 'em a demo.

Oh, no, this is the first thing. Yeah. Thank you, Bob.

Uh, so basically what we are doing is the large experience model so far, uh, was, was deciphering, okay, uh, what is contributing to my team's zoom problem. Now, the marvelous Minis client to cloud data is gonna add to the features that we show under that model to say, oh, yeah, your teams or Zoom is degraded, not just because it is wifi or uh, uh, you know, or the wired or the van, but maybe it is, you know, this third hop into the, uh, into the data center that's causing that sub features. Now, Marvis Minis is feeding into our large experience model, and together the idea is that we are landing them into marvis actions, right?

Into what do you need to do? For us, the crusade has always been for Bob AI is artificial intelligence. For me, AI is actionable insight.

That's it. It's, it's not that much more complicated than that is how do we deliver actionable insight to when you need to take action, what do you need to do? So what we've done in terms of the demo, uh, in terms of the product delivery here, so, um, last year we've talked about teams and Zoom integration.

And so what that is now, you know, fully available for our customers, Marvis customers worldwide. And what that entails is, you know, if I connect to my teams and Zoom using, uh, um, an API token, so now cloud to cloud, we are integrating with teams and Zoom data and getting that data for every individual user every minute. We have an a judgment on that minute, is that a good teams minute or a bad teams minute?

So every user, every teams minute, if there's a hundred people on a teams call, that's a hundred user minutes for every physical minute. And then once you have that kind of data, our true north is, we gotta get this to, you know, a hundred percent good, right? And so here's an example.

This is Juniper's Global Network. When this model was started, you know, we had 30% failure, 20% failure, lots of actually really bad, uh, you know, teams and Zoom user minutes. This is our headquarters in Sunnyvale.

And, and you can go in and say, okay, 16 aps are having, you know, delivering some of these bad teams calls outta the 629 aps. Now how do I understand this? How do I make this better?

And that's where the Shapley model came in and said, ah, okay, we can actually, you know, pinpoint to, this was VPN, this was CPU on your laptop as causing a, causing a bad teams call, wifi switching. This is the end result, basically, where we were able to come to almost 0% on, um, the Juniper, uh, teams experience today at a global level. Now, what we're talking about is without connecting teams in Zoom, you know, God forbid you're still using WebEx, right?

Or, uh, you know, you know, uh, uh, Google Meet or something without connecting to those, um, you know, uh, collaboration systems. Can we pass judgment on your experience? So what this, you know, generalized model is saying, with no data coming from, uh, any of the collaboration applications, can we tell you if you're having a bad teams, if you could be having a bad team's experience, and if that is the case, what could the, the contributors be?

And this is the sharply model that's deciphering that. And so here's an example of an organization where we don't have any teams in Zoom data coming from that organization, and we are saying, you know what, likely you're having almost, um, you know, 175 overall bad minutes. A lot of it's is contributed by the van.

So this is the purpose of the generalized model, right? Think of how large language models were built is just learning language, learning the internet, learning the pages, everything that's available for us. What we have done is now literally billions of teams minutes that we've studied over the last, uh, you know, 12 months or so.

We now have been able to legitimately build a generalized, you know, model for collaboration judgment. And, and that's what we've been able to break through. Yeah, I really like the idea of, uh, finding like a, a general model.

I think people are expecting that nowadays with ai. Correct. My next question would be, uh, could you extend the model for other applications?

The UC I'm thinking about is, you know, university and they have like a, a online exam, and they want to know if in that room the network would support the capacity for X amount of devices. It would be great if we have like a model one day that we can just ask, you know, do you think, you know, the network would support that in that room? And then it will tell us yes or no.

You probably need to add aps or something. Like, I think it would be a great and a, um, um, enhancement to a model like this. I'll, I'll give a, an answer, Bob, and, and, uh, you step in as well.

So, Francois, the, the, the most stringent app on our networks today is the real time collaboration apps, right? This is the hardest app on our network, which is Teams and Zoom with basically real time, you know, voice and video. And so the, we went after this saying, this is the canary in the coal mine.

If, if this one is saying all green, you know Yeah. You're able to do the, the assumption is a Salesforce or a ServiceNow will work. Okay?

Or even an education app or an EMR app in healthcare should work. Okay, given this is actually declaring victory. So insert, that's why this is sort of the, the, the harbinger for, okay, what is actually the application experience on the network.

That's, that's how we focused on this. Bob, your thoughts? Yeah, I would say, I mean, the other thing we're looking at right now is that Marvis Minis large experience model, Marvis Mini is gonna bring much better WAN feature set into that particular model.

You know, the other thing, you talk about other applications, I think that's where Marvis Minis is gonna come in. 'cause Marvis Minis is gonna give us a bunch of connectivity information across all these other applications. You know, that's where we're looking to see if we can apply LY to those, that those type of models.

So I think that's how you extend what we're doing to other applications and how Minis is gonna bring better data into the existing model. And How can you, apart from installing an agent on the laptop and, uh, devices itself, um, how can you get data, uh, client data, you know, into the cloud? I know it's a challenge, but I, I'm thinking it would be great to have like a wifi frame that allows the clients to tell the ap this is how my application is doing.

But yeah, I think analog, the team, I mean, there is a marvelous s St. K, you know, so for customers who actually wanna bring the client side into the model. Yeah.

Uh, I would say probably the most of our customers who don't wanna put something else on the application, we're looking to, like Zoom teams, actually gives us visibility into the client. So usually that application, that end-to-end application has visibility on this side here, like Zoom and Teams, they know the CPU issues. They know when someone's downloading something in the background, right?

So I would say five to 10% of all our Zoom team issues are due to something here. And so that's, you know, the application itself is starting to bring us in client information. Short answer is actually, it's a great idea.

And we should take that to standards because right now, without an agent, I can't tell if the CPU's pegged on a, on a device, right? That's a fact. And so that's why we had to go down the Marvis agent.

Either we, we are getting the data from Zoom and Teams, then we don't put any agents. If we don't have that, then the Marvis client is, is an option to put out there to feed into this model. Yeah, We could start simple with just having the clients telling their RSI That would be a good Start, correct?

Yes. Well, How, how much, I guess the question is really with like Zoom and Teams and maybe other collaboration platforms, is how much data do you collect from those platforms that helps feed into the SHAPLEY model? I would say we're on the order of hundreds of gigabytes right now.

You know, so we're getting up close to terabytes of training data. Okay. Yeah.

And so how would that impact the metrics you provide here using the generalized large experience model when you don't have the collaboration? Um, or the API integrations there? Yeah, so, so Nage, who's actually building the models in the back there, I would say we are very close to making sure that the generalized model is almost on par with the Zoom teams model.

Mm-hmm. And so that's the generalization effort we're going through is making sure the generalized model and accurately predict, I think this is answering your question, right? Yeah.

How close does the generalized model to the Zoom team's model? I'll let Nage come and get the answer, but it's getting very close. Yeah.

In terms of the results, what we do is we test the network with the data, pull the data out, and make sure we get the same shapley results. Okay. Yeah.

Right. And those are getting very close, 90%. I mean, um, Wes, I, uh, I don't know if your mic is on, but, um, you know, just one of our customers has literally millions of teams minutes in a day, right?

I think in one of the large universities. Yeah, correct. So, so Elle basically with one customer over the last year per day, we were getting over a million teams minutes.

So the model actually has had enough learning to, and that it, it that you are right, it is all how much we are able to learn and the data. Mm-hmm. Uh, but, but I think we've been blessed.

Lots of customers, obviously, as you can imagine, uh, uh, you know, an organization of 30,000 employees, the number of team calls that happen. And if we measure every user every minute, every teams call, that's a lot of data. Yeah.

And that's what we've been relying on. I Mean, so the model here is more about you need a lot of data to train the model, it's back to the right data. Yeah.

You know, that's where Minis is gonna bring in much better WAN data. So we'll get much better visibility of if there's a WAN problem going on. You guys mentioned the Marvis client.

Um, are you integrating data that you're receiving from Marvis client today for the AI models and the generalized models? Or is that a roadmap item? 'cause we do have some clients that would prefer to deploy an agent out to certain laptops.

I would say that's a roadmap item. We have not integrated the Marvis SEK into the generalized model Yet. Kumar is saying, we are, we are doing it today.

Is that correct? Yeah. Thank you Katie.

Don't tell me anything anymore. I think, um, yeah, so from the Marvis client, if a, if our laptops have a Marvis client or any Android device or iPhones, we do get the CPU and the battery status and the RSSI being reported from the client, we do, uh, kind of, uh, include that into the large experience model. So it 'cause, uh, um, uh, Alan has a lot of existing Juniper missed customers.

And so he is asking on their behalf, is this something that they can use now? Or, you know, in a, in a, in a couple of months and quarters type of thing. This is, uh, in couple of months.

So right now, this is in our staging. We are evaluating, like, like Bob said, we are doing out of site, uh, model evaluations where we kind of take the data out, ensure that, uh, the without label data, we are close enough, 1990 5%, we are evolving this. And then within two months you'll be able to enable it across your customers.

Those who can't give either teams or Zoom label data, or they're using a different collaboration tool, we'll be able to enable it, uh, ga in couple of months. If you haven't figured out, Alan, Bob and I don't do any work here. We just take credit for other people's work.

He's the lead product manager for Marvis. So thank you Kumar. And we do have a question.

Um, so I'm trying to monitor Blue Sky. I think others are monitoring Twitter and other LinkedIn things, um, from Brian, who Thanks Brian, um, wants to know about the minis and can you tune in for checking, um, what itch, check what it's checking if you do something like if it realizes you failed over to like a backup link, like cellular or something that's gonna be less bandwidth, maybe higher ency? That's a good question.

Yeah. So, um, minis, uh, as they are always on, um, so Marvis is something like in the backend, in the cloud, it's, it's constantly observing, right? It can pick up config changes and it it, if it knows that there has been a policy change on any of your s dwan policy, it does understand that now it has to check the traffic path through both node zero and your, uh, backup node as well.

And it does that. And one of our customers did catch an issue as they were doing the, such a change window where their policy hadn't taken care of traffic being passed through the nor one. And Minis right away, uh, kind of indicated that, hey, this application is not reachable through your nor one anymore.

But, But, but let me rephrase. Uh, JJs question here with Brian's question is, if the network automatically fails over to cellular because there was an outage on my primary link, will Marvis now alert, oh my God, something happened? Or will it learn the new baseline?

Is that a question or, or Re let me read it verbatim. Reduce the checking. It does if it detects you're on a metered WAN circuit, like a Tail over.

Ah, Good question. So I think he's kind of getting to the how much money are you gonna cost me while I'm tail over too. That is, that is true.

So the way, uh, we kind of, uh, ensure that that doesn't happen is, for example, minis does a speed test. Uh, it's smart enough where if it's, if it's a LT link, it doesn't do that speed test on that LT link. Actually, it does understand things where, which, which uplink, primary uplink or second uplink is being used as such.

I would say the access points are very lightly loaded. They look like a a four type of call, 60 10 K bit per second call on the telemetry. Okay.

But you're, you're not gonna be racking up major. That a good question. And then I wanna ask you guys about, you know, some of the teams monitoring, 'cause I understand with all of these communication platforms, and I don't manage these, but I know with customers, you know, some of them are always client to cloud, but a lot of them have proxies on site.

That's right. Are you guys differentiating that user experience metric? So actually, uh, the, one of the most important outcomes of this Marvis Minis client to cloud is now we actually can tell you, um, uh, because we understand the entire path, right?

Not just the last mile last hop is you, we used for baselining, but we are actually, you know, tracing the entire path and saying, you know, and we have a baseline for last time, what was the latency at each hop? What is it now? And hey, was this call now going through Zscaler or some VPN?

Uh, you know, and if that's contributing to it, we actually will highlight that, right? Okay. So It's path aware enough that, you know, if it's going through a proxy, that expectation Very much path there.

Yes. On day one. Yeah.

Very good. Bob, keep going. Yeah.

So next gen AI conversational interface. As I mentioned, I have been a fan of natural language interfaces since I started this. I'm a Star Trek fan.

Talking to my computer is one of the goals of life. Um, I would say we have had a great conversational interface in 2018. It did a very good job of understanding queries, but it really didn't do a great job of answering, you know, with CHATT PT in 2022 and 2023, Marvis now has a voice, right?

And so this is where we're starting to integrate this Gen AI technology. Gen ai back into Marvis last year was all about public doc, making it easier to get those answers out of our Juniper public docs. This year.

It's all about customer data. You know, you can think of this as a replacement for Looker, tableau. When you want to know what's going on inside of your network, Marvis is gonna be able to help you understand what your network is doing.

You know, it'll be able to tell you what clients are on the network, how many clients have been on the network, how many iOS clients, how many Windows clients, basic questions that most IT people like to know without having to be an SQL expert going forward. So that's what we're doing. Sadir, This is one more, uh, this is, um, definitely a lunch bet.

I lost to Bob because when Bob in 2018 said, we're gonna have a conversational assistant, I said, who the hell is gonna talk to their network? This is networking nerds. Nobody wants a, uh, you know, a a, you know, chat bot and then chat GPT comes and, uh, you know, definitely I had to pay up on that.

And so now this is the new interface for Marvin's conversational assistant. This is, uh, um, uh, something we are excited about, um, a brand new interface, uh, essentially very chay Chachi PT ask and gives us a framework to go into agent TKI and MCP and a, uh, you know, all the new technologies. But basically now the marvelous conversational assistant will grow up into this kind of, uh, you know, interface and, and essentially be able to answer questions for previously, you know, uh, you know, uh, uh, un uh, uh, you know, previously, you know, people had to build customized dashboards for now you can ask, uh, unstructured questions that, uh, weren't, you know, nicely, properly laid out in a database.

An example being, Hey, how many of my aps across my entire organization have a hundred megabits, right? We deployed a, a, a network with 27,000 aps, uh, you know, six years ago, um, 272 of the aps came up in ten one hundred. And that was, uh, you know, the epiphany saying, shouldn't the network just tell you that?

And so that's why there's a, a marvis action for it. But you can ask a question, you know, uh, you know, uh, in terms of this new interface to be able to get answers. Now, uh, another thing is summarization of some of the, uh, some of the visualization here.

Not only now Marvis, uh, the new Marvis conversational assistant will have, you know, completely customizable visualizations, but summarization of what you're actually seeing. When did it happen? What did it happen?

Again, a lot of, um, uh, LLM uh, you know, work involved in this. The one thing that I want to completely, uh, you know, uh, um, you know, emphatically state is 100% of our customer data remains private, is not exposed to any LLMs. While we are using LLMs, um, in a lot of this formulation of the dashboards, um, you know, our data, our customer data remains completely private.

This is, you know, the, the, the tough part, it took us a long time to actually make sure, um, that, uh, you know, we, we keep this data private, keep these, uh, customer concerns of any data exposure to LLMs, uh, off the table. And then, you know, now that we have that, now that that's in the past, basically we are able to say, okay, I wanna be able to create customizable dashboards. Um, you know, the, as an example for us, you know, uh, wireless service levels and wired service levels are, uh, you know, uh, sort of how we see if the network is healthy and being able to create sort of customizable dashboards that reside in the network for us, uh, super critical right now, the last frontier here is, um, sorry, you had a question?

Yeah, my, my question's tied into the, are you learning the questions people are asking? Good question. That's, that is, that is the piece I, I will answer Keith right now, but yeah, Tie into that.

How do you teach people to ask good questions? Very, You, you've got solutions and dashboards to show them, but they, they don't even know it's there. How do they know to ask?

Great question. Uh, so let me answer the first one. Uh, first is, you know, uh, agent TKI is all the rage these days in, in across verticals, across industries.

And so our first forer into this is going to be, you know, leveraging and, and one of the aspects of agent TKI is that there is reasoning and there's memory. And, and so this is for the first time as we launch into this multi-agent, you know, um, uh, framework for us, um, under the hood for Marvis conversation assistant, yes, it'll actually learn, it'll have reasoning, it'll have memory. And, and that's sort of the, the basis of this, right?

So, you know, agent TKI, for us is a fundamentally different, um, you know, we have lots of work underway on that. And, and you'll see more, uh, you know, through the rest of the year and, and, and, and beyond. But this is our first foray into Agent TKI, you know, and, and Bob talked about, you know, MCP servers and stuff.

You'll see more from us, uh, on that as well. But, uh, that's your first question. I think your second question was how do, how do we teach people to ask good questions?

And, um, that's something I, we, I think we have to think through. Uh, uh, I, you can answer it really quickly. So, So, um, we have, um, like intent classification agents, which, uh, wherein whenever, um, you ask any questions, it could be like trend list or whether, uh, it could, like how is my network doing?

How is my network doing yesterday versus today or what for last seven days, right? Any question that you want to, whether it's a debug question or whether it's a documentation where you wanna understand how is, uh, like how do you configure like missed access assurance, right? Anything you ask, um, there is an intent classification agent that goes through, and then based on that it classifies, Hey, this is a general documentation question versus this is like a trend list or like, uh, probably like a troubleshoot question, right?

And when it kind of gets into intent further, the agent keeps classifying on every input. Like, for example, in one of the example, as we indicated, hey, mini is already indicating our failures. Like, uh, there is an action already indicating, uh, our failures.

And as we do that, like when, when we say like, gimme more details, it does understand that the user is still trying to get more details around that intent and goes in further in line that, right? So that way, yes. Um, the intent classification is the one that does the magic where we don't have to train people as it kind of as use as the questions are asked, these are saved, and then the model is trained over and over again where if somebody gives us a thumbs down like, Hey, this was not my intent.

That kind of adds to, adds to that training back to the, the next time when it answers the question. Yeah. I, I, I would, I would add Keith, I mean, I dunno if you use chatt PT lately, the user experiences slightly changing, you know, chatt PT will come up and say, I think the user is asking something about this now.

All these technologies now have history in it. So I think as people start to learn to use this technology, I think they're gonna see a history of it, right? That's the kind of how they get feedback from this is what I think you ask.

If you get the wrong answer, you can re-ask the question again. And actually the LLM will actually take that history into account. So I think that's kind of that people are gonna learn how to use this.

The Reason I brought that up is you had some dashboards that were dynamically built because someone asked a question. If Sam built one of those, I would like to know what Sam's asking, right? Because it's already been done and I didn't think about it.

So there's, there's a lot of Data we could do better on, I think. But I think there's an add to your dashboard feature. Yeah.

Yeah. I think there's a little feature up there. I dunno if it was in demo.

Yeah. But, but I think Keith's question is how do I know the intelligent questions others are asking? I think we could do a better job.

Yes. Yeah. We, uh, yeah, let's, let's move forward 'cause we're running outta time.

Yeah. Peter, Um, in terms of the data that you're using to feed it, obviously you've got your network data and you've got your docs. Mm-hmm.

Are you also using any external data? Um, so I don't know, like data from the Network alliance on client capabilities? Is any external data being fed in or is it just data held by?

So I, I'll, I'll take a first stab. Uh, and then, um, so what we are doing, what Marvis has actually become now is we are voracious consumers of data from external sources as well, such as teams. And Zoom is external to us, right?

It's, and teams. And Zoom is not just telling us about Teams and Zoom, it's actually telling us about the laptops, you know, CPU and memory, all this kind of stuff. So that's one category of, of people that are giving us data.

We integrate natively with Cradlepoint. And so we get all of that data directly into the dashboard. We integrate natively with cel, uh, we're getting data there.

We're integrating natively with, um, you know, some of the, uh, the, uh, security application providers, the top two major security providers. They both have their digital experience, uh, uh, applications. That data is also natively integrated to Mist.

And now, you know, Slava is gonna present very integrated with MDM solutions. So we basically have created this northbound pipeline that is easy for us to bring in, uh, external data. So we are voracious intentionally.

Like, as an example, one of the next things that we were thinking about is should we integrate with application performance monitoring tools, right? Um, we obviously already integrated with ServiceNow, should we make that bidirectional where ServiceNow tells us something that there was a config change and a firewall that we don't manage and that kind of stuff. So there's, there's a few more things we could do, but we are open to actually consuming a lot of this and making meaningful, I was more thinking of like informational documentation, like standards and things that, because, so for example, if I've got a wifi, um, six network, yeah.

And I say how many of my clients are currently wifi seven capable probably from data on the network? 'cause it only understands wifi six. Well, no, yeah, but it, because you classify devices and you know, there's an iPhone 16 or whatever.

If you knew that in external information, you can answer that question. I think for, I would say that's on the roadmap. I mean, right now we're ingesting data from the network from Zoom, you know, third party partners.

You know, your question is, yes, we will start ingesting data from the internet. That'd be the next step. You know, if you ask me a question, I'll probably go off to the internet to see if there's an answer on the internet.

There's An answer. Okay. Very good.

Okay. Alright. Lemme wrap it up here.

Yeah. Um, Marvis actions, this is our baby. This is this cup of coffee, self-driving component of Marvis.

Um, after I took my self-driving Waymo Uber, I am on the mission now to move us from assisted self-driving to fully self-driving. If you look at Marvin's actions, Marvin's actions has always done a great job on assisted actions, right? We do a great job of helping.

Its, you know, here's an issue, here's a recommendation, here's the evidence for the recommendation. With this Marva actions facelift, you're gonna see us start driving more towards self-driving. This is where Marvis knows how to fix a problem.

Marvis is gonna start asking permission to fix that stuck port, fix that missing vlan, and then ask, would you like me to put this on the trusted self-driving list? So this is the Marvis actions facelift where we are gonna be moving from that assisted driving experience to more of that Waymo Uber fully self-driving and earning the trust of it. Awesome.

So, uh, this has been, uh, ever since Bob came to WLPC two years ago, I think we've been on this thing, uh, uh, crusade. And so, um, so what we are doing now is, is this new facelift for Marvis actions. There is a new self drivable, you know, icon there saying, Hey, these actions, you know, if you give us permission, we could fix them.

We don't even have to just alert you. And so now these are the list of things. Port stuck is an interesting example.

You would think that's, uh, uh, you know, that's something we may not be able to actually self-drive. It is, it is, uh, fascinating. You know, basically think of a thermostat, a video camera, a sensor that has a certain periodicity of traffic.

And we are observing that periodicity and saying, Hmm, you know, this port didn't go down, but certainly the traffic disappeared. And with high efficacy we could actually say, Hmm, this video camera, something needs to happen and we need to bounce this port. And literally we bounce, uh, uh, you know, ports, uh, uh, like this, uh, around the world, uh, I would say north of, uh, 20,000 ethernet ports, rebounds, um, on a daily basis, uh, uh, around our universe.

And so essentially what this eliminates is if somebody go, somebody goes in to check the video footage, you know, five days later they don't have the Epiphone s**t, oh my God, this thing did not actually send video for the last five days. We are remediating. So self-driving can, can, can come to even sort of very legacy use cases like that.

Another example is if we have all the access, uh, VLANs on a switch, but one of them isn't, you know, uh, trunked up to the up on the uplink port, right? Actually we could go change configuration on switching. So the trust part of this is obviously you're going to first, uh, you know, we, we have to earn the trust of network it, because at this point we are starting to touch config, and that's scary for a lot of people.

So, but the moment you get that trust one item at a time, you know, our goal is everything we can, we want to get marvis actions to be self-driving, right? It is, uh, really powerful, right? So, um, with that, what, uh, you know, our fundamental focus as Sujay pointed out at the very top of the session is, look, every vendor is gonna come here and talk about AI and models and all kinds of stuff.

Our claim to fame is we basically are focused on customer outcomes, right? And so, um, you know, for all of the Marvis, uh, uh, you know, things that you've seen today, um, you know, our focus has been around how do we deliver a customer experience enabler? How do we enable a better customer experience?

That's right. Uh, you know, uh, take all of the entire Marvel session, the summary of it for us is happy customers. And, and that's what, you know, across verticals.

Imagine what we do today. We go in and get deployed in a small lawyer office or a dentist office to literally the largest networks in the world. The, uh, the, our current largest network from a wireless perspective, 400,000 access points deployed in production.

Uh, after deploying 400,000 a aps, uh, across their global estate, um, they've chosen to deploy, um, uh, switching with us, right? And, you know, that's why we have a, a small switching section at the end of this, uh, thing. 90,000 switches deployed, right?

It is delivering amazing customer outcomes is what the summary of this, uh, Marvis session is with Chris. Yeah. Actually on the actions.

So your, your, your goal is to add more and more into, into the actions. What's the goal when it comes to, uh, minimizing risk with detection of issues and rollback? So you've taken action and oh my goodness, things have Gone wrong.

Yes. Yeah. Um, what's the plan to detect that and roll back those changes?

So, Uh, Chris, I think the, um, our entire focus from day one has been efficacy, right? Because if we don't get efficacy, right, this is no different than, you know, our grandfather's alerting that we had just five years ago and, and oodles of alerts and, and you know, which one do I even start with? So our and entire focus has been efficacy.

So we, when we take a self-driving action, it's near 99% efficacy. Your question is, but what about that 1%? Right?

Because s**t happens in real world, right? And so for that, you know, we, we actually, we have to institute, I don't think today we have an auto rollback of this, but, uh, we, we, uh, but predominantly anything that is self-driving, we have extreme high efficacy, right? So Bob, I Got a Question on that.

Yes, please, Mike. As a network administrator, how do you know what changes were made? Because generally you want to go through some kind of change control process in case you do need to roll back, right?

Very good question. So actually, uh, marvis actions previously did not have that. And so, um, the very last part of the demo, uh, is we actually ended up, uh, essentially with a, a now a history not on of all the both self-driven or driver assisted actions.

Now there's a complete history and visualization for that. It didn't have, we didn't have that, uh, till this launch. Yeah.

For, for the customers that have tested this for you. Yeah. How, how do they tie this to their change control process?

Like, uh, what I'm thinking through is since you've got ai, can it learn when my maintenance windows are submit a change ticket? Uh, 'cause I I have an organization where the change control person is probably gonna be very scared of this. That's Correct.

That's correct. So actually roel, uh, this port stuck as an example, the only reason we do it is there is such high efficacy that we knew that port is shot. We know there is no video coming in.

This is not a change control. Think of this as a, for that microcosm of an application, which is that video camera, it's a network down situation, right? And so it's not a change control kinda situation.

It's a, Hey, this application is not functioning, go remediate it for me. And that's the only time we are actually doing self-driving. Yeah.

I, I would say more like RRMI would think of this as radio resource management, right? And we started that 20 years ago. It did not trust it, right?

They had had an on off switch and most, you know, like I could do better. We're in the same point with network resource management. So I mean, so your question, if customers don't trust it, they're not gonna turn it on.

So I think this is the transition to network resource management going forward. Now to your question about change, I would tell you looking to see if anything changed statistically before and after a change, that's much easier math than try to do anomaly detection. 'cause I have the event, I can look and see did anything statistically change in the network?

And I think you'll see us go back to rollback, you know, we won't be able to check if we change, if a update did something statistically change in the network that would, would trigger a rollback. I have a question that along the same lines too. 'cause a lot of organizations have zero trust solutions and network access control solutions that are tied into the infrastructure.

So obviously, you know, when we do that, we, we have to have one source of truth of where those, that the change control stuff is happening from. I, what's gonna be the answer for that for the organizations that have that type of integral For, for, for, for, for the most part, every action that we produce on Marvis actions is coming in as a web hook into a ServiceNow or, or some other, uh, their source of truth of, of, of all global actions in the enterprise. So that's where we record all of this, right?

There is they will never miss one of these. And, and now there's a trail of, you know, yeah. And I'm Less concerned about auditing that it happened, but more that a, a lot of the products that integrate with like local network resources, especially on the wired side, can't have, can't have something else making changes to the infrastructure when it's trying to do port changes.

VAM status changes. And I know obviously you guys have access assurance. Will that be able to, I think the answer would be something like, it's like right now we work very closely with ServiceNow, right?

And so ServiceNow becomes that source of truth, you know, where Marvis will send the request to ServiceNow to actually execute Something. Bob j what JJ ISS asking is, um, a port may be intentionally changing profile because of network access, policy change. And so how do, how do we not have a race condition between us trying to self-drive?

And I think my point is we change, we, we don't touch the port, we send the message to the source of the change like ServiceNow because you know, a customer may trust ServiceNow to make the workflow change, right? They may want all changes in network go through ServiceNow. And I think that's what you're thinking.

Yeah. Not, not quite. But I guess my, my question is will, will the access assurance solution be able to be a, will it fit into this if, if somebody needs to be doing that type of port level control?

That's, yeah, And I would say, you know, you to your point of, you know, are there multiple sources of truth? I mean, no, you don't. You want one, right?

And so depending on where your source of truth is, you know, if it's missed, most of the time it is, is missed, especially if the switches are managed by mist. Um, and that's where, you know, this port stock would apply. Uh, and so if, if you choose to turn on the, you know, the self-driving aspect of it, then mist is your source of truth.

And, and with, um, you know, some of the, uh, NAC solutions that, you know, want full control over network state, um, they're already talking to our APIs, get network state and uh, okay, they're already talking to the APIs to get network state pulling that back in. And so there's already a, a sync that's happening because, you know, configuration is already coming from missed. Okay.

Awesome.