Manage Edge AI Using ZEDEDA Kubernetes Service

Uh, welcome everyone, uh, to this session. Um, uh, my name is Hariman and I'm a distribution engineer here at za. And in today's session, we are going to discuss, um, how to manage HGI deployments, uh, using s given ates services, and walk you through the capabilities of how we can manage models and other components, uh, in an AI solution.

With that, let's get started. First of all, why do we need to, uh, to, uh, discuss the topic of AI at the edge? So, AI at the edge is getting, uh, more and more important with, uh, uh, with the, the reason is that a lot of iot devices getting connected, and then a lot of in, uh, data is getting, um, data is getting generated from these devices and the devices which are offline, uh, in the manufacturing, they're all getting smarter, they're getting connected to the internet.

And, um, a lot of, um, uh, uh, inputs are, um, basically coming from these, uh, devices. And then, uh, we, uh, more and more, uh, business requirements are coming around, uh, how we can analyze this, uh, this, uh, uh, data sets and then derive insights, um, based on the inputs. And then how do we take actions, um, based on the, uh, uh, the data sets and how we quickly do, uh, corrective actions at the site.

And, um, uh, why can't we do it at the, uh, at, in the cloud? How, what should we do it in at the edge? And the reason is, uh, that there are multiple factors here, right?

One is that the bandwidth is simply not going to be sufficient, right? The, if you look at the scale at which, uh, these, uh, these sensors are pumping, uh, information, it is simply not going to, uh, on the internet, bandwidth is not going to be sufficient to send everything to the cloud, and there's a huge cost associated with that as well. And then there's a latency issue.

You can't send these, um, it's not practical to send these data sets to the cloud and then have the inference, uh, uh, uh, in the cloud, and then send the inferences and back to the edge. That's too, uh, the latency will be too high in certain environments, which is again, uh, concerning some of the I deployments. And then, uh, the data, uh, needs to remain private, uh, in some of the edge, uh, uh, locations because, because of the, um, legal or intellectual property concerns, you can't send all these data sets to, uh, the cloud for inference.

So then how do we address these issues, right? The, uh, the solution is to bring AI to the edge. And, uh, we basically, uh, instead of sending the data sets to the cloud and having AI in the cloud, you bring AI close to the source of the information.

And that's how, uh, you bring AI to the edge. And then when you, uh, bring AI to the edge, uh, it has brings its own challenges, right? Each age, uh, deployment scenario is different, and the underlying hardware platforms are different with different architectures and then different GPU types vendors.

And then each of the device, uh, the device ranges, uh, from, uh, very low end to, uh, from raspberry pi kind of devices to Jetsons, to the high-end servers with, uh, high-end GPUs. So how do you manage the, uh, model, uh, and then, uh, the other components that are built around this model at scale, across these distributor locations? Mo most important thing is the internet connectivity outside.

So the, the edge AI needs to, uh, be, uh, autonomous, right? Basically, even if it's not connected to the internet, we should be able to, uh, run the business at the edge. The AI pipeline should keep running.

So how do you manage these things? And then, uh, how do you bring the, how do you update the models at scale? And how do you bring eventual consistency where all the sites periodically, uh, uh, check back whenever they're online and bring their model up, uh, updates, uh, and then, uh, keep the antilier up.

So, so the, uh, how do we solve these challenges? So that is where the media, uh, Kubernetes services comes into the picture, right? So, uh, why do we need even, uh, Kubernetes for a, i if you look at this, right, AJ is not about just, uh, developing a model.

If you look at this picture, uh, of a Jeffson news case, uh, there are cameras connected. And then if you see the components involved in, in bringing an a g solution, you see many, many components coming together apart from the model. And there is an input processing, uh, stage.

And then, um, there is an inference stage, and then there's a post-processing stage. And then there, there are monitoring, uh, agents who, uh, uh, keep, uh, a check on the performance of the model. And then there is the API gateway, uh, which basically, uh, routes the APIs to different components.

And you need to package these, uh, components in, in a single logical unit and, uh, be able to upgrade the individual components. And then imagine the 10,000 devices, 20,000 devices running these, uh, components. So how do you manage them at scale?

And, uh, since the a JA deployments are just much more than model and then a lot of components around it, the convenient way to package them, uh, uh, is to use a Kubernetes. And that is where Kubernetes becomes relevant for managing a J. And why, uh, Kubernetes are not something else, because complex workload, uh, we see Kubernetes, a lot of, uh, um, uh, aspects of Kubernetes, uh, are basically helping, in this case, for example, helm Charts can be used as a packaging format where you can bring, um, sidecar to help the inference engines to, uh, uh, having a business logic, uh, in a separate component, and then having connectivity in terms of deployments and services and having a scalable approach.

Uh, that is a big plus that comes from Kubernetes, and Kubernetes are getting lighter. And with K threes, you can basically run Kubernetes on the relo and devices, which makes it suitable for the edge, and it's getting better and better at the edge. Uh, the Kubernetes ecosystem also brings in the CACD, uh, uh, with the, that is very mature in the cloud.

And then, uh, uh, and the technology, uh, can be brought to the edge or can be extended to the edge with the modifications. And same, um, uh, the GI Tops based ci cd pipeline can be, uh, leveraged for updating the models at scale or updating the different components at scale. And then, uh, Kubernetes is also good, uh, with its custom operators, we can extend the scheduling mechanisms.

You can bring multiple edge devices together right now, the form of a cluster, and then, uh, you can schedule the inference, uh, workload on the, on the device where the GP is available, and then, uh, schedule other components and other nodes. So Kubernetes is very rich in terms of, uh, giving those capabilities in terms of resource management and scheduling capabilities. And then you have Nvidia, uh, uh, Nvidia ecosystem is also very rich in Kubernetes, along with Intel and other players in the space.

Basically, uh, with Kubernetes, uh, this, the, the ecosystem is getting standardized, uh, in terms of operators and how we can interface with the underlying content runtime and, uh, integrate the, uh, and basic, the accelerate the AI pipeline using underlying, uh, accelerator, uh, hardware. Um, and then of course, the HA comes with Kubernetes, which is, and that is where ZA has a Kubernetes service, uh, comes into the picture. So ZA offers, uh, first class support for Kubernetes, uh, uh, workloads at the edge.

It helps us to, it, it helps us from right from the beginning of onboarding a device to, uh, the, um, to Z cloud, to the controller, to all the way to form, um, uh, to group a number of devices into a cluster and provisioning a cluster at the edge in a zero touch, uh, fashion. Uh, and then, uh, once the cluster is up and, uh, uh, and onboarded with zaia platform, you can deploy Kubernetes workloads, and it comes with its own Kubernetes marketplace where you can upload or you have been point to a third party, or you can bring your own HE chart repository and then, uh, and be able to directly deploy these helm charts, um, on the, uh, edge devices or, uh, or the edge clusters, uh, or set of edge clusters in, in a, in an automated fashion. So, uh, so one of the use case mean, even though we have many use cases, uh, on top of, uh, HQ es in this particular session, we are going to take one example of, uh, let's say an H AI use case, and then see how we can manage that using the data.

Kubernetes. Uh, let's a session, uh, platform. So in this particular demo, we are going to take, um, a car, uh, an object classification example.

Uh, specifically we are going to take, uh, let's say consider, uh, a use case at the edge. It could be a, a car wash location, or it could be a car rental place where you want to, uh, detect, uh, which car, which model, uh, make the car and the car belongs to. And you might want to do it locally, uh, to take some actions, like imagine a car wash, right?

You want, might want to customize, uh, the, the amount of liquids that you want to use, or you want to charge based on the type of the car. Let's say it could be a, uh, a smaller amount for a smaller car, and let's say the considered minivan kind of category, then you might want to charge a little bit higher based on the, the usage, uh, um, size and dimension of the car, and then consider rental use case, right? We might want to charge based on, uh, uh, on a type of the car that is getting, uh, rented, or you might want to take some, some, uh, uh, some pricing model or some service, uh, decision based on the car, um, that is, uh, they're seen in the picture.

Um, so if you want to automate these things, obviously you don't wanna send all this information to the cloud and gi, it's much more economical and, uh, more, uh, uh, it makes me sense to do it, uh, close to the, uh, the dealership. And that's where, uh, the edge, uh, AI comes in. So here are the models.

Uh, what we have tried trying to show in this use case is that, uh, to come up with such a use case where you want to detect what core, uh, uh, type it is. Uh, there are several components that need to come into the picture, as you can see here. One is the, uh, the inference server, right?

Uh, this particular example, you're going to use an Intel platform. So we are going to use a open window, uh, model server, and, but it could be different for different hardware platforms. So that is where the, uh, orchestration comes into the picture.

You can customize these things based on the type of the deployment, uh, and it, I'll see on how we can actually, uh, change some of these components on the fly. And then the models and then, uh, helper containers to help, uh, pull the models, uh, from the external thing. One, uh, aspect that does the reader, uh, shine, uh, where the shines is, uh, you don't need to, uh, basically keep the models in a central location where Zaida or Z Cloud or Z Z's orchestration platform, uh, uh, would be able to see these models, right?

It can still operate without having access to these models. For example, you can actually have these models in, in a local on-premise network where only the edge cluster can access. It could be in private network, as long as the edge cluster can access and pull these models, uh, it can still be orchestrated from that cloud.

So in, in this particular example, we are just demonstrating that by having just a S3 compatible storage. It could be anywhere, it could be in the cloud, or it could be on-prem. But, uh, with suitable configuration in the sidecar, you can actually fetch these, uh, models, uh, without actually pulling, uh, showing or exposing them to set cloud, right?

Um, yeah. So let's start with the demo. Uh, before that, I just want to, uh, show you a high level overview of the components involved.

Uh, you can, let's focus on the, the middle portion where the, the components are, are given it this, uh, resources. So you have, um, an open weo, uh, inference server here. And then, uh, there is a, a side card which pulls the model, and then storage stages it for inference server, inference server exposes, APIs.

And then we have the upper box, which is a demo app. And that we are, we have, uh, written to show how we can actually send inference request over the, uh, network, local network or virtual network to the inference. And then, uh, we, we also opened a port here so that we can access the, the demo app to see what is happening inside the, the edge cluster.

And then the site card switches it from the enable, uh, endpoint where the models are stored here, and then the container images themselves can be in a private OC pository, or could be in a, in a repository somewhere on the cloud. With that, let's, uh, go to the, um, the demo. What we have here is, uh, the edge clusters, and, uh, this is my cluster that I'm going to show you on the demo and can see the edge cluster has already, uh, some of the components are by default, uh, started to start to talk to, uh, the leaders given at this orchestration.

And then if you go to the device, this is the Edge node vendor search for the My Edge note, I want to show you how. So this is the Edge note that is, uh, um, that is basically hosting the cluster and this, the Edge node onboarding processes of the standards. It does, uh, uh, onboarding process where this runs Eve, and then you basically call configure these, uh, h device to, uh, talk to Z Cloud.

Once the uh, device is visible in Edge Cloud, you can form the cluster out of, uh, this edge node. In this demo, we are having one node edge cluster, and, uh, you, the process is almost same. So once you have two, uh, uh, edge nodes, you can basically select the list of edge nodes and form the cluster, and you can see all the standard, uh, metrics coming out from the edge node.

And that, uh, edge node is basically added to this AGI cluster. And here on its left side, you also have Edge cunet is marketplace. And in the marketplace we have, um, the Edge AI core classification as one of the items.

This is basically helm chart that, uh, has different components here and basically actually talked about, uh, this earlier. This, um, has a minimal endpoint to store the models, and it has a server po. This has the inference server or server and help site helper for that.

The client part is the demo app that we, uh, we'll show we'll use for demonstration of the whole solution. And then the, uh, the, the shared word shared storage is where the model will be staged for the important server. Um, so I just wanna show you, uh, we have already deployed this for this demonstration.

So I will go to the install applications, and so AJ demo, uh, install. So after you install this, uh, l chart on the, uh, on the, uh, on this cluster, so you basically see the status, any status of different components here, then the, and the status, and then where its cluster, uh, this installation is running off, and then you basically, uh, go to, uh, you cannot also go to the cluster and get a detailed view of, uh, the deployments. Yeah, we are connecting to, uh, to get more details from the cluster in terms of what are the pods and deployments like, yeah, you can go to workloads or the deployments, and you can see there's a side car and there's a classification client, and then a classification server.

We're running here. Um, we are going to connect to the, uh, client. So, uh, so this is the client.

So we are basically connected to the client's, uh, model. We're just running on the edge cluster. So what we are going to do is, uh, we are going to do some test inferences.

And currently there is, uh, hosting two models. Uh, one is based on, um, uh, one is resnet RESNET 50, which is a CNN for object classification. The other one is a modified version of, uh, uh, uh, resnet with additional, uh, transformers, uh, added in.

So it's a model from, uh, developed by meta, uh, back in 2022. Uh, and, uh, this is basically, uh, a higher level model. Basically it's more number of parameters.

And then this is, uh, a faster model with this number of parameters. We are going to compare the performance and then, um, and then the, and the accuracy of the model between these two with some testing measures. So let's get started with the test image.

So I am, uh, passing a Honda ODC minivan photo and to see, and then I'm going to select car predictor and then submit it for inference and the right side. Yeah, so correctly created that it is, uh, Honda ODC, minivan 2 0 0 7 model and the inference. And it also shows some numbers, like, uh, how much time it took for this, which is all running on CPU.

So, and it took close to a second to, uh, complete inference and, uh, what model was used. And you can also see the stages of the, the inference pipeline. Basically, we uploaded the image, it went through, uh, the pre-processing stage where it changed the, uh, dimensions of the model according to the format expected by the, uh, with the model and then other parameters.

And that, uh, were reused to convert the model to the, basically this is the pre-processing stage before we even send the input, uh, as feature set to the model. And then the model used, and then what ran time was used for this particular, uh, model. And the backend is CPU in this case, if it is a, uh, if it's a Jetson or if this is a device with GPU, then use a GPU to show up our GPU here.

And then there's, yeah, so that, uh, is a number that, uh, I mean from, yeah, yeah, right, right, right. So that confidence is actually not a percentage showing this percentage. It's a confidence score, actually it's called confidence interval.

So that's how we can interpret that number. And then is a post processing stage, uh, uh, where we basically take the top, uh, cast the top classifi, uh, confidence value or confidence interval, and then, uh, uh, go for that as the result. And, uh, we are going to change the, uh, type now to, to another model, which is basically there's a transformer based model, which is, uh, little bit heavier than the current one that we selected.

And, uh, this time it is, again, it is predicting correctly, but it is a heavier model. So it is expected to take more or same time, but look, uh, the confidence interval or the, uh, that's supposed to be higher because it is trained with lot of lots and layers and other enhancements, even though it uses a resonate backbone that other optimizations done for that. Again, you can see the processing high.

If you want to, you, uh, get more information about this model, uh, you can, uh, go to the model info here. It shows if you want to do, uh, develop your own business logic, uh, you just a demo app, right? So if you want to build a business logic around it, you can basically test the inferences and then using these commands and then, uh, get, uh, basically get more familiarity with the model, uh, server, and then develop your app logic around it.

That's right, the, yeah, so this is the client, uh, uh, uh, client software that, uh, that is running on the edge cluster. This is for the, this is going to be the starter kit or, uh, uh, to get started. So this will be the default, uh, client application that will be deployed.

Uh, but if you want to change this to a, a custom container, which is specific for a customer's use case, that would replace the software, basically. Okay. And the, and is that for, you know, image processing and you have another one for other types of, uh, other types of models or, or, um, so what do you, what?

Yeah, I'm just wondering if you have sort of a variety of these client software packages that can then be modified or if it's, So it's basically hel chart that is, uh, we can write it in whatever we, uh, we want here. We can even break this client software into two, uh, deployments right now for the demonstration purposes, we have kept this, uh, this app demo happen. I, I think we've gotten o off topic on that question guy, because, um, the, the, this helm chart and all the code will be written by the application developers.

It's not, this wouldn't come from Kubernetes, uh, from Zita, From Z Yeah, so we just for demonstration purposes, but the customers can bring their own health chart and deploy. But this is just one, uh, just to show that this is one way of packaging the a g solution, right? The model.

So, uh, uh, so that's how I had actually, uh, I just wanted, uh, to point out, uh, uh, the, to this, uh, basically, uh, this helm, uh, example is, uh, again, can go the edge field days there either, uh, examples, uh, repository, it's open and you can go the edge field day, and there's a edge, uh, AI card classification folder where all the code and health jobs are, uh, published. Can use that and then, uh, modify it and then play with it, and then let us know, uh, it can help.