Monmayuri Ray – MLOps: DevOps for Machine Learning

Hello and welcome back to GitLab, How To's, Tips & Tricks here at GitLab commit twenty twenty where our theme is You belong here. Our next speaker is Monmayuri Ray and or as we call her, Mon and Mon is also a GitLab team member that works on the team at GitLab. But I actually didn't know Mon before we started working on GitLab Commit together.

She is a solutions architect and the team has gotten so big we have over thirteen hundred employees now that I just don't know all of the other GitLab team members. But when Mon a bit of this talk on MLops, I thought this is really exciting. Here's somebody who's an expert in the field and has a lot of knowledge to share.

And on a new topic. I know that as I've worked with Mon and listen to her, I've learned a lot about MLops and I'm excited for you to learn a lot, too. Let's check it out.

Hi, everyone, welcome to the session on MLops: Devops for Machine Learning. I'm Monmayuri Ray. I'm one of the solution architects with GitLab in Sydney, specializing in MLops as well as Devops.

I have put in my GitLab handle as well, as well as my link to all the repos on MLops, just so if anyone is interested. A little bit background on me. I am classically a data scientist who was working on applied machine learning and natural language processing within health care and human capital AI apps.

And and from then I sort of transitioned more into the topic today, which is on AI ops and MLops. So for today, for the next twenty five minutes, we will be going through what is a little bit of a history. Where did this all start?

Why is this topic relevant today? The value of MLops Devops MLops. Some basics, what is a model and what then is MLops?

And then the principles and processes and simplifying that with GitLab. Where does this all start and why is this topic relevant today? It's a very interesting story and I don't think a lot of people actually know that.

But AI, the journey of AI started with these mathematicians all on the left in Dartmouth College for a summer program where they decided that in three months they could actually solve for automatic computers. And how can a computer be programed to use a language? So that's sort of where it all started.

No idea that that would actually still be a very relevant topic. I. even now.

So where are we today? In 2018, there was this conference on basically how AI can be used in every part of operations, in every domain. What what is the next chapter of AI.

And Professor Daniel Kahneman, who is the picture on the right, he's he's a AI psychologist and a Nobel Prize winner. And, and then there was actually in the same conference the chief AI adviser to under Obama, as well as the chief AI strategist for Google and Microsoft. I.

And Professor Kahneman ends the conference with this tale, which is my favorite AI tale and very, very relevant to MLops. So the tale goes like this: A well known novelist wrote me some time ago that he's planning a novel. The novel is about a love triangle between two humans and a robot.

And what he wanted to know is how would the robot be different from the individuals? And Professor Kahneman proposed three main differences. First one, obvious robot will be much better at statistical reasoning.

The second, the robot would have much higher emotional intelligence, very different to how we actually still see. We as humans, we think we have the higher emotional intelligence. And that Professor Kahneman, he puts this other narrative of the robots having the high emotional intelligence.

And third one is that the robots would be wiser. Wisdom is breadth. Wisdom is not having a narrow view.

That's the sense of wisdom. It's broad framing. And a robot will be endowed with the broad framing.

And I really do not see why, when it has learned enough, it will not be wiser than we people, because we as humans don't have broad framing. We narrow thinking, we noisy thinkers, and it's very, very easy to improve upon us. This is a very, very bold statement.

I. Devops, AI MLops, all of that to complement us humans as the noisy thinkers. And have these little little strategies and tools and frameworks to help with the broad framing and and leverage AI.

I. and it's it's almost just everywhere. There's some news where it's someone doing machine learning, Siri, Alexa, all of that.

So then came this era of The Economics, and they look at things very differently. So for them, it's just all about the value. So what does this actually mean to us when we do economics 101, it's all about the demand supply.

A certain price drop in something means more consumption of that. When one consumes more of that, it actually substitutes something else. And then the compliments of the more consumption one is also gets higher and people buy more of that.

So, for example, when we say tea, if there is a certain price drop and people buy more of tea, it is substituted by coffee and people end up buying also more of sugar and milk, which is a compliment of the tea. Now what does that actually mean to AI, Devops, and all this technology? So when we look at mobile phones, what we see is it's actually the drop in price of communication.

When we look at AI, it is the drop in price of machine prediction where it is supposed to be substituted by human prediction and the complement is data and human judgment. For Devops, it's the price drop in transaction cost in building technologies to create cross-functional teams and a little bit similar to the complement is automation and human judgment. When we look at MLops, we see there is a price drop in automated decision making for machine predictions, which is substituted by the repetitive decision making and and human prediction, and it's complemented by human judgment, data automation.

So just to summarize all of that into one sentence: what is MLops? A movement that gets us closer and closer every day to cheap, almost automated decision making by complementing precise judgment through machine predictions? So giving a rough understanding of a little bit of a history, economics, fundamentally, theoretically, what is MLops now?

We will first go to the basics. Now basics being what is a model. We are going to build a model together about an experiment that we I personally was part of this for a hospital in Sydney where the experiment goes like this, we're going to predict high risk heart patients within 24 hours of entering the emergency department.

Now, to do that for any model, you need data. So in general, classically, without an AI model, the way it is done now is through through a couple of rules that a human doctor or clinical nurse would ask the questions to the patients based on three four rules. Make a judgment on that.

Now when we are doing a model because a model can handle much more than three or four different rules, we ended up taking data for 80 different variables from historic medicine, understanding to preexisting conditions. Any sort of test results, obviously, the clinical notes, any in-house medication and additional change, any change of the notes when you are in that hospital through that relevance in that twenty four hours. Now, that itself is already a lot more than a human can actually process without a machine.

So all of it it kind of is taken and then it goes into this cycle where this is basically the fundamentals of the model, where there is a training, some data set which takes all these sort of variables that we went through, builds features and takes the predicted variable, which is a classification of high, medium low of high risk heart patients historically already classified into a training data, sets into a machine learning algorithm, into the model. And it sort of goes into the operations of running that algorithm, into the prediction. The input data is all about any new patient that actually enters the hospital.

And we we basically fill the missing variable of the classification of high heart risk patient, medium heart. risk patient, a low risk heart patient. That whole operation then is then evaluated based on model accuracy, also based on actually if a doctor would actually classify a patient like that looking into the business rules as well.

It's a combination of all of it together and then back into retraining the model with every input data and taking that feedback and getting and dialing up the accuracy of the model so that. Is a model, basically, is that input that leads to the output of prediction, this whole operation is what is anomalous all about. So we looked at kind of the features.

This is one relevant feature, which is actually a real value feature of an ED notes, where this is how ED actually comes into the database. Very noisy, as you can see. And and as you can see, it's a human written with a lot of different mistakes in an emergency department.

People are writing really fast. So what we go through is actually processing through natural language process to clean up and take those relevant features that we actually need to input in the model. So the NLP process is the relevant feature that goes into the model and then the model then detects it to be the feature that is of the highest importance is severe tachycardia, and then it classifies it to be a high risk patient.

So that is what in general a model processes. Now, when we come to actually the whole process of the operations, what we see is the patient enters the emergency department. There is tests run through the clinician takes the notes, and that is sent to a database an EMR database that is linked to to an instance where the machine learning model is built.

A lot of times it could be a data scientist's laptop or a cloud vendor or wherever that instance is setting. This is sort of the simple process from a patient all the way to a data scientist to building the model, to deployment, to production. What we see here in that MLops process, that there is a problem, a business understanding what is relevant is we need the right relevant data.

And also with that, we need the right element pipelines to extract that data in that in real time, load it and transform it. Then we need a model that goes into building those features from from the data into feature engineering to training the model to evaluating. And then the final round is the deployment where we are scoring the model are looking at the performance, whether it's creating with new data or and storing it and monitoring and all the whole continuous feedback that is that whole MLops process with people as well as tools.

Now, when you look at that whole process, we do see a couple of different parts that comes out. Now, the first one is the once the data is loaded, the data scientist has to do a lot of different experiments, a lot of different iterations to really, really get to understand the data and build the features. Most of the time, each data scientists have their own set of tools, own set of platforms where they they feel comfortable doing doing the experiments.

Now, that is fundamentally quite different to traditional software development in when it's a rule based Java engine where it's traditionally just Devops. That whole process of experimentation is quite, quite long. So that first is is is one of the reasons why MLops or this whole thing is is actually born.

the second the serving models becomes extremely, extremely harder. So the whole experimentation is done in one instance. But when it's the model is served post-production.

There are lots of differences and nothing. And that and that can also to serve. That gets much more complicated specifically for NLP models, where we see there's lots of clinical notes and things like that touchpad with all of that is because it's all of that very, very siloed laptop to program to cloud all of that.

There is no traceability. There's no orchestration of putting it together. And currently, we are in this very interesting stage of people moving from AI model proof of concept to full production.

And and so there is this need of deploying more and more models. So as the need grows, the data is growing, the infrastructure needs to grow. At the same time, the cohorts of different sort of people building a model is also growing.

So it's not just a data scientist, it's the doctors, it's the clinicians or hold part of building that model, as well as software engineers who actually deploy the model. Even with data scientists, it's sometimes differentiated between supervised learning, unsupervised learning and NLP models, hat was sort of that and NLP process that we looked at. So, again, a wide variety of people and there needs orchestration and harmony.

This is actually a list of principles from a responsible machine learning open source framework, which goes through eight different principles. For MLops number one, human augmentation. Number three, explainability.

Number four, reproducibility. And number seven, trust by privacy is is the framework for those MLops principles. So human augmentation goes in, automating a lot of these manual tasks, explained ability is having the logs of really, really understanding the black box of the models and justifying it.

Reproducability is being able to actually reproduce the same model in in a development experiment or production environment and be able to share it seamlessly. And trust by privacy is making sure you communicate, are able to communicate the whole process and have credibility and visibility through that whole process of building that model directly or indirectly. And those kind of goes through those principles of MLops leading to this whole triple infinite loop of ML Devops where the Devops part of the ML is a little bit different to a traditional Devops where it actually looks into also the algorithm training and testing upon the pipelines of the model, as well as the integration and the model inference and model management part of it as well.

So all through all, what we kind of saw was the need was to be able to have a orchestration and integration of all of these different phases and steps that goes into building a model. One of the key differences of which hasn't been used classically traditionally is using a CI service to be able to orchestrate through it. So and the other part is being able to look into containerizing in every process of the pipeline.

So sameway this shows that there was a CI service with a source code container registry integrated to the model training, serving environment that is integrated to the object storage. So every change in the source code triggers the right part of the pipeline to model training and registers to the container. Now, this part can be easily, easily simplified with GitLab CI CD, where it's integrated to the source code into the Web IDE of the data scientist, whether you work in your laptop or you using the GitLab Web IDE or PyCharm or or your favorite node book into the container registry, the model training can be easily integrated into any pod, whether you're doing it on the cloud or on Kubernetes or or you're actually using the GitLab runner to train the model and that can be totally integrated to the object storage as well as the frontend admin.

So the GitLab can trigger the pipeline and continuously integrate into the source from bulding to validating and deploying and have explainability, transparency and visibility for the data scientists, Devops engineers, for data engineers, as well as if you want the doctors and nurses to actually have a part in that feature engineering or any business domain. So we looked into the CI CD pipeline and how we actually orchestrate all of that. I.

YAML file for a machine learning model is a little bit different to a traditional. Java application or or any of the other non-ML CI YAML. So it goes through a seven steps.

This is a very generic framework where it goes to first to prepare the right environment for the data scientist. So I'm for this part, it's just a python environment. But it could be R or could be Spock.

Then you create the workspace for the machine learning service. You submit the training job on the local python environment or anywhere else register the model to the workspace, compared the performance of different models and select the best. That's still a part of that experimentation for a data scientist where you don't even just try one model.

You do try different sorts of algorithms to actually see how you how accurately you're able to predict. So that goes into giving them a CI pipeline that actually helps with comparing those different models. So one could be 80 percent accurate, one could be 70 percent.

But having that part of that YAML file. And then you score the model, so you create a Docker image for scoring Web services and finally you publish those artifacts to release pipeline. These go into that step.

This is just an example of that pipeline in GitLab. It's even simpler than what we saw where we go into validating the environment, building the model, deploying it, testing it, and then cleaning up the inference as well as the training workspaces for the model. On the every part of that CI YAML file or the pipeline, again, just like any other services within Devops with GitLab, everything has its log and can be then traced and justified as well as explain.

So for example, if there is one model to another model, we are comparing it. These logs can help justify why one model would actually give that 80 percent more. It is more 80 percent accurate versus maybe the other being 90 percent accurate.

Further, where it can be used is the visibility and transparency. So the GitLab API is a is a rest API, which helps with going into further deep dive on a let's say you're the predictive analytics, the insights, dashboards for a Devops lifecycle. For the kind of metrics that a data scientist or an organization would want to see is what kind of model is actually working.

What kind of model takes longer to train for a certain kind of data? Is it eight days into three days into two days? These sort of insights, again, is still a black box.

And through this envelope cycle where we look into visibility and transparency, the GitLab rest API can easily be extracted to help with these sort of insights that gets the organization to know which models to use when for the time, the bugs, the scanning of all the containers for the models and so on and so forth. So this gives a little bit of a flavor of the MLops with GitLab. And this slide is just a rough framework of the process that actually runs quite similar to to a Devops engineer.

Just with a little bit difference with how you actually write a code as a data scientist. So all of the code is written in a way that it can be packaged as a container. So the source code is divided into training as well as the inference phases of the model deployed as a single package that gets pushed into a branch.

Any sort of changes similar to any pull a much merge request can be can be changed, which again triggers for an approval to QA it and then goes into the new training data sets with a CI pipeline triggered for it to run the new new model into the kube flow or Kubernetes or GitLab runner or wherever you would feel comfortable running the model. The final steps of that pipeline is generating new model artifacts. The evaluation metricw, qnd then getting that automatically into the model reference json files and getting that approved.

The approval, then once that is done, in the experimental instances, it then goes into the deployment stage of redeploying and the inference application to the latest trained version of the model. I. applications through this MLops version makes that operation of the decision making quite seamless, cheap as well as automated too.

And this we're going to go back a little bit actually to the beginning where we started. So we started off with with a very futuristic high psychologists and how he looks into the robot's being as equal to humans in certain frame. I.

has been proven. I. in in just our our daily business decisions, or whether it's because of the operations, whether it's because of not getting production faster with its data, there is some sort of a dissonance to actually use that those sort of applications.

Now, when we think about it, we go back to a little bit of that original process that we saw with the clinicians and predicting the high risk heart patients. I'll probably repeat that process again. But so the way the process now is the patient enters the department, runs the tests, clinician takes notes.

Data gets sent into the database, the notes gets extracted and the high risk heart patients predicted. Now what if there comes a time when the data gets richer and richer and richer and the moral accuracy for the patient also dials up the accuracy more and more and more. So it goes from, let's say, 60 percent accurate to 80 to 90 and ninety five and so on and so forth.

What actually then happens? We can then look into this way, reverse the story, patient enters emergency department runs through all test machine takes all the personalized relevant knows where in that's in the slide. We looked at the NLP process already does.

The clinician then doesn't spend that time actually figuring out writing those noisy notes that we saw, but actually looking into making the real decisions, automate decisions to looking into the notes that get sent into the EMR. Not just that it helps with the current strategy, but with using of MLops, making the AI model richer, making the data more accurate helps with actually making this new strategy. To end it, addressing that dissonance is all about that pieces of time.

So the more we are able to dial up the accuracy of operationalizing, machine learning, operationalizing and models, the data collection, the whole pipelines to dial up the accuracy of the model from thirty to eighty to ninety five, it not only helps with our current strategy, but also we rethink about new ways and new ways to execute those operations and needs of humans. To make it more clear, and I always speak about it, is actually the Amazon recommendation engine. So right now I would say that every time I buy a shoe, it recommends me the right one 20-30 percent of the time.

Now they collect the data, the operations get better and and the accuracy actually gets to ninety five percent. And maybe one day Amazon decides that they are not only they're not going to wait till I actually buy the shoe, they're going to actually ship me the shoe. That's theoretical.

But that's again the beauty of automated decision making by enabling MLops. And with that, thank you for your time. I hope you really enjoy it.