DataOps: Streamlining Data and Analytics Pipelines through DevOps Principles – Techstrong Con 2023

Hello everyone. My name is Sanjeev Mohan. I am an analyst with sanchmore and this is the first time I have the honor to participate in Tech strong con i, i as a background.

I am primarily responsible for researching data and analytics topics. So when Alan shiml asked me to come speak at this conference, my first question was but most of your viewers are infrastructure and applications and I cover data and analytics. So what would be a good topic for me to talk about So after talking to him for a few minutes I decided that I want to talk to you about one of the most important trending Topics in this year 2023.

It's called Data Ops data off is very simply a way of taking the all the good. If you've learned from devops principles and deployment on the software engineering site and applying it to the data processes and data outcomes. So that is my topic and the agenda for today will consist of three topics.

So we will talk about About the current state challenges. Why do we why in why are we even talking about data Ops then we will look at okay. What is it?

So let's get a little bit deeper into what is data is why did it come into existence? What are the benefits of of doing data Ops and then finally we will end with looking at some components of what makes a good data Ops tune a practice. So with that I'm a jump straight into what are some current state challenges.

So I want to talk about Technical and the challenges that then translate into some business repercussions if you may so all the technical side the number of challenges that we deal with today to be honest. Now that we are 15 years into this whole Cloud migration people. Most of my clients are heavily for the last many years into digital transformation modernization of their data and analytics practice.

But the promises that they were given have not been met. We in fact if I may we've taken some of the habits from our own premises where we highly constrained and we moved it into the cloud. So what are the problems that we are seeing first?

We have very high complexity we have now gone from having only a few known data sources and Transformations with a few known users into hundreds of SAS tools that we now need to integrate. The number of users of data has grown exponentially and so has the number of Transformations and complexity use cases that we need. So it's no longer that all my users are analyzed who will write SQL statements or run a report or dashboard.

I don't even know who my users will be in future. Maybe they'll download the data. Maybe they'll use apis rest API graphql, maybe that data scientist and they want to access the data through the pi spark notebooks.

So as you can see we have more ways of using the data and that and more complexity. Second problem. We still have not solved our data quality problems.

We've been talking about data quality for decades now. So if you if you are following the news, you know charge gdpt came out and then Google responded quite anxiously with Bard and and when they did a live demo it showed wrong results. Well, it was nothing wrong with the large language model.

The problem was it was trained on data that had the wrong result so garbage in garbage up. Other problem is unreliable Pipelines. It's now very common for large Banks and large organizations to run thousands of pipelines a night.

So they've got data coming from different branches from retail stores getting aggregated and then getting fed into a dashboard one pipeline breaks and it's very easy to miss now. The CFO has a wrong numbers on his or her desktop. How do you know where that that pipeline broke and can you predict it?

Can you stop it before it becomes a problem that lack of transparency is yet another problem? And then overall we're developer experience developers there. They have such a coveted role in my opinion these days because because every company is a software company.

In fact now we say every company is a data company. Yeah. What's Walmart?

It's basically sitting on massive amount of data. That is analyzing. But if it takes so long to produce results, and as you are deep into producing results the CF or knocks on your door and says, sorry fire fighting mode go find out where this problem is and go fix it.

So so support developer experience. What is the impact on the business? So let's look at that.

First of all long time to Value the in fact, it's improved tremendously. I remember when I started my career, we would take a year two years to create a data warehouse dinner Mart and by that time the business requirements would have changed and the businesses. Thank you.

Very much. I am going back to Microsoft Excel. That used to be the red, but even today we haven't really moved to a space where we are doing multiple releases a year.

Sorry multiple releases a week so I can keep up with the changes in. Business requirements, of course the businesses don't trust data that they get they don't know the dashboard has right numbers privacy concerns. How do I know?

I mean you the numbers are the data I'm seeing is Deering to gdpr usability is still a problem. We still use our cake tools and cost is and sustainable. So common question I get is what does a CFO?

Do you have an answer? What does a supply chain analyst? Do you have an answer?

What is a data engineer? Do let me see I munch data and if I transform data to produce reports and dashboards not good enough. So we don't have a good way to explain our roles.

And why does it cause X millions of dollars to produce these? Report, why is it so complex? So what is going on?

Is that data? Is is a much different Beast than infrastructure and applications. Now, you'll see on the left hand side.

I have stacked up from Bottoms Up layers and layers of how this data gets transformed and consumed and different tools that are used. So we have to worry about servers the entire infrastructure security network or all of that then then we start worrying about where am I going to store this data huge huge arguments Brewing up in the data space. Should I use a cloud data warehouse or should I use a lake house or dinner Lake also today is morphed into lake house.

We also have arguments on what is Big Data can I not persist into into a single server with very large non volatile memory Drive. And you know, I can go up to a few terabytes. So why do I even need distributed?

Then we go through arguments on what is the right transformation? Should I do ETL elt? How do I orchestrate my data?

In all of this the E to success is a layer called metadata where I am collecting the Telemetry. I'm collecting the access the logs the metrics the kpis and somehow I need this metadata to work for me and advise me on how I should build my architecture. Devops comes next and this is what we'll be talking about right after this and how do I improve my operations?

And then finally I am now ready to access it through my my business intelligence tool my AI machine learning or through any other access methods. So this is the reason why we live in a very complex society. So what is data Ops so data of does couple of things?

The way to think of data OBS is it fixes the build and run time issues pertaining to data outcomes? We have openly and happily borrowed Concepts from devops. Now.

One of the first Concepts that came out on in software engineering early in 2000 was Agile development approach which basically said don't just build and build and build software and then through it over the fence. But as your customers built it in an iterative manner in an agile Manner and then, you know keep refining and improving it that is a very important concept called agile. Google did an amazing job of creating this site reliability engineering which which wasn't under the devops place, which said that Is great you saw the bill problem.

Congratulations you you're not doing it in agile manner. How long does it take to to deploy it? Oh, wow, it still takes as many months to deploy then what what is the point?

So devops came out as a result of bringing the build and the runtime teams together so we could deploy our software products easily. We have taken that into Data Ops. These are all people process and Technology.

Then on the Technologies purely technology side. We've also now creating a devops automated platform. So we can build these data outcomes very quickly in like a factory model.

There is a separate talk that I do on yet another Hot Topic in my space for for this year and that's called Data products where we are treating data as a product putting it on a production line with Automation and then and then delivering better time to Value higher reliability and hopefully lower cost. That is what makes up data Ops Finally, I will talk to you what are the components of data Ops by the way, I also want to point out this data Ops there's ml Ops. There's model Ops the AI Ops, so we are starting to see this this.

Close coupling of operations with many many different spaces and they all have their own nuances the goals by the way don't change but but the the details change so the four components of data Ops first is orchestration. So one of the most important things that came out of devops is the idea of infrastructure as code. We are now applying the same principles even on data and and we want the orchestration which is orchestrating my data flow all trading a dag or directed acne graph to be scripted and you see Apache airflow, which is an open source, obviously tool extremely popular, but there are many other preferred Daxter.

In fact, I think one of the slides I mentioned some of these ones so So nausea even taking it to the next level where we are saying, why don't we use the metadata to figure out how should we orchestrate our task? So it's no longer just rule-based. So rule base says if a run successfully go execute B, if B runs successfully or be fails then notify.

So and so that's that's a graph but now what I've been proposing to to our vendors is why don't we bring metadata into the equation? What is a metadata says metadata says a succeeded fine. So B should be a candidate to run but I suspect there's a data quality problem.

I'm getting too many nuts. Why am I getting so many nuts maybe a pipeline broke? So instead of sending the data Downstream and causing a cascading effect.

Why don't I stop The stop it in Step a investigate do root cause analysis and then I go to B. So this orchestration where I'm looking at the metadata. Not just the rules is is a very important step.

the second thing I will talk to you about is automation automation is Or should be everywhere whatever we can automate now there of course dangerous. I don't I want to be careful here because sometimes you need a human in the loop. Otherwise, we see plenty of examples where automation just kicks in but the the point is that automation is an extremely critical every time a decision has to be made using a human a I'm adding time to it be I may may include some margin of error.

So automation across the board, for example Let's say I have I have a database in let's just Snowflake and and I want my users to be able to experiment build their own dashboard. I don't want them to learn how to create a new instance in Snowflake. So I want to automate that process that they make a request.

They get their own configuration. They get a copy of their data. It's all automated and then they start building their applications.

So I've given some examples discovering what data I have and tagging it appropriately finding out. How can I connect disparate data sources based on common Keys. These are some some examples.

How do I automate? So let's say I'm based in San Francisco and I go to my company's office in Germany and do I need a new policy to see data because now relocated to Germany? No, I won't automate the the so having roles and new policies.

I want to automate my Dynamic data authorization. So you see this many many examples of automation. One of my favorite topics for last many years and I have many medium articles on this topic is called data observability.

Once again, we have stolen I mean, sorry borrowed from our infrastructure and application Brethren's how to do observability. I've already touched upon it when I talked about orchestration, how can I Info that I have a data quality problem. My pipeline is running slowly.

I don't have the right instance type because the cost is going to high or the data is not getting utilize or the resource utilization is really low. These are some examples of data observability. It is different from infrastructure and application like APM because it's not at the application Level.

It could be at at very granular level. What sequel query is taking the highest amount of time which users are having the the biggest jobs. So this is all observability is data quality.

It's pipeline reliability is phenobs. So so that observing that is a is a hugely critical part of data Ops finally we come to the last piece. Which is infrastructure, how can I automatically containerize my my data products for in instance?

How do I create new instances as I've mentioned and make our users highly productive. So at the end of the day the goal if you remember from the beginning was how can I get better time to Value improve my quality improve my performance why keeping cost in check? Thank you so much for your kind attention.

I hope this gave you a flavor of what data Ops is and I'm happy to take questions and answer and take it to the next level. Thank you so much.