Enhancing AI Observability with Coralogix’s Liran Hason

Hello, and welcome to the latest edition of the Techstrong AI video series. I'm your host, Mike Zurich. Today we're with Lauren Hassan, who is AI officer for CoreLogics, and they just bought a company called ria, and he's gonna explain the relationship between these things.

But the new company is called CoreLogics ai, which is an arm of CoreLogics. Lauren, welcome to show. Thank you, Michael.

Thank you for having me. All right, So explain to us how this acquisition came about and how do these pieces all fit together? Absolutely.

Um, so what we've done in por, you know, we started actually before AI was called back in 2019. Um, and we sat on the mission, like we realized that on one hand AI is extremely powerful. We all know that by now.

Um, and with that we also know that it's very, very risky. So we realized there has to be some human oversight and control so that we, you know, human beings, society companies can actually rely on it. Uh, and that's how we started aporia.

And over the time we built our platform to provide full observability for AI systems and AI applications as well as guardrails for this system. So the companies that are using these technologies can do it in a responsible way. I don't think a lot of people understand what observability is and how it applies to ai.

And then, you know, we can then connect the dots back to, you know, how these things might then be used to govern this stuff. So walk us through it a little bit. Absolutely.

So we've all been using applications and software, right? And we all know that sometimes, sometimes, um, it might not work. It might have a bug or something could break down.

For the companies who develop these applications, they need some way to identify these issues, you know, as soon as possible so they can mitigate any, you know, potential negative outcome. Um, so observability is really providing the means and tools for these developers to constantly be in control. Things like the monitor, um, in an ICU room, right?

Like he can see there's always a heartbeat. So essentially observability systems give you a heartbeat for software. Now what we've built is this heartbeats monitoring system, but not for traditional software, which already, you know, there are quite, um, wide variety of companies that, uh, provide solutions for that, but rather for AI applications that could hallucinate, make up fact, um, you know, um, um, perform or act in an irresponsible manner or unethical way.

Um, so this is what we've done. And then how does that get used to build the guardrails? That is, are those guardrails just code that I'm writing or are they themselves AI agents that are trained specifically to protect other AI models and agents?

So, it's a tricky question, right? Like on one hand, I, I wanna, I wanna tell you that yes, there's 100% deterministic, no AI involved. Um, and honestly like when, when we started, you know, that was our main approach.

We'll make it as deterministic as possible. Uh, but as time went by and as we worked with more use cases and processed more data, we realized, okay, we have to include ai. Um, and the way we do it is really, we built a very unique detection engine that is built on small language models or s SLMs.

So if chat GPT is like this massive hundreds of billions parameters language model, we've built an engine that is built of multiple small language models, each one's specialized in the specific area of issues or challenges in the ai. So you've come to the heart of the challenge that I think a lot of people are having is so many of the processes, especially in it, are deterministic and the AI models are probabilistic. And so there's a chance that the AI model is not going to pursue, present the same result the exact same way every time.

And people are expecting that it will do exactly that. So how do we kinda incorporate something that is probabilistic and adds value, but into a deterministic workflow so that we can figure out, you know, what's reliable and what may be the best guess? Right?

So I think it's all about kind of aligning expectations with the users and with us or society as users of these apps, right? Uh, because if we do expect, you know, CHE G PT and this kind of a applications to be 100% correct, even in the cost of, you know, they would make up something, then I don't think it's realistic to get there. But if we kind of limit the boundaries, right?

So the way I like to think about it, AI is like this crazy machine or crazy monster, um, that moves around and is doing all sorts of things we need to have and draw some boundaries in which we say in these areas our, under these subjects for example, the, we trust the AI to provide a reliable trustable answer. Okay? So let's take an example.

Um, let's say we have a customer chat support, right? That aims to support customers when they have an issue. Um, and let's take an example company.

com, right? So it's okay for that chat bot to answer on the subject that relate to purchasing an item with Amazon, um, I don't know, maybe refund policy and so on and so forth. But as we get kind of further away from this main subject, the chances that this AI agent is going to provide a correct answer is reducing Now without any proper guardrails mechanism, the error will operate, it'll continue to output something, even if it's completely far off, right?

Even if you'd ask, Hey, should I buy Nvidia stock? Right? Like, I think today it becomes a very, very interesting question.

It would answer it even though it shouldn't, right? So providing guardrail is kind of for us human beings is how do we limit it to a known region or a known, um, list of subjects that feel more confident with, if that makes sense. Mm-hmm.

Who should be In charge of the guardrails and observability? 'cause sometimes I feel like if I ask the data science team to go do this, am I not essentially asking the proverbial fox to guard the hen house because you know, they're gonna have a biased opinion in the first place. So do we need like a third party here to kind of be the, the overseer of the observability as it were?

I, I think we have in, in total, uh, like three different parties that should be involved. Uh, we will focus on the should, uh, because unfortunately we're not there yet, but first and foremost government, right? Like the governance rely, are reliable to the safety of all of us.

So regulation, rule system, enforcing and making sure that every company that adopts this technology use it in safe and responsible way. Um, then the leadership of these companies, now, it's very nice to go outside and say, Hey, we have this, we've brought this new AI capability into our system. But what happens when later on you get on the news with someone who committed the suicide due to their interaction with an AI chat box, right?

Um, and lastly, yes, as software engineer, as data scientist to actually build these kind of things, it is important to be aware of that, not only from the safety part, but also how do you make it reliable. I think, you know, uh, building something you wanna ensure it works really, really well, it's really accurate, you wanna be proud of it. Um, so having proper guardrails and mechanisms against these edge cases is just, you know, I think mandatory in part of this.

So these are the kind of the triangle of government leadership company or corporate leadership with practitioners that should all collaborate together to ensure safety of ai. Ultimately, what will be the relationship between the large language models and the small language models you discussed? Um, will the small ones outnumber the large ones eventually and they'll be the things that we're using to drive the agents we're gonna build.

And I guess if they're smaller and they're more narrowly focused, will they be more accurate? Does that that sense? Yeah, it does make sense.

Um, I think it's not if they're smaller, they're more accurate, it's kind of, if they're smaller, they're cheaper, it, you can more easily fine tune them and you can easily achieve something that works real, real well. Uh, while with the large ones until last week, you need to be an open opening the eye or meta right to, to have something like that. Uh, I think the entire game playing has changed in the last week with dipsy announcement.

Uh, right. And, and as this field is constantly changing, this is actually why you constantly have to have the ability to observe, monitor and track how the system behave, whether they're small or large language model of they're open source or commercial. So you believe that we are on the cusp of some less expensive way of training these AI models 'cause a lot of controversy around the, how this was all done.

But, um, and I'm not sure anybody's had any way to validate or test that, uh, claim that's being made by, uh, the folks outta China. But um, what's your assessment of that whole conversation right now? First, I think that the advancement by deep with the R one model is no less than amazing.

And, and great for us as AI community, it takes us really, you know, few steps further, uh, on our way to a GI. With that, I will say I think, um, there's a bit overreaction in the market and the way people perceive it. Um, yes, it is a game changer.

Yes, it is changing the playing field. Um, what I'm actually most excited about is by the fact that the cost to run these models, you know, got produced by 30 times, uh, or so, it suddenly unlocks a lot of potential applications that, you know, a week ago were just considered too expensive to make commercial sense. Um, so I think in general it's, it's all good news.

Um, how the market is going to react. I think in a, we we're all set and, and interested to see Regardless of the hardware side of that equation, um, is the victory here for open source. And basically we have now a mechanism where open source can keep pace with commercial developments and um, ultimately bring down the cost and to your point, make it more accessible.

There are definitely advantages for open source in, in general as an approach. Um, but I, I do wanna point out something here, like there's a lot of buzz about the fact that, um, Dipsy is open source and LAMA is open source and yes, it does allow us to build on top of these models or you know, to take them as a base model and further optimize them to something new and even better. Uh, but we need to remind ourselves that this is not completely 100% open source.

It's not like we have all the base data that was used to train these models. Um, so there is a kind of small caveat that we need to remind ourselves, uh, with that. So is it a huge leap versus commercial models?

I think, um, I think it's an interesting question. Uh, I'm not sure yet What's your best advice to folks? 'cause I think they understand that they, uh, wanna take advantage of ai, but there are governance issues and they have to figure out how to operationalize it all.

How do I get started? So I think when you start an AI project or multiple AI projects across the organization, it is important one to set clear goal and short milestone. Like what is the first deliverable we wanna produce out to the market?

And once you get to this MVP or POC working in your environment, um, really have an evaluation or observability system in place even before production, just so you can actually test and ensure you are going to, to succeed in production. Uh, just to share with you, like usually when we meet with different accounts and different enterprises, what we hear is that they have about 300 different use cases for ai. When we talk and ask like, Hey, how many of them are actually in the works?

The number drops to about a dozen or so. And then when we ask interesting how many of them are actually live in production? This is where you see people literally changing colors with one, two at most.

And, and the reason being is building something became with, with gene AI and elements became quite easy. But to get to the point from working 80% to 99%, something that I can actually rely upon with my brand, with, with our name in, in production, there's a huge gap to get there. And evaluation and observability and proper testing Is a key to get there.

All right folks, you heard it here. Just 'cause we have AI doesn't mean that we don't throw out all our fundamental principles of which observability is one of them. And if we want all this AI stuff to work as advertised, we better know how it works.

Hey Lauren, thanks for being on the show. Thank you very much. All right.

Thank you for all watching Love. Latest episode of the Techstrong AI video series. You can find this episode and others on our website.

Until then, we'll see you next time.