Why Accuracy of RAG Systems is Critical in the AI Age – Techstrong AI Podcast EP35

Hello and welcome to the Techstrong AI Podcast. I'm Amanda Ani. I'm so excited to be here today with Aaron Zuko.

He is the director of Solutions at Appen. How are you doing today? Hi, Amanda.

I'm doing very well, thank you. How, how about yourself Doing well, and I'm excited to speak with you about, uh, re today. But first can you share a little bit about appin and what are y'all working on?

Absolutely. Yeah. So Appin is a, uh, is a leading provider of data for the AI lifecycle.

Uh, essentially we've been, uh, supporting the, the kind of AI data needs of companies for the last 27 years through sourcing and collection of data annotation and model validation and evaluation, which is a whole lot of things. Um, and we're gonna focus in on one aspect of that today. Wonderful.

Yes. So let's get right to it. So that leads us into our topic, which is rag.

And you have mentioned that the human element of RAG is critical when it comes to retrieving data. So can you share a little bit more about that? And then what are some of the problems you see business leaders facing in this area?

Yeah, absolutely. So, so RAG in itself, which stands for Retrieval Augmented Generation, is essentially a, a version of an LLM that has kind of more limited knowledge where that knowledge is essentially your own kind of knowledge base, your own documents that are specific to the company. Uh, right.

So generally speaking, an LLM or a kind of base LLM is trained on, you know, the internet, right? So it's, it's scope of knowledge is very large. And generally speaking, when you're trying to deploy an LLM, um, either internally, right, which could be helpful for your employees to find information, or even in a customer facing experience, you want the knowledge of that LLM to be, uh, kind of specific to, you know, your, your data, your company, uh, right, anything that is associated with, you know, policies or information that is not general knowledge and not gonna be found publicly.

Um, so when we think about the human involvement there, uh, it's really gonna come down to ensuring that the, uh, the, the data itself is the correct data. Um, it's up to date, it is accurate, and it is processed in a way that gives the, the rag system the best chance to find the information that best answers and users prompt. Uh, and that can mean a lot of different things.

And, you know, organizations have lots of, you know, data sources sporadically, um, you know, distributed and, and different contexts involved in the, in those data sets. So ensuring that you have the right humans with the right domain expertise, right, which is gonna be in a lot of cases for rag somebody who's an expert in your organization or the, the data that you're trying to surface to that system, uh, is, it's very much in the loop of that system as it's being built to ensure that it's factual, uh, and relevant in its responses. And some of the challenges that we see there are, um, a kind of curating that initial data set, right?

So instead of saying, Hey, here's all of my company's data, go and synthesize this into an answer for me. Um, we find that companies struggle initially to do, curate that data set, um, to, to limit its scope based off of the goal of whatever that rag system is. And we'll dive into some more of that, um, I think throughout the conversation.

Yes. So, um, during this process, uh, let's go ahead and go into that a little bit more, share a little bit more about that, and then what are some of your tips? Yeah, absolutely.

So, so I think the, you know, thinking about how a rag system works, right? Is, is, again, you're, you're taking these, these separate data sources and trying to distill an answer out of them. Um, so depending on the, the scope of what that rag system should, should understand, ensuring that you can get out the relevant pieces, um, from the overall dataset, not just what the LLM is pulling out, is gonna be important not only to limit, um, kind of confusion or hallucinations of the model, but also from a cost saving perspective, right?

Generally speaking, LLMs will, um, you'll, you'll be charged on your usage of tokens, right? A token is essentially, how many words is it having to, to synthesize for an answer. So what we see is if you, if you kind of have this, you know, let's call it this more unstructured pool of data, there's gonna be less of a, a, a, a, the l m's gonna be less confident in, in terms of what it's retrieving, right?

And synthesizing into an answer, um, to the, to the end user. So the more that you can limit, you know, the, the scope of what that RAG is looking at, you're gonna have more relevant responses, and it's gonna be more cost effective because it's not having to, to process all of this text data to understand, hey, here are the 10 things I think you're talking about. Um, ideally it can say, here's the, the one thing or three things that I think you want, and then synthesize its answer from that.

Um, and there's a few different strategies that can be deployed there, and it really comes down to a, just limiting the size of that pool. And then b is adding some contextual annotations to that data, um, right, such as metadata about a particular piece of data or a summary about a piece of data that can live alongside of that data in the, in the kind of data center, or you call it a kind of vector DVR embeddings. Um, so that when the LLM is going out and looking for that information, it has, you know, the best chance to, to kind of match the intent of the user's prompt with a relevant piece of data.

Okay. So, um, what is your advice to business leaders when it comes to whether or not they should be, um, or I would think it works in harmony together, but when should they use an LLM versus when should they be using the rag? Yeah, so, so Iraq is essentially just an LLM, but again, you're just kind of limiting its scope, right?

So I like to think about it as like, you know, if, if you're going to a doctor, right? You may have, you know, something that's generally achy, you're gonna go see a general practitioner, right? And that's gonna be your kind of like base LLM.

If you have a more specific issue, my foot really hurt something along those lines, you're gonna go to a specialist right about that because they're gonna have more knowledge and, and kind of a deeper knowledge on that particular subject and will likely assist you. So the, the, the foot doctor in that case is going to be the RAG system, and again, it's deeper knowledge on a particular subject in which you wanna deploy and leverage that LLM. So they're, they're the same except, you know, they're just different in terms of the knowledge that they have to be able to keep you back a response.

Okay, good. To clarify that, so this technology, it's advancing rapidly. What do you see in the future as far as, um, data science say a year from now or so?

Yeah, that's a, a big question. I think, um, you know, we're seeing a lot of improvements in these LLM kind of base intelligence, right? So you'll see a lot about reasoning, right, in terms of being able to justify and step through the thought process of how it came to an answer, right?

And if we think about, um, you know, an LLM is essentially, you know, not dissimilar to essentially a search engine, right? You're asking to go find something, it's pulling responses. But in this case, you know, the LLM is doing the heavy lifting of basically saying, this is what I think is the best result.

And then synthesizing that too as a single answer, as opposed to, you know, a kind of page of search results and you're clicking through understanding what's relevant or contains the information. So I think, you know, as we see these model kind of the base intelligence of these models improve, being able to, for them to be able to kind of express their reasoning, right, in terms of how they came to a given answer, that is gonna be a huge improvement even in the context of rag, right? Where again, like it's not just regurgitating, uh, uh, a retreat piece of, of data as is, right?

But it's kind of stepping through reasoning, understanding the intent of the user's query, um, as opposed to just following a particular instruction to a T which again, you know, you might not know the right question to ask. So the, the more that the an LLM can help reason its way to what your intended question was, the better experience you're gonna have as an end user. Awesome.

Well, if there was one key takeaway you could leave our audience with today, what would that be? Um, I would say to, to really understand how you want these systems to be deployed and ensure that you are, are limiting the scope of that goal, right? So again, a a base LLM is gonna be very good at general knowledge type questions.

And if you want to deploy a rag system in a particular area of your business, really not only the data source, um, but the, the kind of experience as you're testing and evaluating that system should also be within the context of that goal, right? So be very specific about, um, you know, what, how you want to use that LLM and how, you know, your, your customers internal or external are gonna use that and really kind of build the, the system around that goal, right? And when we see, um, you know, one thing that we always recommend to customers, right?

Is because a rag system has a lot of different components, right? And as you're trying to improve the system, you wanna, you know, be very scientific about how you go about that, right? So changing a variable at a time, right?

As opposed to, Hey, I'm gonna change out my base model, I'm gonna change out my data source, I'm gonna change all these components, kind of focus in on one piece at a time, and then have a, a way to measure, right, or benchmark those responses throughout those changes that we generally will recommend kind of doing an, an end-to-end evaluation to which will help you establish, uh, a kind of a, a roadmap, right? Of which component you should tackle first, uh, and then go ahead and tackle just that component and then essentially do the end-to-end test again, right? So you're benchmarking these different iterations, um, against essentially a set of golden prompts or scenarios, right?

So again, going back to that concept of understand the goal and the scope of what this system is supposed to be doing, you basically want to have a consistent set of, you know, golden prompts or golden scenarios that you can use to test these systems against. So you can measure, right, ideally, objectively as possible, um, how well it's performing against that goal. And then as you tweak different components of that system, you can come back to that measurement and say, okay, that did have an impact and it had a very large impact, or it had a small impact.

Um, so let's move on to the next component and repeat that, that cycle. Fantastic. Well, thank you so much for coming on our show and sharing your insights with us today.

Absolutely. Thanks for having me. And thank you to our audience.

Stay tuned because there's more.