Developing an AI Agent for Subject Matter Expertise with Vicki Reyzelman at AIE 2024

Hello everyone. Thank you so much for attending this session today. I'm so grateful for you being here.

Today we're gonna focus on the topic of developing an AI agent for subject matter expertise. My name is Vicki Zelman. I'm a principal strategic consultant at UKG.

And I have spent over 25 years in the tech industry working for companies like Deloitte, Yahoo, Akamai, Salesforce, and most recently UKG. I've been in this space for a long time, as I mentioned, and I've been focused on the internet technology. But when AI came out, I saw an incredible potential around generative AI and what it could do for the business.

And so I'm happy to share with you today an application that we've developed at UKG to help our subject matter experts be more productive. So let's talk about what we're gonna cover today. The challenge, why did we even pick this problem?

The, the problem of subject matter expertise. I'm very lucky to be working in a group of many subject matter experts. These are people with tens and 20 and 30 years of experience that been doing their job for a long time.

They know exactly what they do, how they do it, and it's an incredible talent. However, as our organization at UKG scaling, more and more questions are coming in around their subject matter expertise. And so by developing an AI agent that allows them to train a computer a a generative AI model in, in their knowledge, allows us to answer those questions faster and better.

So we came up with the Vertex AI agent builder, which is a Google solution that we've used to create this agent that marries the chat interface with domain specific knowledge of our subject matter experts. But as we gone through this project over the past year, we've ran into a few roadblocks, things that no one knew the answers to that I'd like to take you through today. So you don't have to do the same thing.

Specifically three areas I wanna cover in this presentations. The first one is you need to ensure you have good data. If you put garbage in, you get garbage out.

So we don't want that. Also, adjusting the design process for AI agents is very different in terms of interaction than the SaaS applications we've all been building, which are very heavily workflow based. So a lot of considerations going to the design itself.

And last but not least is determining how will humans fit into this entire system? Listen guys, this is scary. You go out there and you tell people you're gonna take over their job and have an AI agent do it.

There's not a lot of people talking with you or around you or about you or wanna get involved in your project. So I was very lucky when Natasha Lev, my friend, who is a SME and cloud sql, decided to be our Guinea pig for this adventure. And of course, we're gonna cover future AI agents.

Future AI agents are gonna be completely different than what we talk about today. The evolution of these agents towards autonomy is happening at rapid speed, and so we'll cover some of the points around that. So hope you're ready for the next 20 minutes.

This is quite an adventure. So let's talk a little bit about the challenge of scale of subject matter expertise. Now we can talk about technology subject matter expertise, but I've also spoken to many people around the company, everyone from people in training to people in customer success.

And they all have the subject matter expertise. They have these incredibly talented people who need to train more junior people or provide answers around the company and they just don't have the time to train them. So think about this network, the scale and the framework of the subject matter expertise, not within the specific use case that we've done, but how it can apply to your company and to your business case specifically.

So customers are more demanding than ever. They want timely responses, they want accurate information, and they want it now. And that demand continues to increase.

Whether you are working with your internal customers or external customers, more and more questions are being asked. There's also a limited availability of subject matter experts to respond to these questions. So these people with many years of experience with incredible domain knowledge around UKG specific information are far and few between.

And that is why our group was organized. So how do we scale these talented people to make sure that they're working on the most pressing and important problems? This is where the AI agent comes in.

And of course we've all seen it once again, working in corporate America. There's inconsistent knowledge. People come, people go, we hire new people.

But that knowledge transfer never happens and the huge amount of an institutional memory just walks out the door without you ever being able to capture it. With the ability to input this information into the AI agent, you are actually capturing the knowledge of your enterprise. You are making sure that that knowledge can stay up to date and you're monitoring it for any inconsistencies.

This is exactly what we've been needing for a very, very long time and will allow our organizations to be much more productive in the future. So it's not just me telling me, telling you about the subject matter expert problem, it's actually many in the industry are seeing this as well. And this is a, a chart that I've got from AI chatbots, user interfaces from Intercom.

And what they found when they did their research is that the volume of questions is increasing. And right now we're still kind of highly reliant on humans to answer those questions and some questions or answers coming from the knowledge base itself. As we progress into the future, really what we're seeing is more of these questions can now be answered by the AI agent.

And it's all because of the generative AI capability around allowing us as humans to interact in the matter that we're most most familiar with, which is having a conversation or having a chat. So as we will progress in the future, what we're gonna see, the number of questions will continue to increase, but we're we're gonna see is more AI agents taking on those questions and answering them with accuracy and consistency and the number of human responses will only be necessary for the most creative and critical thinking answers that the agent can't possibly come up with. And that is really the beauty of what our work will look like in the future.

So this is a much more complex model, but I wanted to cover the architecture behind this design. Again, this is very, very high level, but the core of it is based on land chain. And land chain is really a framework of how we do retrieval, augmented generation, how do we combine the information we have in the large language models which are trained on the internet data with that custom information that we contain in our enterprise?

Now, there's a certain way that cloud sql, for example, can be installed as a database, but there's also a very specific way that we do it. At UKG, there are also specific projects you need to set up. There's specific compliance requirements and things that are not generic.

And so that's why we needed to work with Natasha in this case who our who is our cloud SQL expert. And what we did is we worked with her to get all her knowledge based documents. Natasha has been advising in cloud SQL for many years.

So she had designed documents. She also had a lot of experience around the types of questions that the customers come to ask her all the time. And her concern was, I answered the same question over and over, I often have to repeat myself.

And so why not have the AI agent do that for her? This is where we created a specific QA q and a question and answer document. It's essentially document of the top 10 questions she gets asked over and over with the answers she provides back to her customers.

We were able to upload that and embed it into our vector database, which is then used to answer the questions from our customers. So as you can see, there is this whole knowledge flow in terms of how we collect the information that's very specific to UKG, how we embed it and how we put it and we index it in the vector database. But there's also the second flow and that's where the user, in this case, maybe a database administrator that's trying to stand up a new database at UKG will interact.

And the way we saw that interaction is kind of like them calling up Natasha, Hey Natasha, I need to stand up a new database. What do I do? In this case, the agent would go and essentially retrieve the documents that Natasha has put together around how to stand up the new database and augment the generic response of how a database like cloud SQL has to be installed using the large language model.

In this case we're using Gemini. Then it will combine and that those responses in the prompt and return that back to the customer. Now the customer has more information.

And so this is where memory becomes really important. It has a follow up question. Oh, thanks for the information Natasha.

I now know how to stand up my database. Well how do I actually scale and make sure that I build it enough capacity for future growth? That's a separate question that we will then need to do a similar kind of roundabout analysis to get that response back to the customer.

And so it's not just about creating this document system, but it's also capturing those additional question because you, you might not actually know which questions the customer will ask next. And so you can start expanding your agent with those questions. So this is our first very, very rudimentary, I call it MVP model using the Vertex AI agent builder and I'm happy to provide it to you guys so you can play around with it so you can see how it actually works.

But it combines those two concepts that we just discussed, the RAD model and the chat interface. It asks you to essentially provide recommendations. It asks you if you wanna learn more about the subject, it can actually connect you with Natasha and give her information to you.

And then of course the wild ask me anything option which kinda allows you to interact. And this is where we can actually capture more of those questions that we need to have the answers to. So while this is a specific example, what we're trying to create is really framework for creating and scaling subject matter expertise in a very low code way.

So what we wanna do is say, okay, this is not just for cloud sql, now we can expand it to our Mango DB experts or, um, how about our training experts? What about our customer success team? And give them essentially cookie cutter way that they can stand up these agents for their customers.

In this case we use Google Vertex AI Agent builder, but you can use Microsoft products and many others out there that can build very similar prototypes. The ability to continue to improve and add new resources and sources of data is what's gonna make this agent powerful. Not only do we need to think about what, how we can expand this to make it more useful, but also ways we have to test this agent and continually improve it to make sure that we actually have other agents testing it and make sure that it's providing consistent answers, right?

Because the threat of hallucinations is so very high and it's actually rewarding based on the best answers provided to the customer because in the end, we want the customer to come to this agent and engage with it for those first initial questions. So let's talk about some of the considerations. I'll be honest with you this, we've been working on this almost a year and it's been a lot of stops and goes and we've used Google to answer some of our questions, but to be honest, this is very new and it's changing all the time.

But I wanted to make things easier for you. So let's go through some of the considerations you should be making as you're building your own AI agents. Very important thing here is garbage and garbage out that refers to everything.

It hasn't changed, um, and it's super relevant for AI technology. It is very, very important when you work with these subject matter experts that you ensure that the quality of content. The first time we tried to build something like this, we've actually taken all of Natasha's documents and we threw them into the database and we started asking questions, what do you think our um, answers were?

Like, not very good. I can tell you we realized very quickly from that experiment that the quality of data is king and just taking every data source that we could find and just pulling it together is not gonna give us the results we needed. This is where we came up with a very clean approach of question and answer, Natasha, you know exactly which questions people come and ask you every day.

Let's write those down and let's find the best answers that you would provide them and and have that being our central repository. Now this is a very fixed kind of way of doing it, but it helps you clean up the other types of data that you might add into your collection. Essentially, the data that you provide needs to be accurate and correctly represent the reality.

It also has to be complete. It. You can't just provide, uh, value here, but then say something different in a different document and expect AI to know what it is.

It has to be consistent and not contradict a other data. So just piling your data in doesn't mean that you're actually gonna get that. That means that data has to be very much curated and identified to make sure it's accurate and consistently reviewed throughout this whole process because the data needs to be fresh, it needs to be recent and up to date.

Cloud SQL is constantly making new releases. There's cloud SQL, cloud, SQL Plus. How do we interact with AI when a cloud SQL engages with that technology?

And so we have to consistently improve and make sure that we remove some articles, we add others, and it has to be unique data. So if you have two documents that have the same question but have different answers, that does not work very well, you are gonna get one time, you're gonna get one answer the next time the customer's gonna get another. And that is absolutely not what we want.

So make sure that your data is clean, it matches these requirements and that you're constantly curating this data to make sure agent has the latest information. The design process is completely different. We're so used to the SaaS model of, Hey, let's figure out the best workflow 190% of the time that the customer takes through our system and we're just going to implement that, right?

And the rest 10% will learn what to do with it. This really does not work for an AI agent problem. And the reason is, and the beauty of it is, is the AI agent is a custom design tool.

Every AI agent res kind of interaction will be custom to that customer. The interaction I have with this versus the interaction that Brian has with this agent will be completely different. And so it's very, very important to spend time understanding how the subject matter experts work.

What are those interactions? What are the questions that they've seen across the board? And really design the agent to be that flexible.

So we also need to uh, think about potential error states. So how do customers get value from the beginning? What if there's no answer or poor output?

Um, how do we make sure they come back if the answer is wrong the first time, we only have one time to get this right? So you have to think through the right error messaging around these types of responses. You have to also evaluate the risks.

False positives or false negatives will happen. So how do you address them in those in the design process? What will happen if the wrong response leads to serious or percussions?

What if Brian goes ahead and I implements the cloud SQL database, but the answers that he got from the agents were actually incorrect? Now the system is not built in the right way. So what are the, the repercussions are even higher for healthcare, maybe finance.

Um, so you really have to think about at what point do you make that human AI transition? If the AI agent is not a hundred percent consistent or sure about the answer in a healthcare setting for example, you might automatically transition them to a human, a nurse or a doctor administrator that can answer their question. You also have to consider new constraints.

What are the unrealistic expectations or fears for the product? What are the technical constraints about costs and limitations? I can tell you that AI agent building is very expensive, especially the first time you're doing it.

You are running into a lot of interesting questions that you've never had to address before. Every time you make a call to Gemini to practice, to essentially process that new prompt, that's a cost to your business. So the question becomes is how do you scale that model?

How do you scale that model where it's more efficient for your AI agent to answer the question than to transfer it to a human? And what are those interactions between them? What are the constraints, um, and costs and limitations?

And track that very, very carefully because those costs will also change. And don't think it's just technical costs. That's the other thing.

The amount of time you'll have to spend testing this, curating the data, making sure everything is up to speed is actually also needs a specific team that will be working on this. So these chat agents are not kind of set and forget systems. They're living organisms that need to be supported and fed with new information and make sure that they're working in proper order.

So it's a very different design process and most important of all, and I we always forget this part because we get so excited, I personally do about technology, is how will humans fit into the system? I will tell you, while Natasha has been very gracious with her time and working with us on the subject matter expertise agent, other subject matter experts were not as gracious, they were afraid there was a lot of fear for them in terms of how this agent will create responses. How will that reflect on them and their work and in the end, will they still have their job?

All these things need to be addressed. So understand the human constraints and use of the system. Remember as technologists, we want the agent to have as much autonomy as it can, of course given that we test it, but human intervention is should still be allowed or even required.

You wanna spot check the agent. You wanna make sure that the human is also interfacing with your customer for that special touch at the end where you are dealing with complex problems, you also wanna continue to monitor the features while it's in operations. Again, that's why you need the team constantly looking into, into it to make sure that it's still working on track, that the AI hasn't run away with all the information and now providing crazy output to your customers.

And you probably need some sort of approval step to inquire in in the quality. Now I'm a big fan of using technology, that's why I'm in this field. And there are actually AI judges.

So I wanna talk about two AI judge models that I think will be very useful in monitoring this technology. The first AI judge should be based on consistency. This is a judge that will consistently te test your agent to make sure that every time you say one plus one, the answer is two and not three and not four and not minus 10, right?

The other agent that you should really be thinking about in terms of an AI judge is reward agent. Now the answer might be more general than two, right? So at that point you start awarding points in terms of the quality of the response that generated by that agent.

And so you want both of those judges in place to constantly checking the system because to be honest, it's really hard for us as humans to be on top of it all the time. And there's more stuff for us, fun stuff for us to do to be honest. So evaluate the feature after lunch.

So what does that mean? How do you look back on how the out automated system performed? Compare that historical data that you've gotten from the judges.

Is it giving your customers the responses you want? Are they quality? Are they consistent?

It's very, very important to make sure that you build that into your lifecycle of designing the AI agent. So what does the future of AI agents looks like? Well, we started with this chat bot interface, right?

Which was very, very simple back and forth chat. A lot of the questions predetermined. We're now trying to augment it with more information around our customer implementations, our secret sauce.

Um, and that's what we've kind of done with this, this agent that I presented to you with this cloud SQL agent. Now we're moving into more of the co-pilot situation where we're interacting with the agent, but the agent already knows how to do the work. And what we're doing is we're actually, we're something or checking what the agent has done to complete the task.

But the end goal, and I don't think this is coming, I think this is coming very much quicker than we can ever imagine probably in the next few years as we're looking at many more fully autonomous agents. Because we will have the judge agents checking them all the time. We will have the humans in the loop.

We will continue to monitor and update the data to make sure that they have the latest information. By building these fully autonomous agents. We can all spend more time at the beach and let the agents do the work.

So I hope you found this presentation helpful. I definitely enjoyed speaking with you today about it and I would love to hear your questions. Please reach out to me on LinkedIn.

I will try to also be in the chat during the conference. And I look forward to hearing about your business applications and the agents that you are building. Thank you so much.