The Rise of the AI Platform Engineer

Hey everyone, welcome back here to techstrong TV. My next guest is Harish Gaggar. No, it's Gaggar, I'm pretty sure.

But let me ask him to pronounce it. Harish, how do you pronounce your name? It's Harish Gaggar.

Harish Gaggar, easy enough. Harish is the staff's software SRE over at Credit Karma, and we welcome him here today to techstrong TV. Harish, first of all, welcome.

Second of all, tell us how did you come to be an SRE? Yeah. First, thanks, Alan, for having me here.

So, as everyone knows, Credit Karma is an Intuit-owned personal finance platform service which offers consumers free access to credit score, credit reports, and daily monitoring and tools for comparing the financial products. So I came as an SRE because as you know, that we support various different financial services and platforms. And SRE is playing a critical role in managing that infrastructure in place.

So yeah, my role is mostly managing this enterprise AI infra platform, securing the AI agent deployment in a highly controlled environment in Google Cloud, and also helping engineering team to adopt these modern AI capabilities, which is more safely and efficiently. Got it. Were you an SRE before joining Credit Karma?

Not really. I was working as a BI consultant in the past. So I've been with Credit Karma for the last 10 years.

Mm-hmm. But before Credit Karma, I was working in a SaaS BI product company, which is known as Infor Birst. So I was helping clients and delivering these BI and data warehouse solutions by using the Infor Birst product.

And then previous to that, I was in manufacturing domain, where I was working for General Electric, where I was again helping customers in building the BI solutions in SAP Business Object and Data Integrator. And again, I used the same technology in the past, when I started my early career in consulting companies like iGate, Pattni Computers. Got it.

So you made the move from sort of business intelligence to SRE- Yeah ... while at Credit Karma. That's correct.

And SREs handle a lot of different areas, but it sounds like you're really focused in on the AI- Right ... platform aspects of it, correct? Right.

That's right. I think the main thing is the SRE comes in in managing the AI infrastructure. Mm-hmm.

So we build AI managers, like the whole ecosystem of AI platform in the terms of building the GKE cluster for MCP gateway, MCPs, AI agents, and all the CI/CD orchestrator underneath. So yeah, to managing that platform, all these SRE skills come in to make sure that all those agent deployments across the company are going smoothly end to end. Excellent.

com. Yeah. And really, the SRE functions and a lot of SREs consider themselves platform engineers today, right?

Because basically, if they're not building the platform, they're operating with this platform. And so they need to be part of that. What's interesting is that, and I was just at PlatformCon in New York a week or two ago.

Yeah. And what was interesting I saw there is this move of platform engineering, and the platform is now AI platform engineering, where, you know what? Our AI needs the platform to run on.

And so for a while, if you look at platform engineering, it was, all right, we got cloud native, Kubernetes. Right. So that whole cloud native stack, and that's a platform that we need to manage.

And then we have the internal developer platform. Right. Which may have Kubernetes and cloud native in it, but it's IDP.

Right. Right? But now we have this, it's not a new platform, it's the same platform.

Exactly. Same stack, but now we're using it for AI. Right.

Right? And there's some twists. It's not exactly the same, but- Yeah ...

predominantly the same. But it presents its own challenges. It presents its own unique kind of things you've got to do.

So, that kind of takes us to what I wanted to talk to you about today is building secure and scalable enterprise AI agent platforms. Because when we talk about AI today, it's all agentics, right? Right.

That's kind of where we're going. Talk to us about what's different about building and maintaining or managing the platform for agentic AI versus before we were doing AI. Yeah.

So before AI, like in the era. That is where mostly we are deploying the services internally, especially in cloud. We're just building the Docker images, promote that, deploy that in Kubernetes, exposing that UI as endpoint, and let customers to log in.

That's a typical flow of deploying any services. But now, I guess, the thing is, everybody in the notion of how we can automate this whole workflow. Rather than I manually create bunch of, let's say, GitHub PRs to build some infra.

I have to go through a lot of process. I have to manually create the code, get it reviewed, deploy, promote, and then enable the UI endpoint, enable the authentication in place, take care of other part of the application, like managing DB and et cetera. So this takes long time, and now we are in a process of how we can automate this whole system through the agentic workflow.

And this is where the needs is coming in. And now what happening is, I think firstly, enterprise are mostly thinking about that agentic AI should not be just one-off experiment. The real challenge is, it's not just building a one clever agent and do everything magically for you.

It's all about now how you can create a secure and repeatable platform path that let many teams to build, deploy, and operate this agent consistently. So what we do is here, we build this in a three phase, by the way. One is in the build phase, one is deploy, and one is operate.

In the build phase, what we have done is now we have created the internal dev portal workflow. It's just a UI where user can go in and supply some of the information, saying that, "Hey, this is my agent name. This is my framework," let's say LangGraph or LangChain, whatever framework you want to use for your agent.

Then we ask users that, "Hey, do you need a memory for your agent? Do you need a tracking for your agents? Do you want to eval on your agents?

" So once we ask all these questions, user just submit that request on that UI, and then we kicks in our orchestrator, which actually create a GitHub repository for you, which contains all the framework for you. And that cover all the guardrails as well. So that's one piece of automations.

You no need to manually create a GitHub and pull in 100 different things to make that agent compatible as per your company standard. Now we have a dev portal workload which is going to do things for you. Second is the deployment phase.

Once your GitHub repository is ready, user will come in and they can download that repository locally, and they can do customization, whatever they want, and get the agent up and running on the local. Once they merge the PR, then again, we have a orchestrator underneath, which going to kicks in and deploy that agent into GKE cluster. Now, main part come in.

Once your agent get deployed into GKE cluster, main part is coming is the workload identity. This is where we allow that agent to communicate securely to rest of the company resources. The last part is operate, which is where we comes in, where we talk about how we can let these agents to communicate back to our agent memory database, so tracing and eval, or how it can manage the secrets, how it can build the observability on top of it.

So this cannot be afterthoughts. So every agent needs to plug in into a centralized memory tracking and eval and security controls from the day one. And what we do is we don't let that burden on end users.

So the moment that orchestrator when build and run your agent on GKE cluster, it automatically create all these dependent resources into the Google Cloud for them. So that's the automation, the whole thing. User just focus on submitting the request on dev portal and get the repository done, and then test things locally and merge the PR.

Rest all stuff, like deploying into GKE, all these resources automatically will happen for them. So again, to summarize this, the core message here is that enterprise agent or enterprise-- Basically, enterprise scale the AI agents by creating a pay path road, so that developer can move fast. And all the security, identity, memory, observability, and compliance, that should be baked in into the platform for them as a default.

I love it. So this is stuff you're working on right now at Credit Karma. Right.

You think Credit Karma's just that far out ahead of a lot of people in the industry, or you think most companies are at this place? I think everybody's catching up. We are also trying to follow the industry trends.

For example, most of the companies, they are maybe leveraging some remote MCP gateway, or they build their own to build their MCP servers to connect to all the end resources. We also build our own internal gateway. Similarly, we also build our own agentic pay path internally.

The key challenge here is that in Credit Karma, we cannot use the third-party software that easily because of compliance requirements. So everything we have to build from the scratch. The whole platform and services, including gateway, including MCP servers, everything has to be built from the scratch, and they will remain in air gap environment.

So that process takes some time, but eventually we catched up, and I think now we have a platform where we have lots of use cases coming every day where people just leverage on our portal system, and they can deploy their agent in a day, literally in a day. Everything is so automated. Anybody can bring their particular business use case, and they no need to rely too much on infra engineers to do things for them.

It's kind of made a sell, so for them to let them roll out their agent and run their business use cases. Excellent, man. And you said something there that bears-- Credit Karma's finance.

You guys are in a much more regulated industry than a lot of people out here Haresh, got one more area question I want to ask. There are people out here watching this. Maybe they're SREs already.

Maybe they want to be SREs. But they're saying, "I listen to the video, I hear what he's saying, but how do I do this? " Yeah.

How did you learn it? What's your advice to people? Yeah.

My advice here is that focus first depend on your industry use case, right? What business problem you want to solve first. So the way I started is I started talking to all my internal stakeholders to understand their use cases first, what they want to achieve.

We got lots of these different type of use cases and the technology requirement underneath. So I went through them. Then my next step was once I have all those requirement, I was just literally talking to our security and compliance team to understand that how we can make this safe, right?

How we can implement this solution for end users to make sure that they are fully compliant. And then, what I learned is that, especially in Credit Karma use case, we are running completely in Google Cloud internal network, which has absolutely no connectivity to internet. So now the challenge is that how you can build a platform completely in isolated fashion, right?

So that's where we learned about more deep, in terms of how the Google Cloud resources can be provisioned, how we can safely secure our endpoint, like a UI for agent or any particular services. So we have studied those. And then, we've gone through lot of internal best practices.

We run by with lot of teams. And then, once we tested those things manually, then finally, we went ahead and further and automate those process through our dev portal workflows, right? We don't go in a process of creating those individual PRs to create your instance, right?

Always we have a mindset of automate things. So whatever learnings we got it through all this process, we automated those through our dev portal workflows. So we have hundreds of dev portal workflows to provision different things, to make life easier for not just for developers, but also for our stakeholders if they need something from us.

Fair enough. Haresh, 15 minutes goes quick here. I apologize.

But, I want to thank you for coming on and kind of opening the kimono a little bit and showing our audience how this stuff works in real live companies. Not just vendors trying to sell you something. Yeah.

But how a company like Credit Karma is adopting this, and in a highly regulated industry. Yeah. And making it work for them.

Congratulations to you for doing that, and thanks for coming on here. Thank you. Thank you.

Haresh Gagar, Staff Software Site Reliability Engineer over at Credit Karma here, talking about building secure and scalable enterprise AI agent platforms. You heard it here. We're going to take a break on Textrunk TV.

We'll be right back.