Optimizing AI Model Performance with Invisible Technologies’ Adam Haney

This is Textron tv. Hey everyone, welcome back to Textron tv. Our next guest is Adam Haney.

No, not that Mr. Haney, but, um, Adam Haney, head of Technology, invisible Technologies, uh, that's pretty cool. Head of Technology and Invisible Technologies.

I didn't think about that before, but let's welcome Adam to the show. Hey, Adam, how are you, man? Welcome.

It's great to have you on. Thanks so much, Alan. I always appreciate it.

A, a good green anchor reference and says, definitely the head of technology is a little bit different than the other Mr. Haney, you might know. Yeah, I, I, I, uh, I'm gonna give you extra points to that.

I don't know how many people out there caught the Green Acres reference the stores, but, um, anyway, welcome and thanks for having, having you on, being on here with us. Adam, before we jump into Invisible Technologies and what I want to talk about today, I always like to give our audience a sense of who they're talking to. Give us a sense of Adam Haney.

Sure. So I, I, uh, have worked in, in various different roles across startups, usually in the scale up phase of building platforms. And so, uh, I'd say at, at, at my root, I'm a tinkerer, and so I, uh, you know, have always continued to go examine different technologies and as well as things outside of technology.

Uh, so I've been very mechanical since I was really small. Um, but, uh, most recently before Invisible, uh, I worked at Meta where I, I focused on, uh, news and personalization problems there. Uh, and then prior to that I built multiple different, uh, technology platforms for managing labor workplaces.

And so that's been kind of my background overall. Very cool. Excellent.

Now, so that's not terribly different than me. I'm also curiosity killed the cat kind of thing. Right.

I'm always looking to poke into whatever's new and exciting. And, and of course for, for people like you and I, Adam, that revolves around AI these days, right. It's, it's the new exciting thing.

It's like Absolutely. You know, the final frontier maybe, well, I don't know if it's the final Frontier Quantum will be next, but we got time to play with Quantum. Let's play with AI for now.

Um, you mentioned, I mentioned that you were head of technology with Invisible Technologies. A lot of people out here probably don't know Invisible Technologies. Adam, what would you tell them?

Like, what's Invisible Technology's about? So Invisible is an AI process platform. Uh, we help enterprises to be able to deploy AI at scale.

Um, and, and we do that through various different mechanisms. So whether that's helping them to run evaluations so that they understand how well their models are performing, as well as helping to connect those AI models to their existing software tools in order to be able to actually solve problems with ai. And we have a heritage of also working with several of the foundation model providers in order to help them build the data sets that ultimately were involved in creating Theis in the first place.

So we have strong experience both in building AI models as well as helping people to deployed them in a, an applied way. Excellent. You know, obviously a lot of talk about AI in general over the last, well, two years certainly since, you know, GPT Jet GPT burst on the scene two and a half years, whatever it was.

Um, but recently there's been an increased kind of focus on how do we train these models, right? What goes into training these models? Should we all be using the giant LLMs that the, the hyperscalers are using, for instance, right?

Should we be developing our own LLMs? Will we see the sort of an LLM marketplace, if you will, where, where you can buy specific LLMs? Should we train them with data from the internet, or should we have synthetic data right there?

There was a little bit of irony there. When Open AI accused the, uh, deep seek folks of, of using their data for training, deep seek, well open AI used other people's data for training open ai, right? Um, but nevertheless, I mean, these are, if we're gonna move forward with ai, and, and it's something we talk about on Textron Gang and around here all the time, we've gotta get this stuff sort of a little bit more locked down, right?

That we, we shouldn't be worried about. Is there confidential data going in there? How is this thing trained, right?

Uh, poisoning, hallucinations, all, all of the above. How is invisible helping with this? How do you see that problem?

You know, what, what's the way forward here, Adam? Yeah, absolutely. So, I mean, we think a lot about evaluations as a, a, a way to think about safety as it relates to ai.

And so nearly when we work with a client, we come in and we try to understand what is the business problem, as you rightfully called out, what are the risks if this model hallucinates or if there's a problem, uh, with the way that the model tries to solve that business problem. And then we'll usually start a program to build evaluations. And so, um, you're probably familiar with things like MMLU and several of the other sort of industry benchmarks for broad model performance.

But what we find is that, you know, the problem that a particular business has isn't usually captured just by a single existing benchmark. We need to help them to build a program around the problem that they're solving. So if we work with a hospital system and we want them to be able to parse, you know, specific kinds of medical data, there are different forms of evaluation.

Some of those can be automated. So you can use LLMs to think about building, uh, you know, an evaluation program. But at the end of the day, especially for really critical problems that involve safety or, or business critical decisions, you want a human in the loop to then be able to make sure that we have, you know, a domain expert that really deeply understands this problem space to be able to say, yes, this ai, you know, answered correctly or didn't.

And so we think a lot about the, the same way that you would think about automated QA for deploying, you know, new pieces of software. It's necessary to have an, a robust and effective EV evaluations program to deeply understand is the model, or, or is this overall applied AI system that I've built solving the problem, you know, in the way that I expect, uh, for, you know, any kind of, you know, business use, uh, of an AI model. Excellent.

So there's, I wanna hone in specifically on using sort of synthetic data versus, I call it real data. Other people call it human data. Look, I it's a little bit wild, wild west right now.

I get it. Yeah. But when we get civilized that of what do you think the right path here is?

Yeah. I don't think that there's a, a one size fits all answer in either direction. So certainly, you know, when you think about problems around PII or, or, um, you know, uh, privacy in general, there are places for synthetic data.

So that allows you to generate a data set, you know, from something that might be confidential or unsafe as, as well as it allows you to really increase the data volume that you have from a, from a smaller data set. Um, however, you know, synthetic data that the process overall involves training a model on, you know, a smaller, uh, data set, and then generating, you know, either a representative data set or a new larger data set. Well, that process amplifies the biases of the dataset that you start with.

And so there's not a future that I see where we're able to purely move over to synthetic data sets. I don't think we've generated all of the data that we're gonna generate, you know, across all of society. And so you sort of think about the process, there's gonna be some amount of model drift, or the, the underlying data distribution is not always gonna be representative of new things that happen in the world.

And so we believe that there's a place for humans in terms of generating new data sets, uh, as well as, you know, if there is any problem in that underlying dataset, creating a a synthetic dataset is only gonna amplify that problem. And so that's once again, a place where we see a need to make sure that any training program, evaluation program, fine tuning program, uh, has a strong need for human data, uh, as a, as a part of that process. So let me be real selfish a little bit here.

Sure. Right. So we're, we're a publisher.

We publish, I don't know, 120 articles, 30 to 40 videos every week, every week, week in and week out. We probably have, like, for instance, on DevOps, we probably have the largest collection of DevOps content in the world. Mm-hmm.

I am sure that that data has been used to train models. Yeah. Good.

You know, and I, at some level, man, I'd like to get compensated for that, obviously. Right, right. But also, I would like to be able to use that to create maybe my own models, right?

And, and, and make it a smaller data set. But if you're looking for DevOps, I'm your man. Yeah.

Right. And, and so what about people like me, right, to bring it really home. Can we create our own sort of smaller data set that's very focused on that?

And, you know, how, how do I, how do you market such a thing? How do you, I mean, do you envision a market evolving around those kind of small data sets? Yeah, absolutely.

I mean, I think that specialization for some of these models makes complete sense. You know, when you think about the number of parameters that are going into the really large models, um, it, it's sort of like, you know, having a PhD student do a fifth grade math problem. Um, it's just not necessary for all kinds of problems.

And so we're gonna see, you know, not only is there a massive, you know, energy consumption component to that, there's a huge cost, uh, component to it. And, and overall it's, it's the same sort of optimization process that we've gone through, you know, over the last couple of decades with cloud or, or using appropriate tools for any kind of data processing. You wanna use the right tool for the job.

Uh, and so I, I think that certainly if you know that you're gonna be only solving DevOps problems over and over again, it makes complete sense to say what is the premier leading, uh, you know, model for solving DevOps problems as well as, to your point, you know, you as a publisher have an immense amount of expertise you might choose, you know, in the future. How you think about, you know, the, the thing that you do publish, how you think about curating your catalog of content in order to make sure that, you know, your expertise is being appreciated. 'cause right now, you know, obviously it is the wild, wild west in terms of the, you know, the way that, uh, the entire internet is being used to train models.

Absolutely. Absolutely. So a lot of talk around, are we kidding ourselves in that?

Yeah. The bi, you know, the old story, the big guys will always be able to do what they want. They'll create their own LLMs, they'll create their own data sets, they'll customize it better, and they'll, you know, squeeze every bit of usage out of ai.

But for the rest of us, do you think, Adam, where we're gonna be able to do custom data sets really EAs, you know, easily, easily enough to make use of them, will we really be able to get the ROI out of AI without having to do that and, and you know, how big a lift is having to do these kinds of things? Yeah, I mean, so that's one of the things that, that invisible helps with very frequently is when we go and we work with, you know, a business or somebody, somebody that has a, a unique data set that needs to create a model that's specific to them to solve their problem, um, you know, data as an asset or these businesses is a huge part of their competitive advantage. And so, uh, I, I definitely think that there's an opportunity.

You're already, you know, seeing publishers and people that would normally put their content on the internet that are starting to sort of pull back and put it behind paywalls. I know Reddit recently, you know, signed a huge deal in, in terms of how they're thinking about distributing their content to model providers. We work with, uh, enterprises that have never published that data.

Uh, you know, the, the sit on huge, uh, you know, treasure troves of data from their operations or from running their business overall, uh, they want to be able to train a model on something that was never publicly available, and thus models, uh, do not perform well on, on the kinds of problems that they're trying to solve. So I, I certainly think that there's an opportunity for, you know, whether it's fine tuning or dis distillation or otherwise creating, uh, you know, applied, uh, ai, uh, solutions that solve problems for those businesses. And, and that's where Invisible tries to help.

Absolutely. I may have to talk to you after this interview. Okay.

But beyond that, let's talk a little bit about ROI in ai. Sure. Right.

A lot of people think that's an oxymoron, uh, at this stage of the game anyway, right? I, I just came off the set of Textural Gang, and we were talking about, look at, at this juncture, AI is helping a lot of people in their jobs. And really specifically, if you're a marketing, AI's a huge help in, in creating marketing material sales as well, coding.

It may not be generating code, it may just be fixing code, but nevertheless, one or the other, it, it's kind of helping and, and every other way. But, you know, if you listen to Jensen Long over at, uh, Nvidia this week, you know, we're gonna have, what is it, 10 billion digital workers, you know, agentic ai and all of these things. As I sit here today though, can you show me what jobs have been eliminated because of ai?

You'd be hard pressed, you'd be hard pressed still. Not that that's gonna be like that forever, but again today, so when you're working with customers, what's the ROI that you give showing them, Adam? What's the ROI that they are gaining?

So, I mean, I, I think, you know, I recently read a study that something like only 15% of AI projects ultimately end up making their way to production. And so, you're right, there are a lot of prototypes that are being built right now, and a lot of people that are experiencing with ai, but they're not actually seeing it move the needle in terms of providing value for their business. Uh, and so that's where I think Invisible is uniquely positioned.

We deeply believe that you have to integrate into the current ecosystem. So we're not out to try to replace somebody's existing software tools. We're not, uh, there to try to replace their workforce.

We want to integrate with and, and help them to then, uh, set up the data sets that they need in order to be able to train these models, build in a suite of evaluation tools, so that that way they can confirm that the AI model is working the way that they expect it to for their business. Uh, and so I think that without, you know, all of those pieces, the data preparation, proper model selection, evaluation, you're sort of, you know, doomed to failure, um, because you're not gonna be able to make a model that works effectively, and also you're not gonna be able to trust it. Uh, and so I, I, I agree with you that right now it seems like, you know, we've, we've discovered fire in some ways and we're looking for applications of, you know, this, this new tool.

Uh, and I think it's really important that, you know, you work with a partner that thinks about how that's gonna integrate into the way that your business, uh, actually operates. I think that's a great analogy. I may steal that from you.

I'm just telling you right now, I'll, I'll incorporate it into my data set. Um, but I, I appreciate it. Hey, Adam, we're, we're almost outta time.

But for people who want to go dive into Invisible Technologies a little bit more, what's the website? Sure. co.

Uh, and you can, you can find us there and, uh, we're always happy to chat with people about ways that AI can solve problems for their business. I love it. Adam, sounds like you're having fun doing what you're doing though.

Yeah, absolute. What a great time. Yeah, what a great time to be in this, in this position.

Good luck to you in Invisible Technologies. Come back, keep us posted, and thanks for coming on today. Thanks so much, Alan.

Adam Haney, head of Technology, invisible Technologies that's invisible. Do co check out, especially if you're thinking about how ai, how you're gonna leverage AI at your organization. We're gonna take a break here on Tech Drunk tv.

We'll be back in a moment.