Techstrong TV September 22, 2025

Hey everyone, it's Alan Shimmel. Welcome back here to Techstrong tv. Our next guest is Edon Ekman.

Uh, first of all, we gotta give Edon to a Mazel. Tuffy just, just had his first child, a boy last weekend, so we are thrilled to have him join the, uh, the parents club. See you.

Thank you so much. Thank you. Thank you for joining us.

So very exciting times for you having a baby, your company coming at his stealth, you know, savor my, my advice to you, having been there, done that, right on both counts. Savor every second of this time because one day, 20 years from now, you're gonna look back and say, wow, those were the best days of my life. Um, so good for you.

Good for you. Thank you so Much. Yeah.

We're not gonna talk a lot about the baby right now. We'll save that for when you're working in the family, but we're gonna talk about COI security, but even before we talk about COI security, let's talk a little bit about you. You're a co-founder there and CTOI believe, correct?

Yeah. Give people a little bit of a sense of your journey, how you got to here. So I started my journey in the, uh, intelligence force of the IDF.

I did, uh, uh, cybersecurity there for about six years. And I was in charge of case security training for, uh, for the intelligence community. And then most of, after I was released from the military service, I dealt with incident response.

Almost my entire career. I was a consultant, one of the first employees in, uh, Signia. And after that I had my old firm.

And I really, really loved the human aspect of it. Like working with security practitioners many times it was in a very, in very hard times for the organizations. 'cause incident is a difficult time, uh, but also other times, uh, preparing for the incident, trying to prevent it.

And I think, uh, it was fascinating to see how different organizations think about security. What are the priorities of, uh, different security teams and so on. And then, uh, a few years ago I started working in a startup called, uh, Canon Security that was acquired by Zscaler.

I worked as the head of SA security research in Zscaler for a while. And almost two years ago I founded, uh, co security with, uh, my two co-founders. Excellent.

So, so you guys have been working at Koi Fruit, working at Koi for two years already to get to this point. Um, you know, even founding a company is not something one does lightly. Right.

Talk to us a little bit about kinda what, what was the mode of, you know, what, what drove you, what was the passion in finding in founding coi? What, you know, every, I speak to a lot of founders, um, multiple serial founder myself. Right.

What, what drove you, what was, in what way were you trying to kind of make the world better solve problems? So I think for me it's, uh, mainly two things. Uh, one of them is creating, uh, a great security product because, uh, during my career as a security consultant, I worked with a lots and lots of security products.

And many of them feel like they're not, you're not even supposed to use them 'cause they're really how to use. And the interface is, is low and it doesn't look so well. And it was really frustrating both for me and for the petitioners I was working with.

And I really had a, a passion of creating a security product that is almost like a B2C app. 'cause it should look good. It should be comfortable, it should solve a, a real problem.

And we knew there was a problem with, uh, software on, uh, endpoints that, uh, we were really passionate in solving, and we wanted to, to create a product that is loved by its users. And the second thing is that I really wanted to create a, a place that is a good place to work because I knew that in my career when I worked in good companies, it was a tremendous experience. The people you meet and the experiences you make together.

Sure. Absolutely. I, uh, you know, as, as I co-founded several companies and rum being the, the last one here, uh, I agree with you Right.

Having a sense of we're all in it together. You know, as you grow, it's hard to keep that feeling of family, if you will. Yeah.

Right. But you want to make it and, and you want to make it a fun place. You wanna make it a place where people want to come to the office, they don't feel like they're coming to a prison cell, you know?

And Absolutely. So yeah, I I try really hard with that too. So when we talk about software on the endpoint, right?

As a potential security mm-hmm. What, what exactly. I mean, there's a lot of facets to endpoint security, right?

There's, there's looking at traffic coming in and making sure no malware gets through. There's exfiltration, there's locking down the data, locking down identity, what, you know, what aspects of endpoint security is COI involved in. I think think that when, when we think about software for the endpoint, like the third thing that comes to mind, uh, for anybody that's been in the industry for a decade or or more, is probably, uh, executables, uh, DLLs binary files.

Like those are the classics. No, uh, like the, the, the classic attack is that somebody sent you a malicious file over email. And I think that many of the current, uh, solutions for endpoint security are still solving that problem.

But this is something that happens less and less as time moves on, because we do make some progress, and now we have great tools that prevent this kind of scenario. Um, but in recent years, we found that most software that actually runs on the endpoint is actually not, not what we imagined at all. It's not the binary software.

It's a high level a language software. Many times the JavaScript, for example, Chrome extensions, people think, oh, it's just extensions. It's just, it's sandboxed.

It's not something serious, but it's, it's software, software nonetheless. And extensions for the IDE for developer environments, those stuff, they can, they have high privileges. They can touch any file on the computer.

So the security world doesn't treat those kinds of software as serious threats because they're not binary, they're not executable, but they still are little pieces of code, or not so little, but run some logic, some business logic on the endpoint. And we found this to be a huge gap in the current, uh, security model for the endpoints. Absolutely.

Absolutely. And, and, you know, so a lot of it is browser-based security, right? So that's what we're talking mm-hmm.

Mm-hmm. Because the browser, more and more, the browser is the interface that users use on the endpoint, and it, it's, it interfaces with the software on the machine. It interfaces with software, you know, up in the cloud everywhere now.

So is that where COI plays, then you're locking down sort of the browser, uh, browser extensions, browser interactions, monitoring web traffic there? So we do provide, uh, security for browser extensions. Mm-hmm.

But our main focus is to secure the actual software that is installed on the endpoint. Got it. Which can include browser extensions, but it can be, uh, NP NM packages.

And right now we have a huge incident in NPM Today. Did you see this? Like 140 packages or something?

I saw. It's crazy. And we have an, uh, updating incident page in our website to help practitioners monitor, because new packages are discovered to be, uh, compromised, uh, by the minute.

Uh, so this is, uh, uh, a great example of how those type, the new type of software, NPM can be marvel for the endpoint, not only endangering the, uh, CITD of the production. You know, I was reading this morning before we got on here, um, evidently I think CrowdStrike is rolling back some of the packages to previous versions that didn't contain the malware. Mm-hmm.

But if, if you're watching this and you're not aware of it, you should get on top of this immediately. Absolutely. Right.

Make sure you don't have any of these packages in there, or maybe go use COI software for this. Right? I mean, this is, this is exactly, you know, the poster child for what you guys are, are trying to prevent, right?

Yeah. And unfortunately, uh, we couldn't have asked for a better demonstration. Um, and even people that are not, uh, using coil coil customers who can reach out to us, and we'll be happy to help, uh, understanding this threat and mitigating Yeah.

This is a serious thing. And, and, and I'm not picking on CrowdStrike, right? I I know George for many years.

I CrowdStrike's a great company, but when it happens to a security company, it's like a double, double whammy, you know? A double double. Yeah, Absolutely.

I don't think it's, it's definitely not CrowdStrike's fault. There are no, uh, secu not many security tools, uh, today that can stop this kind of threat because it's a new type of form. Uh, but it just shows that if it happened to the people that know security best, then it can really happen to anyone.

Absolutely. And it's a threat. And, and it also brings up the whole software supply chain.

And, and, you know, you've gotta know what, what's going into the software you're using, right? What packages and so forth. And I, I think a lot of people, a lot of end users are shielded from that, right?

Whether you use Windows or Mac, you don't think a package is, or Yeah, we don't even think much. I mean, we think of applications maybe, but you know, it, it, it's not like, it's not like you're installing Linux on a desktop and you're picking what packages I want in my Instore, right? Where you're very, you could be very deliberate about what you want and don't want on the machine.

Uh, most of us, you know, we, we click the button and we download and it, we get what we get with it. And, and it, it's, it's scary. Um, it, and we'd be, you know, negligent If we didn't mention that Coy came out of stealth just recently now, and, you know, with a very, very big, a round seed round.

Why don't you tell us about that a little bit? So the way we started COI is we built the risk engine first, and then we started working with, uh, paying customers. And we got the Fortune 500 companies, uh, contacting us because we had this type of risk engine that nobody else in the industry had.

So it really helped us raise, uh, significant sums of money because the value was already proven when we, uh, stopped our seed round and then promptly after we moved on to raising the RA round, uh, uh, $38 million. Uh, that's the, a total funding is, uh, 48 million. And this is great for us because we really see there is a huge gap here and a big opportunity, and we're very determined in, uh, building a product that mitigates this opportunity.

But, uh, you need power, power to do that. 'cause it, this, this is a really big challenge. Absolutely.

Absolutely. Um, so congratulations. Uh, you know, in today's world, it's not COVID times anymore.

VCs have tightened up with the monies. They, it, they don't write checks like they used to. So to raise that kind of CA round combination c and a round is quite an accomplishment.

So congratulations to you and your co-founders and the whole, the whole company. You know, it begs the question, well, what are you gonna do with this money? Right?

A lot of people out here have never been a founder, have never done the startup thing. They say, oh God, they raised $38 million. They must be rich.

Well, no, it don't work like that. Right? We, I wish it did, but really, what did think of it as going to the gas station, right?

And you now filled up the tank with fuel, and the question is, where do you drive? Where are you gonna drive? Where are you going next week?

Exactly. You got a full tank of fuel. Um, so where, where does COI go next here?

So we, we started as a solution for, uh, visual Studio Code and Chrome Ideas and puzzles, uh, which, which was great. And we used the seed round funding to raise great l and d team, uh, funding engineers, really great guys. And, uh, of course the research team.

And, uh, now that we have the A round money, our main focus is, uh, two things. One of them is, uh, creating the, uh, GTM team. Uh, that is really important for our company to scale because now we know that we have a, a great product that, uh, we need to get out there.

And the second thing is to, uh, to hire more people for r and d and start tackling, uh, the harder, uh, problems. Uh, we already have support for NPM and ppe, uh, but those marketplaces are, uh, at different scale and we need to build, uh, we need to build for much, much, uh, larger scale because, uh, we are very ambitious in the, uh, problems that we want to solve. We want to be a single platform for managing really anything on the endpoint, any type of software That's ambitious.

Let's talk about, go to market a little bit to GTM. So is there, like, what's the model? You're gonna hire salespeople to go sell this, you're just going to create market demand from sort of outbound marketing.

Is there a geographic focus? Is there a particular verticals that you're aiming at? Right.

How, how are you going the market? Actually, something that is, uh, pretty unique, uh, at COI is that we get lots of, uh, inbound traction because, because of our risk engine, uh, we keep finding malware all the time in all of the marketplaces that we operate in, and we publish about it. So the risk becomes very apparent for many, uh, security professionals, and then they reach out to us.

So, uh, the marketing of course, kind of created itself just by the research work that we're doing. And, And I assume it's responsible disclosure and all of those things, right? Yeah, Yeah, absolutely.

So we report anything that we find, but since many times it's just malicious items, like malicious packages, malicious, uh, extensions or modules, then we don't need to responsibly disclose to the threat actor. Many times we do communicate with the threat actor, but Yeah, I get it. Um, that, that's fantastic.

How can people, you know, stay on top of your research? How can they, is there like a website, a, a service? How can they stay on top?

Well, we have our blog at, uh, security slash blog, and we have a very active, uh, Twitter or x, uh, page, active LinkedIn page. So we publish all the time on there, usually every week. Very cool.

So if I had to ask you, what's the one, I'm sorry, what? One Last thing that I really want to mention is that we all also have something called Codex, which is our community project and Codex, it's like the index by coi, and you can, it's like virus total, but for extensions and items, uh, you can search any extension on Codex and get a detailed risk report. Uh, even if you're not currently a customer of coi, you can use that.

It's a research that we giving away just, uh, to help people get, uh, safer. How, how do people get to that? What's, is there a URL something?

How, how do they look guide up? So it's, uh, Dex, that's DEX co security. Got it.

Excellent. That's a cool, that's very cool. Um, we don't have a lot of time.

One last question for you, Eden. What's the, what's the biggest problem you're looking to solve in r and d now that you have this a round money, let's say, over the next six months to a year? Uh, that's a great question.

So, building, uh, a risk engine that can scan, uh, millions of, uh, binary software is something that we are aiming at and that we're actively working on. And this is a challenge. Absolutely.

Very good. Edan again, congratulations on, on the baby. Congratulations on the raise.

Congratulations on coi. As I said, take a moment, maybe smell the roses, enjoy the moment, and, uh, come back and talk to us here soon. More on Text Drunk tv.

Okay. Thank you so much, Adam. Alrighty.

Idan Dart. Dart. Gonna make sure I get this right.

Dart Kaman. Yeah. Okay.

Co-founder CTO at COI Security. That's COI security here on Tech Drunk tv. We're gonna take a break.

We'll be back in a minute. ai leadership series. I'm your host, Mike, er today with Brian Moore, CEO for Voxel 51.

And we're talking about why a lot of these visual AI projects seem to be failing. Brian, welcome to the show. Thanks for having me, Mike.

We see these use cases all the time. I think most commonly people are seeing, uh, vision applications and everything from factory floors to cars, but a lot of the efforts underway seem to be still experimental and people are struggling. What's your assessment on what's going on here?

Yeah, definitely. So first of all, just to generalize it to all of ai, I think that's kind of the state of play in 2025. You know, we've had studies from places like Harvard Business Review sharing that something like 80 to 90% of AI initiatives within enterprises are not yet reaching production.

You could call that a failure. I would just call that kind of expected or par for the course. You know, this is new technology.

There's lots of rapid innovation, there's a lot of excitement to try new things and build proof of concepts. Unsurprisingly, uh, something that you cobble together in a few weeks or even months, is unlikely to meet the needs of the production environment that you need to deploy that into. And that's perhaps, you know, uh, most poignant in something like visual ai where we're talking about deploying, you know, robots or vehicles, uh, or automations that have to act in the physical world and deal with all the different sort of nuances, edge cases, strange scenarios that might crop up.

So yeah, there's definitely a need to invest, you know, kind of the typical 80% of the time to get that last 20% of the way to production. Uh, but the good news is that folks are aware of that, uh, and companies like ourselves are building technology to help assist, uh, you know, practitioners of visual AI address those key, uh, needs, which we can dive deeper into and get that model ready for everything that the production Yeah, the real world, uh, will throw at it. Alright, well, to your point on that maturity curve, where are we when it comes to vision ai?

Because, um, I guess there's some unique challenges there, but what are they? Yeah, so the interesting thing about visual ai, and by visual ai, I mean, anything that has to do with image or video or 3D uh, lidar radar data, um, that's an absolutely immense data source. Something like 90% of all of the bits that go through routers on the internet today are actually visual in nature.

Uh, so the vision AI challenge is at least two orders of magnitude larger than the challenge, uh, of building models that can, for example, process text, right? So it's kind of expected that it'll take some, you know, additional time and effort to get these things, uh, ready for production. Of course, the good news is that with all the investment going into accelerated computing infrastructure, you know, data centers, power r and d, all the things you hear about in the news, uh, those advancements are coming.

The, the promise of being able to feed larger scales of data into these systems, uh, is also coming. And so I would definitely expect to see continued progress on some of the kind of bulletin board vision AI use cases that everyone's familiar with, self-driving cars, humanoid robots. But maybe most, uh, exciting to us are kind of the more incremental advancements, you know, automating specific scenarios like maybe defect detection in manufacturing context, uh, or building purpose-built expert systems that can, for example, you know, uh, detect the fall, uh, of a, a human in a healthcare context, uh, or automatically, you know, process, uh, a camera feed to make a decision about whether a part is, uh, high quality or low quality.

Those kind of things, uh, are much more short term. Uh, and we're seeing those types of technologies get to production, which is very exciting for the, the vision AI field overall. Mm-hmm.

I think everybody's excited about the use cases, but it seems to me they're also running into issues around, well, what does it actually cost to run something in a production environment when you add up all the infrastructure and resources required? So do we need to be smarter about what projects we're gonna pick with an eye towards what's gonna go into production sooner than later? Yeah, I think it's a great call out.

So, you know, uh, one of the exciting thing that's happening in the AI space is the progress of these, uh, so-called foundation models. The large models, you know, the GPTs, uh, coming from hyperscalers and, and those models are, uh, have a broad expertise of knowledge. Uh, however, large models are expensive to run.

Uh, and so what you can expect to see is those models knowledge getting distilled into smaller expert models that are more efficient, uh, at solving, you know, specific tasks. Uh, and so, you know, that's what's actually getting, uh, into production, uh, in vision AI especially, is these distilled models that are purpose-built for specific use cases that can run at a much more cost effective, uh, price point. As you kinda sort that out, who's gonna build those smaller distilled models for the organizations?

Is that some data science team that they hire? Or are there specialist organizations that are emerging who are gonna basically make those things available as a service? How does this kind of manifest?

Yeah, so what we're seeing is that enterprises that, um, are having the most success and visual AI are ones that bring the development of these sort of fine tuned systems in-house. They treat, uh, their AI strategy as a core part of their company's competitive advantage. Uh, and so they wanna bring as much of that development in-house as possible.

That definitely means using off the shelf models, uh, data sets and so forth, uh, to sort of, you know, turbocharge their, their development, uh, that they see their ability to develop, uh, a high quality data set, uh, and model that's an expert in their use case, uh, as being critical to their, uh, company strategy. I'm wondering the skills that are available as it relates to this, and I'm asking the question because, well, we're already having a hard time just finding your everyday run of the mill data scientist genius, and how many of them are actually cognizant of visual AI and, um, what does the pool of talent look like? Yeah, so for context, uh, a little bit about myself.

So I have a PhD in machine learning. Uh, voxel was founded by myself and my co-founder, Jason, actually over 10 years ago, uh, initially doing consulting, uh, in, back then it wasn't called visual ai, but rather computer vision. Uh, and so computer vision as a field's actually been around for quite a long time.

In fact, even Nvidia as a company, uh, got its start and spent many decades focused on computer graphics, the kind of, you know, uh, low level computer vision, uh, algorithms that are necessary to, you know, build graphics engines, video games, so forth, right? So there's a rich history and, and expertise in, in the, in the market that exists in computer vision. Uh, and, you know, so that's the good news.

There's lots of, uh, you know, capability out there. And then, uh, what we're seeing is that whenever there's advancements in sort of overall AI technology, uh, those models, those architectures can be deployed not only for language use cases, but also for vision use cases. And so the visual AI field definitely benefits from all of the advancements that are happening in, you know, large language models.

Uh, as an example, the, the transformer architecture that Google created a couple years ago, uh, was very important both in language but also in vision. Mm-hmm. Of course, it takes a village to kind of build these applications and there's developers involved and, um, data engineers and all kinds of folks, but, um, how do I meld them together?

I think a lot of organizations I talk to are struggling 'cause the data science folks have one culture and the developers have a different culture and they're not quite in sync with each other about how to not just build an app, but maintain it and update it. That's a great point. So, you know, historically what we've seen is that some, uh, companies have decided to kind of create separation between what they call their data team and then their model or product team, which might result in kind of a separation of duties where the data org is responsible for building data infrastructure, maybe gathering data, and then throwing it over the wall to the product teams that then make use of that data kind of a one way street type of modality.

Uh, however, you know, especially when we're talking about model failures, what we're seeing is that the ability to overcome model failures and actually get a system into production, like we were talking about earlier, that comes really from the iteration cycle, being able to understand, okay, I built a data set, I trained the first version of my model, where is it succeeding and where is it failing? And inevitably, if it has a failure, maybe an edge case or a certain type of bias that you've discovered, you need to address that, uh, through better data. It's not sort of a one way street, it's an iterative process, and you're in the best position to make that kind of, uh, action to improve your model, uh, if your data and model teams are working closely together.

So, as an example, uh, our software 51 that we deploy to enterprises, it kind of puts the data at the center of the entire development process of visual ai, allowing data teams and model teams to collaborate together in one place. And when they see a model's performance or lack thereof, that underlying data is always one click away. So with just a click of a button, a user who's trying to, you know, evaluate a model can understand, oh, I see this is why the model's performing poorly.

I can see that there's, uh, something I didn't expect about my data. A bunch of it is low light or low quality maybe, or it's having problems in the self-driving yeast case, uh, you know, understanding, you know, sort of crowded intersections and low light conditions. If I'm able to go gather more examples of those problem areas, that'll be the most effective way to improve that model's performance.

Mm-hmm. How readily accessible is that kind of data? I think, you know, you hear people talking about how all the data that's publicly available aren't even sucked up.

So do we have enough of this visual data to train the models going forward? Yeah, that's a great point. So, so that, that kind of, um, quote, uh, is most often used about large language models, people are saying that, you know, the reason that let's say G PT five had a smaller delta than one might have hoped over GPT-4, is that we've already ingested all of the data that's available on the internet.

And that's definitely not true of visual ai. Uh, like I said before, there's, you know, 90% of all of the bits that go through routers on the internet, our visual in nature, uh, and there's definitely a vast amount of untapped data in visual, uh, it's visual in nature, uh, that is yet to be fed to all of these models. Having said that, uh, one trend that we're seeing with our customers is, you know, kind of by definition where you need to spend all of your time are on the edge cases, uh, or failure modes of a system.

And those are rare, they're hard to acquire. And so companies like let's say Tesla are in a, a good position, uh, where they can, for example, trigger, uh, anytime there's a, a hard braking event in a vehicle, they can capture that scene and feed that data back to headquarters and use that to specifically address that, you know, sort of failure mode. So being able to connect to your development process, to the products that you're putting in the real world, that's a great way to gather more data.

On the other side, we're seeing, you know, synthetic data as an example. You know, we partnered with Nvidia, uh, to make their, um, neural reconstruction, uh, models available to our customers. That's a situation where you can generate a synthetic version of the scene, and then you can play with things like, Hey, I've got this scene.

Uh, what if I swapped out that FedEx truck for a UPS truck? What would happen then? Or what if I took this sunny scene and I wanted to consider how the model would perform if it was instead snowy or rainy?

Uh, you can perform those types of, you know, uh, you know, uh, synthetically generated tunings, uh, you know, uh, from a model standpoint, which obviously gives you the ability to plug those gaps that may be hard to acquire, uh, real data for now, I would say that that's kind of a, you know, uh, up and coming technology, uh, and there's definitely interesting questions to be answered about, you know, how do you evaluate how much real versus synthetic data you need, uh, to build, you know, a production ready model. Uh, but it's definitely something that our customers are excited to, to tap into. It also seems to me the tolerance for being wrong in these apps is a lot less, shall we say, than it is in your typical, um, chat GPT type application where, you know, if the thing hallucinates on some sort of summarization, I'll notice, but, you know, if it's not the end of the world, then I'll shrug.

But it feels like with the visual ones, that those applications are a little more, shall we say, mission critical. Is that fair? Yeah, I think what you're getting out there is that, you know, and this is kind of maybe obvious with hindsight, but the key to getting these systems into production is choosing the right use cases.

And to your point, the right use cases, especially early in the development cycle, are ones where the, the, the system or the use case can tolerate a failure. So yeah, if you're, if the task is to summarize, uh, some content, uh, for a human to take action on, and if it's not quite right, you know, there's a human in the loop already, and so maybe it's okay, right? Uh, on the other end of the spectrum, you could see something like a self-driving car where, you know, it makes kind of intuitive sense that it needs to be, at least in order of magnitude safer than a human driver in order for us to kind of accept, uh, any sort of failures that might happen.

Uh, but the good news, like I was saying before, is that in addition to the sort of, you know, bulletin board use cases for visual ai like fully self-driving or fully humanoid robots, there's a lot of, uh, smaller more sort of focused tasks like detecting defects or, you know, uh, automatically processing, you know, user imagery for insurance claims where there's a lot of summarization and sort of constrained environment tasks that can reach production level and will, you know, uh, add lots of value to us while we continue to push towards those bulletin board use cases. Kind of similar to how everyone spends some fraction of their time talking about what a GI will look like when in reality, uh, those use cases like, you know, automating customer calls, uh, in service centers or, you know, summarizing, uh, knowledge work, uh, for enterprise. Those are the real value creation in the short term.

And to your point about that, you know, everybody talks about, well, who moved my cheese and am I gonna get laid off? But when you look at those use cases you're talking about, some of them are things that we probably would never have done in the first place, and many more of them are things that well, nobody really enjoys doing in the first place, and we typically don't do it all that well. So is that part of the thinking about where to make use of something like vision ai?

Definitely. Right. And, and just to put a point on maybe a macro trend that's happening right now and how it's impacting the visual AI space, uh, you know, we're talking a lot these days about onshoring manufacturing for various, you know, political reasons, uh, you know, so forth, which we won't go into here.

But, you know, there's this sense that, well, uh, there's, there's certain tasks, sort of menial tasks in factories and so forth that may be, you know, some fraction of US workers aren't interested in doing, or it doesn't make sense to do, uh, at sort of like human level price points. Perfect use case for vision ai, right? We can come in and invest, uh, as we're onshoring manufacturing in building automation, uh, and they, you know, building out factories that'll put us in a, you know, a competitive advantage compared to, you know, even our, uh, offshore, uh, competition there.

Well, let me ask you this then. Um, is this really a separate discipline in the sense that there will be a separate ecosystem for it, or, you know, are you at all concerned that the open ais of the world and everybody else who's in that space is just gonna, you know, just add this to their portfolio of services? Yeah, so that, that's, um, kind of what I was getting at before when I was making a distinction between, you know, what a foundation model, uh, can do, uh, versus, you know, what's actually required to get that particular use case fully automated and in production.

Uh, you know, there's, there's interesting conversations happening right now. Let's take on the language side for a second around, you know, hey, if, uh, if it is true that these large language models have quote unquote PhD level knowledge in all fields, then why do I need to go to a healthcare provider? Why can't I just go to chat GPT and have it solve, you know, provide my diagnoses and provide a plan of action?

While there's not, it's not just as simple as providing the knowledge. There's expectations around, you know, quality of care and certification and so forth that come out. And for that reason, it's not gonna make sense for a single company, certainly in the short term, to provide sort of expert level guaranteed certified services and all these use cases.

Uh, and I would expect the same thing to happen in vision ai. You know, it'll make sense for vertical specific companies that deeply understand use cases and customer needs and so forth to take a technology, a general purpose technology off the shelf, and build a solution for a specific vertical. Uh, not to mention the, the, the point I mentioned earlier around how it's not cost effective to take that general purpose model and plug it in directly.

That may be sufficient to build a proof of concept to show that something can be done. But ultimately to drive margins up, cost down, you're gonna have to invest in building more expert, you know, fine tuned systems. And that clearly has to be done by a, you know, an entity that's focused on that one vertical and ready to make that commitment.

All right, well, folks, you heard it here. There's a lot of things to be excited about in the AI era, but maybe all the cool kids are starting to hang out in the visual AI table because that's where the new and interesting applications are gonna be. Brian, thanks for being on the show.

Thanks, Mike. All right. And thank you all for watching the latest episode of The Techstrong that AI Leadership series.

You can find this episode and others on our website. We invite you to check them all out. Until then, we'll see you next time.

What's that cracking noise You hear? It could be DevSecOps Foundations, you're watching Textron Gang. Hi everyone, happy Monday.

I hope you've had a great weekend. I'm Alan Shimel, and this is Textron Gang. Um, man, I don't know, my weekend was much too short.

I was, I was looking forward to doing so many things and I, I had a honey do list instead, so it was a bit of a honey do weekend for me, and I don't mean the green melon. Um, I'm almost happy to be here talking tech strong and tech with y'all. Let me introduce you to my compadres on the, uh, panel this morning, wearing his Indiana Jones hat Jack Poller guitar man, Mitch Ashley, and, uh, well, I don't know if he's still a Yankee fan, but he's chief content officer here, Mike Ard.

Gentlemen, welcome to Tech Strung Gang. Happy Monday to you, Mike. com and other sites, other tech drunk sites, non-tech drunk sites, software, supply chain security, DevSecOps.

You know, it's a, it's, it's enough to shake to shake your confidence, you know, but it's the security pros dilemma. We never seem, you know, we just never win, is always something else out there. Let me, yeah, let me, let me go through the list of what we're talking about though.

I mean, Shai Ude is attacking the CrowdStrike environments and other software supply chains. Andro is reporting that people are abusing claw the launch attacks. White Cobra is a criminal group that's targeting, uh, visual studio extensions.

And then we see Oasis Security made a note of the fact that there's a lot of weaknesses in the cursor AI coding tool. Again, exploited and checks marks did the same thing and said, Hey, you know what? You can lie to these AI agents and tell 'em anything you want 'em to know, and then they'll give you access and do all kinds of weird things in your code.

Mitch, none of this sounds good. And, uh, Alan has a piece talking about how the foundations of our DevSecOps world are starting to crack. Do you agree?

And what are we supposed to be doing about all this? Well, I see many futures, many possible futures about this may take place. So, and actually, there, there are, I mean, I think these are real examples, right?

Of how AI can be used or is being used in an offensive standpoint. And, and I don't know if you mentioned in, in the case of even anthropic and generating code, it's about extortion. It's about getting, you know, money from people.

It's about how we craft now software that can go after not just large amounts of, uh, you know, individuals in a particular n and m kind of fashion, but also it can be done on a personalized individual basis. So I think these are all good examples. I hope it is sort of heats up the, the, the water, if you will, and ramps up the interest on getting more secure code generated out of software.

But also more importantly is, is making sure that there are security guardrails that we can contain. I did a piece on is regulation gonna save us? No regulation's not gonna save us.

I think the market has to respond to creating and secure guardrails that we can control and manage. But right now we're sort of seeing kind of if and every, any and everything that can happen with, uh, security around ai. Mm-hmm.

Jack, let me ask you, d does it feel like we're being a little reckless from the security standpoint? It seems like we're just letting these developers turn loose, do whatever the hell they damn well want, and it shows up in a production environment. And then guys like you are asked to clean it all up.

Why should today be different than any other day? I was just gonna say that New boss, just like the old Boss. Exactly, Yes.

We're being reckless. I mean, there is a lack of fundamental cybersecurity hygiene, you know, uh, there's the AI issue, and then you look at something like the NPM attack, the root cause of that attack isn't supply chain, it's actually poor password, it's passwords, right? You had people who were phished that then got access that enabled the bad guys to get access to the code base and then propagated from there.

So why are we still doing password based authentication? We have passwordless authentication, we have MFA, why are people leaving the front door unlocked? Mm-hmm.

You know, and then, all right, So go ahead. I'm sorry, Jack. Oh, I was just gonna say, and the AI doesn't make, it doesn't make necessarily change the attack path.

What it does is just enables people to act faster on the attack path, just like AI does for everything else. It's an acceleration function, Okay? I think it's an acceleration function, and it's, it's a breath.

You can do everything all at once, anything, everything everywhere, all at once, that kind of thing. That's, that's one of the big differences, is not Just it, it's a force multiplier for the bad guys, unfortunately, at this point. A good way to say it.

But, but let me back up a little bit. First of all, I gotta give credit to whoever comes up with these names. I mean, I saw shy Ude, it had me shy.

I was, I was ready to go blue eye and blue and blue eyes, and you know, I, my mind was running away with me, right? You're ready to start the galactic war. What a great name for a worm.

Why didn't I think of this? Where are names been all this time, Right? White Cobra.

What was the movie where there was that guy, fat Cobra or something was his name, and he, it, it was one of these spoofs of like a James Bond thing, and the guy goes out and it's, the guy's name is Fat Cobra, but he, he's like a ruthless killer or whatever. I mean, what, who comes up with the names for these things? I'd love to know, right?

Spider, what's the other thing? Spider this together. Spider.

Spider. Yeah. Yeah.

These are great names. This like outta Hollywood, but, but seriously for a second, all our chickens are coming home to roost. We've known, we've known about our software, supply chain issues, security issues.

For some reason we thought putting SBOs and making SBOs mandatory would somehow fix this. No, it doesn't fix it. No, it doesn't fix it, number one.

Number two, we've known AI is going to be a force multiplier that the bad guys, we're going to use ai. And that if we're gonna rely on these models that are taken from the bad code, we've been, been using all these years to generate new code, and we're surprised that the code it generates is insecure. It doesn't stop us from using it.

60% of the code out there has, has AI's fingerprints on it. And, and we sit here and say, gee, I didn't see that comment. Right?

This is, this is, you reap what you sow, right? Mm-hmm. This is, you reap what you sow.

We, this is, this is what happens here to us. And you know, and look, Jack, Mitch, me, Mikey, we've all been around the block. We've all been in involved in the security game a long time.

We're always one step away from the ultimate calamity, but somehow, somehow we live on to the next episode, right? We, we, we squeeze through, you know, it doesn't turn out as bad as we, we it could be. So let me, let me challenge you on that, Alan, because I think we're in a, in a situation of fighting the next war with the last war's weapons.

Oh, no doubt. And we're, we're, we're thinking about with fracking and scanning and mm-hmm. Doing all these things that are passive after the fact.

SBOs, you name it. All, all good things, good practice. It's a security.

Imagine though line It, it, it is actually, I'm Gonna write in, I'm gonna write an article about that Security management. You gotta, you gotta be over, you gotta be over 45 to know what that means. But go Ahead.

The students of history, students of history know what that is. Are there any students of history left? But we're, we're, we're, the new war is, you know, this is, this is battlefield.

You know, we're, we're the, we're the, uh, red coats or something. Is it, it can't, we're not in a world of, it's about creating defenses. You have to fight fire with fire.

You have to fight AI with ai. And I'm not just saying that because AI is the answer to everything, but you know what, if, if they're doing calculus and you're doing algebra, you better damn well learn calculus. Matter of fact, you better be really good at calculus.

And that's what we've gotta do. And so, you know, my wake up call to everybody, software developers, security professionals, all of this is get on the fricking AI bandwagon. Not just because it's the thing to do and it's popular.

And, you know, chatbots are that cool. Is that, is that's the new war. That's the new environment.

And if you ain't good at it, you know you're gonna get rolled over by the, the digital tanks of Tron taking care of you, taking you out. Right? I mean, to Mitch's point, I mean, I would love to be able to say to every developer in the world that thou shall have two factor authentication.

And you, you're being, you know, maybe subject to some penalties. 'cause you don't, but I don't think that's realistic. I do think when we need something that detects an anomaly, like instantaneously and then applies policies instantaneously.

'cause the amount of time for which havoc is wrecked is now measured in seconds. Right? And this is where the whole thing has moved to It's machine versus machine.

And we don't have machines that can fight the fight we're gonna lose for sure. Well, I think the other part of it is we need to prioritize security hire. Right now, a lot of what we do is productivity.

We focus on productivity, both on output of developers and on making their lives easier. So you don't have MFA because it's friction. You don't, uh, you have automatic updates of your NPM packages because it's easy and quick, and you can press a button and you get an update, but that bypasses a check of what exactly are you pulling in, in your packages, which is how this worm propagates, right?

It automatically gets updated and it often goes to the next package. So if we look at our environments that we're building and the environment that we used to build things, we should be saying, yes, we've made it. So it's super simple.

It's PhD, it's push here, dummy, right? Everything's a push button. You get an update, you get this, you get that, and let's sort of maybe dial that back and say, how do we layer in a little bit more checks to prevent bad stuff coming in, in one way or another?

Right? And I, I used to talk about this in terms of we need to compensate, you know, we compensate developers on feature functionality and not on security. The problem is it's open source.

So how do you, you're not compensating them, they're doing this for free. They're volunteers, so you can't penalize them and you can't compensate them for good or bad security. So we have to think about it in a different way.

You Can though, but I think you can. I think you can. Okay.

You know, here's, here's my take on the software supply chain issue. The fact of the matter is, most developers are not conjuring up this open source software outta thin air. They're downloading it from a repo for multiple repos, whether it's GitHub or, or NPM or Artifactory or Maven or, or whatever, right?

Wherever they're grabbing this stuff from. The point where they downloaded from the repo, in my mind, has always been a choke point. It's always been the point where you could say, wait a second, is this piece, is this script?

Is this component, is this container secure? Is it the latest version? Has it been scanned from malware?

Right? Is it, and, and, and some, some things are still going to get by, right? Because we don't know that it had the vulnerability.

It's a zero day or whatever. But so much of our aggravation here, of our security stuff could be solved at that border of the repo, uh, from the downloading from the repo. Now, I know certain repo security companies have talked about sort of a repo firewall for forever, that if you're gonna download their software before you get that software in, it's gonna be x-rayed, it's gonna be scanned, it's gonna be tested.

But we don't, but developers don't do it. We don't do it. We don't as a rule do it.

We, that's the hygiene we need here. We need Well, and That's, that's my point exactly, is that's the, the, that's the slowing down of the development cycle. Don't do it automatically.

Have it go through some steps to check it in some way or another. But can, why can't we make it automatic Jack? It's easy enough.

It's easy Enough. Well, that's, that's my point, Alan, is, is it should never be a next step is when security happens. It should be never be a, and then we scan it and then we go, and then we put it through a firewall, and then we do this right?

In an age of AI code to, to your earlier point, Mike, the code that we're getting out of, out of LMS today is all the insecure code that we put into it, right? We're getting back. We're, we're reaping what we sowed.

But so to solve that, you're not gonna, you're not gonna improve it by just adding better steps at the end, after we've generated code, generate code that's created by ai, should already be secure by agents, ai, LLMs, all kinds of things can make it secure. Whether it got it through a good firewall or it downloaded from the, you know, crappiest, whatever. That software should be secured before it ever is presented to a repo or to a human or whatever.

Those steps can happen before we touch it, rather than, here you go, Mitch, here's some more JavaScript code. Try that, that now has some new things in it that, whoops, you know, I do, I need to run that through a scanner every time I test it on my system so that, you know, I don't get compromised. Let's get real.

You're not gonna create secure software by bolting on more steps just faster. You've gotta build it in. You've, instead of shift left, you gotta shift in security.

You Know what this, this reminds me of like the Apple walled garden approach to apps versus the Google Play. Not, not Google Play today, but the Google Play store earlier on mm-hmm. Where anybody could basically upload an app to Google and there was a lot of malware infested apps in the Google Play store where Apple, yeah, they were a pain in the butt, but they did it.

They did. I think they did. Anyway, check your apps before it was accepted into the Apple store to make sure it wasn't a security risk.

And so we had much less security incidents from downloading apps. And I'm not saying it 'cause I'm a fanboy, but we had, you know, they did do a better job of, of testing them on the way in instead of just on the way out. It maybe, maybe that, maybe that's the thing to do from, So, Mitch, let me ask you the question here.

Yeah. We keep talking about best practices. You should do the right thing.

We should have some empathy. We should all lock arms and do the right thing, and yet we don't. So at what point are we gonna get to where, you know, it's gonna take, you know, to use a phrase that's popular these days, we're gonna have to lock a couple of people up to make the point.

Well, it could be locked a couple people up. I think the, the real answer is enterprising entrepreneurial companies, people that solve this to make money, that's the way to solve it. Because that's the way We'll touch on.

Yeah. Let me just say Regulation and penalties and all that, that's, that's nice, but that Ain't get stopped. And, and guys, in all honesty, Mike, I gotta take issue with what you said.

'cause of the world we live in, the country we live in today, okay? We don't lock people up because they made a mistake with computer code. People get locked up when they commit crimes, intentional crimes, right?

Let's, let's be clear about, I'd like, in all honesty, I'm, I'm not playing that game no more. We're not locking people up for, for negligence or computer stuff. I would Point at, I would point at, if you drive your car and you know that the brakes are not working and you smash into something, you will go to jail.

So that, so that, that is, well, not if you smash into something, you don't go to jail necessarily. If you hurt something or kill someone, you do. Right?

But, and they, they call that negligent homicide. So, So how, but, but Legally there's a whole hierarchy there. I'm sorry Jack, but I, But I, I think, I think you have a, both you and Mitch have a valid point where Apple created a Walt garden and Apple was responsible for the trust.

So you trusted Apple, apple enforced it, and you went from there. Google's approach today is there's a Google walled garden. But if you want to install something from somewhere else outside of the Google Trust in search circle of trust, you can make the decision.

We've told you we haven't vetted this, right? Personal responsibility. Personal responsibility right now, software development in general is all on the personal responsibility.

It's up to you and there's no circle of trust for you to go to. And enterprising company or entrepreneurs could go and create that and say, we'll, take the open source repositories, we'll validate it, and you come to our repository and you pay us to come to our repository because we've done this extra level of work to create this circle of trust. That's one possible solution to this problem.

It's not. And just with everything else in security, it is maybe necessary, but not sufficient. It doesn't solve the entire problem, but it goes a good step forward.

It, it's a, it is a step forward. I will tell you my last thing, and I'll put this back on the free market. If people are selling you insecure software, don't renew the license.

Boycott insecure software boycott, cancel your subscription to insecure software's. That's the mantra. Hey, we gotta, we gotta take a break here.

I'm gonna leave you with one thing on this. Fear is the mind killer you're watching Text Textron gang, Discover Textron Group, the epicenter of tech innovation. We are your go-to for reaching IT, leaders and practitioners worldwide.

Our secret impactful content that sparks awareness, engagement, and top quality leads with us. You'll access editorial websites, streaming videos, virtual events, custom content analyst research, and more. Join our satisfied clients.

Let's revolutionize your tech journey. Contact us today and tell your story to the world in the most powerful way with Techron Group. Hey folks, we're back and there's a war on and a place that we can't see it.

It's up in space and involves all these GPS satellites and most of this issue seems to be emanating out of the conflict in Ukraine. But Jack, you have an article about this over at Security Boulevard. Educate us what's going on here.

Well, to no surprise to if, unless you've been, you know, sleeping for the last 5, 6, 7, 10, 10 years, there's a war between Russia and Ukraine. And no surprise, Russia is playing dirty tricks in jamming GPS and Russia being Russia, they're sort of a little bit or maybe a lot indiscriminate. So basically just saying, we don't care.

We're just gonna blanket the entire region with jamming signals and corrupting the, in the GPS signals, uh, civilian GPS is insecure by design and is also used everywhere. And not only by your cars and your phones, but much more critically by airplanes to navigate. And in fact, you can use GPS to pretty much land a commercial jetliner today, except when GPS signals are jams.

Don't third use it. We wish. Yeah.

Um, but this essentially goes back to sort of a continuation of the discussion we had earlier, which is, if you don't design security in from the beginning, you're going to have problems. In this case, we have a very big GPS problem where the jamming extends out from the Ukraine to cover all of the Baltics. And in fact, even Sweden is reporting an increase in loss of GPS signals in airplanes and other things.

And it's a very huge problem. I mean, this is life critical, mission critical stuff. When a plane is trying to land and it doesn't have GPS anymore, it potentially lands in the field and crashes rather than lands on the runway 500 feet to the right or to the left.

Right? Yeah. And there, there are other uses of GPS that there are pretty mission critical too, right?

Yeah. It's all around emergency servers, As you name It. Everything, everything.

But, but let me Go ahead. I'm sorry. You go, Jack.

No, Well, I was just gonna say my, I I'm looking at both GPS, but I take this as look at the broader picture, right? Which is really going back to a little bit of the discussion in the, in the, a block of thinking about security, security at day one. Security should never be a bolt-on afterthought, right?

It's, we have to design security into anything and everything we do. And whether it's GPS or how do you print, right? If you're developing software, hardware, it products, you need to think about security at the very beginning.

So I, I think this is more about something else we spoke about before, which is about fighting this year's, this year's war with last year's Tactics. Absolutely. And I, and I think, look, the loss of life in this war is horrendous.

I, I, I, I feel bad for the Ukrainians as well as the Russians and all their proxies who have lost lives in this war. But when we look at the history of this war, much like, I don't know how many of you're a fan of history. I am, I'm a history major, you know, they say that the Franco War in Spain, the Spanish Civil War in the thirties and twenties, was a precursor for a lot of the weaponry that was used in World War ii, especially by Germany and Italy, right?

They were testing, it was a great test lab. And for tactics and technologies that found its way into World War ii, I think we're gonna see the same thing here. I hope we don't have another World War ii, but we're gonna see the same thing here in the Russia, Ukraine War.

Things like our over-reliance on technology, which is not secure like GPS and the ability to take that off as a chip on the battlefield is, is paramount. Things like cheap drones, unmanned drones, autonomous drones make having billion dollar air aircraft obsolete with people driving them, right? There's been a lot of new tactics, new new technologies that are being deployed here on both sides that I think the folks in war colleges are gonna be digesting for years in terms of how do you fight the next war?

Maybe, maybe, you know, look, here in the US we pride ourselves. We make the best damn weapons in the world, don't we? No one makes a plane a fight, a plane like us.

No one makes a tank like us. Damn. No one touches our missiles.

But have we have, we made an overreliance on, on vulnerable technology in our defense or war fighting ability that leaves us vulnerable to very kinda low level jamming like this, right? And we sleep under the security of that blanket is quote, semi quote Jack Nickelson in a few good men, right? We sleep under the security of that technology and is a problem To, to, to use Minter's phrase.

We are ironically, literally fighting this war with both next year's weapons and last centuries weapons. So Russia has chosen to degrade the Western technology capability by jamming GPS, which affects the high marsh rockets. It affects drones.

GPS guided drones, Ukraine is fighting back with. And, and sort of the, the impetus for looking at this is new technology, which is using laser communications to communicate to the drones, right? Also, fiber optics.

They had literally, a drone will have a 10 mile spool of incredibly thin fiber optics that it, that it spools out as it goes to attack the Russians from the Ukrainian side. And there are areas in the fields that this battle is bought that it now look, the, the, that it now looks like a spider's web of all these trails of thin fiber optic cables lying on the ground or caught in trees. Russia, on the other hand, is using artillery pieces and tanks that date back from World War ii, right?

1940 1950s tanks and artillery, because they aren't technolo, they can't be interrupted by GPS jamming. It's, it's a ballistic shell. You fire it and it goes where he wants to go, and that's it.

And, you know, and, and, But they're doing that outta necessity, Jack. 'cause there are other stuff got blown up early on. I, I agree with you.

I'm not, I'm not saying, well, so, you know, it's, it is what it is. Now the other part of this that gets interesting is the loss of life, which I think is abhorrent. But talking about history, Russia has always had a historical basis of treating its own civilian population as cannon fodder.

That's actually where the term sort of comes from, is we have 300 million people in the country, and we don't mind if we kill 10 or 20 million of 'em in order to get what we want. It's okay. They're just peasants in the middle of, you know, the, the, the Siberian Peninsula.

Why? What do we care? Right?

And so the more they, the more people they can throw at the problem, they're happy. And the western world doesn't wanna fight a war that way. We don't like the death and the indiscriminate killing.

And so we're trying, we're always trying to find other means, technological rather than human to solve this problem. Agreed. It's a shame.

It is. It is. So do you guys think, though, ultimately, and, you know, coming back to this GPS thing, let's say that there was some sort of conflict involving another country to another country, is the GPS thing the first thing that's gonna go, because that's the communication.

Yeah. No, I, I, I think that is the new, you know, you, you wanna black out your enemy like I do. I think the Israelis did this in attacking Iran.

Absolutely. Absolutely. You know what, I, you know what I'm gonna go buy, I'm gonna go buy some R McNally maps, you know, the kind that used to stick in your pocket.

And so you, the Question is, once you open it, can you fold it back up, Mike? That's always the problem with those maps. Um, alright, well heres the 3D book kind.

It's okay. That's, let's take a break. We're gonna come back on our C block here.

Mike took a little field field trip to Mongo. No, not the Congo, Mongo MongoDB. We're coming back at you.

com is the leading resource for news analysis and education on challenges facing the cybersecurity industry. com covers all aspects of cybersecurity, including data security, DevSecOps, cloud security, application security, network security, security threats, and more. com has the largest selection of security content featuring breaking news, blog posts, podcasts, and more.

com to learn more. com. Home of security bloggers network.

All right, to Alan's point, yes, I did go visit a friends at MongoDB. They had a developer event last week, and it was interesting. They were talking about three things.

The first thing wouldn't surprise you, they're gonna embed search and vector search into the document database, and that makes it easier to manage, and you can have all this stuff in one place. The second thing though, they're also noting that you can now use AI agents to reverse engineer other applications faster. So you can do these, uh, application modernization projects that might have taken a year and a half can now be done in maybe a more reasonable four to five month window.

We'll see how that plays out. But here was the most interesting thing in my mind. They were also talking about this whole notion of context engineering.

And they're talking about the fact that one of the reasons that a lot of these AI projects are failing is because we're just slamming data into these things, and there's not enough context to drive a workflow. Their argument is, is we need smaller chunks of data thrown into these Z LMS and other, uh, processing engines that are, uh, more efficient. So that I'm not doing these massive amounts of processing, but I'm doing it in a way where there's context remains as I kind of do what they call a nested doll approach as I'm processing things.

So these smaller, uh, chunks as technical term, I know, um, we can maintain the relationship between these things, AKA knowing the context. And so what they're really saying is databases are gonna be one of those shovels that drive AI and the, the goal rush associated with it, but we need a better shovel. So Mitch, I think other folks are starting to talk about similar concepts and ideas, but do we need a different way about thinking about databases to make this AI thing really work?

I think that's why, you know, you, you hear the term context engineering, right? Beyond just more prompt engineering. And think all of us who've worked with AI know that instructions are just part of the answer, right?

It's now here's, here's the context in which I want you to perform this task or analyze this data or search this information. So if we review, if we review the data that AI has access to, it's just a giant repository of everything in the world. Kind of like we think about how the model's been trained, right?

I've been trained on the entire internet. Well, little, little more nuanced than that, right? It's, it's contextualized in a much different way.

We have to contextualize how we use ai. So a bot or a prompt, or whatever agent, whatever it is, needs more, more understanding of what you're trying to do. And that's why you see even in the consumer products, right, that we do with chat, GPT has had, uh, memory for quite a while and now you see Gemini and also Claude, uh, coming out with memory, meaning it keeps some memories about your past, uh, actions, things that you prefer.

Some of 'em are intentionally saved, some of 'em just your prompts, it's a combination of things. But that's all to add context to what you're doing. So I think what Mongo is saying is, look, we can be the repository for your data.

Anybody can do that. How do you do this in a much more intelligent kind of context to wear way? So that way, and, and also use structure when you need to use structure.

So for example, they talked about J-S-O-N-J-S-O is is actually a really good thing for AI because it, it takes what we like to, to type in as natural language and it puts a structure around it, makes you really clear what it is, um, sort of a lexicon, um, some definition around it. And it helps it, it gives more structure to what it's doing. So I think a lot of what we're doing today is experimenting with prompting, and we're gonna learn much more about how to do better contexting around prompting.

Can I, can I, I just wanna make sure I got this right. A company that sells small databases for unstructured data says what we need to make our AI models better are smaller databases with unstructured data, Is that, that are processed, that are processed more efficiently. So they're well that be processed more efficiently if you use a smaller database with unstructured data.

And by the way, we sell that, so, so, Well, It doesn't have to be a smaller database, right? Document databases can get pretty large these days. Yeah.

And I can string those together in a way that makes them, you know, feel like one logical entity. So I don't think size is what matters here, But yeah, Alan, you make, you make It sound, I've spoken like a troop, like a true, I think there are people would who would disagree with you, but I'll leave it that wishful thinking in that. Jack, go ahead.

I, I think it's all about the Effort. And someone once said, prove me wrong, but go ahead, Jack Said it Conversation back, Email panel. Of course, Jack, take us outta here.

Come on. You make it sound like a conspiracy theory, Alan, but I believe that there, there really is some validity what they're saying. So I'm, as you can tell from the hotel background, I'm on the west coast today and this week for a, uh, securities field day.

And last week I attended a teleport event, and one of the presenters there told a little story, which I think is applicable here. And it's basically, you know, I used to go, you know, visit a town and you'd say, okay, I wanna find a place to eat. So you go and you hit Yelp or something, you say, go give me a list of my favorite types of restaurants.

And you do a manual search and you hit Google and you look at reviews and you say, okay, now Google is this restaurant open right now. And eventually make a decision. That's how we all used to do it.

It was very manual intensive process. Now you basically go to your little app, AI app on your phone and just say, yeah, hey, I'm interested in a restaurant. Find me a restaurant.

The app has context and history that says, I know what type of food you've chosen in the past, and I know where you're located, and I know what time of day it is. And I can take all of that information, that additional context to the query without you having to do that. And it's those little pieces of information that you need to have give to the ai.

The problem today is the AI databases are designed for a specific task. And that's the big stuff, right? That's rag retrieval, augmented generation, where you have a, a very big set of stuff that you're looking at all of the data that's in a company, right?

So you're, you know, gigabytes and terabytes of data. And here what we need to do is track little tiny pieces of information that just, uh, make the prompt and the AI work better. So I think they have a point, maybe whether they're the right people for that.

That's a different question. Maybe. I think it's prevagen for AI agents, if you think about it.

So Like, Come on. You know, and I remember when you used your photo guide, you didn't have Google to make recommendations and all those things, but, or you just stopped it was ever was open and you took your chances and you found new places like that. It was half the adventure.

But let me, let me come back to our friends at Mongo, you know, they're a little late to the party because I think a lot of the people I know who were looking at operationalizing AI and using ai, you know, had the idea of creating small language modules or vector database kind of information, exactly what we're talking about, to give it more context, to give it more, um, subject matter expertise on a given narrow, uh, field that you won't get when you just use that big LLM. And I think for a long time, Mongo was sort of deaf, dumb and blind to quote Pete Town said, right, um, to this use case. And now here we are, and Mitch, you know this, you're shaking your head.

You were at that AI operationalizing AI workshop we did those two years ago, Then you quoted Tommy, so you got me on both. Well, you, you, you got you on that one. But now, two years later, all of a sudden they're saying, oh yeah, we should use that vector database or vector search.

Well, let, let, let, let's be fair, they had us, they had a vector database of their own or a vector search capability. They just didn't embed it in the database. They had it as kind of an ancillary kind of thing.

It's not like they've been blind to the use case. And a lot of folks do use Mongo because of what Mitch was pointing out. It's JSON, the js OI don't have a whole lot of conversion.

Most of those AI models are speaking JSO already. So to that end, I think, you know, as far as the convergences, yeah, that could have been done faster. Mm-hmm.

But I don't think they've been blind to the use case. They're, you know, if I go look at a lot of these AI projects, you're gonna find MongoDB in there. I think what's curious to about What we A I'm sorry, Mitch.

Now what, what's curious to me, uh, Mike, and I'm not sure if I, if I caught enough to really put all the pieces together, but what Mongo also said is that repositioning databases from being a data store that you access data from is to, as an AI platform for workflow and for agents that combines like, you know, native vector search with semantic retrieval and, you know, lots of, you know, kind of nice language around, you know, technical terms, queryable, uh, queryable encryption, things like that. I wasn't quite sure that I got the, the transition from data to orchestration or data to AI platform. Um, did, did, did that jive with you at all?

Can you fill in the gaps for me at all? You, I think that, you know, they will have to decide to what degree they wanna orchestrate those AI agents versus just kind of expose data more dynamically to them. Mm-hmm.

And there's a level of orchestration that's required to do that. But I imagine that they will probably expose their database to the orchestration layer APIs to provide more of that coordination, because there'll be the agent and then there'll be the data that has to be accessed, and the data will have to be in something that feels like persistent memory. And I think that they're saying that, you know, you're gonna cache the crap out of a, uh, MongoDB database to provide that memory.

And that they're gonna provide, I guess I would call it the metadata around the memory so that they has that kind of awareness in the context. So be the data in whatever forms, memory, context, data source, all of those forms that, that would fuel an orchestration layer or workflows that agents do. Is that what you're saying?

Yeah, I believe that that's what they're in as well. I don't think they want be the orchestration layer for the agents themselves. Okay.

Right. That confused me. Okay.

Thanks that, that helps a lot. But yeah, I'll it you in my next paper. Yeah.

But of course, you know, of course next week I'll turn around and they'll buy some orchestration framework and I'll Yeah, true. Yeah. Interesting.

Okay, Thanks. The, and all, all kidding aside, you know, it is quite the, a a bit of an unknown story or an under-reported story exactly how big a powerhouse Mongo is. MongoDB is in the ai Yeah.

In the AI space. If you, if you're a certain, if you're a certain database company owner who's recently seen his fortune go up by 34% because he's trading GPUs, may, maybe that makes Mongo a target for some of that money. Trump change.

I think that there, all the database companies are gonna have to address this issue, and everything's gonna have to be in these kind of smaller, dynamically processed chunks. I don't, so I'm not quite clear that one of them is gonna be, uh, better than the other. I just think that the way we consume that data needs to be changed and we need to think that through, because that's what's holding up a lot of these projects in the enterprise, right?

Yeah. They're just not, I do think Mongo's the new MySQL, it is sort of the new default when you're gonna move to something beyond. And, you know, what happened to MySQL?

Well, I'm, I wasn't trying to foretell that, but maybe what, um, but it, it is widely used and not just for ai, but it's used in a lot of cases. You see sort of the default on a lot of open source and a lot of different projects is you needed database. It's really the Mongos or Mon Mongo, excuse me.

And that's because developers are picking the database. And developers don't really like relational databases. They're big, hard, and clunky and hard to manage and document databases are easier for them to deal with.

Whether the IT team likes that when they get handed the app is a whole nother conversation. Hmm. Mm-hmm.

Excellent. Gentlemen, I think we gotta call it a, a wrap on this, Jack. Enjoy the West Coast.

Uh, I I will be joining, uh, the tech field day, I believe on the 24th, isn't it? I'm, I'm not gonna be there in person. I I'm gonna be remote, but, uh, I'll see you on there.

It should be fun. Yep. Um, I won't be on tomorrow's show.

I'll be observing the Jewish holiday of Rosh Hashanah, or is when I, my grandmother taught me Rosh, but, uh, you know, it is the Jewish, uh, new Year and start of the Holly High Holy Days. And I'll, I'll be off for that. But Mike, you'll, you'll drive the ship.

I'll be here. And I'm, I assume you'll be dreaming up new fancy malware names. 'cause you know, now that you've got a new hobby, I'm gonna, I'm, yeah, I'm going to, I'm gonna think about those.

Some good. Cool. Some ones.

Hey Mike, why, why are the cats away? Just to plant a seed of an idea, let's come up with some intro theme music like they do at the sporting events. And if Jack's on, on our next show, I wanna make sure he gets Indiana Jones Like they do at Yankee Stadium.

When you come up, right? They bring in Mariano. Exactly.

You, it's funny. So I was at Swamp Up, uh, last week or the week before, and they're, they, they had a gal or dinner and they had a celloist as the, as the town, you know, as the entertainment. I'm saying to myself, a celloist, my God, this is gonna be boring.

What am I doing here? And turns out she's a very famous woman down in la She does a lot of movie soundtracks, Mitch, including Dune. And she played the theme from Dune on her cello, accompanied by some musicians.

And it was amazing. Amazing. Wow.

That sounds Awesome. It was really cool. Really cool.

I'll find out about her. Alright. Hey, enjoy the rest of your day.

Enjoy Tech drunk TV immediately following this. And by the way, on that tech field day, you'll be able to watch it live too, right here on Tech Drunk tv. I'm Alan Shimmel, we're out.

Hey everyone. We're back here at our Swamp Up 2025 coverage, a beautiful Napa Valley. We, we haven't started drinking the wine yet, so don't worry it's early.

But let me introduce you to our next, uh, guest here on Tech Drunk TV to my immediate left, uh, kind of a VIP guest here. I hope I get his name right. 'cause he's the VIP Tar Tarik Shock.

Perfect. Thank you. It's great to be here.

Tarek. Welcome. Tarek is the CEO of Sonar.

And if you watch Tech Drunk TV or read any of our sites in Sonar and Sonar Source and everything, he is a pretty well known brand and company we cover. So thank you. But I don't think I've had the pleasure of interviewing Tarek before.

To my far left, I've had the pleasure of interviewing him many times. My friend, gal Marter of of Jfr. Gentlemen, welcome to Text Drunk tv.

Thank You. Thank you. Great to beer.

So Tark, you're the VIP guest. We're gonna let you go first. You were up on the keynote with, with Shlomi this morning, along with, uh, folks from Nvidia and Service Now Service.

Yeah. You know, I wrote a little article that's, I think it's up already, but, um, you know, my mom always told me, show me your friends. I'll show you who you are.

Right. And, uh, or Lived by. Yeah.

Yes. And, and so that was a great grouping of, of companies up there as the CEO of Sonar. Let's start there.

Talk to us about the relationship with Jfr. How are you working together? How are you working with Nvidia and some of these other companies as well, and why that's important for our listeners and readers and watchers here.

Well, um, again, thanks for having, uh, having us on. I think, you know, we were super, um, fortunate, very grateful, uh, to Shlomi for the invitation to join today. And, and it really, what you saw on stage with Sonar, with Jfr, ServiceNow, Nvidia really is in this AI world, it's like a complete life cycle of the software, uh, of software development, right?

Um, from a sonar standpoint, we start with, Hey, you are writing the code. You're a developer, you're writing the code, you're checking code in. How do you make sure that that, um, code is high quality?

Whether a developer writes it or AI is writing it, or some combination you are then, um, uh, basically giving it to Jfr to build the artifacts and to secure them and to make sure that these are, you know, that they're rock solid for you. And to provide the evidence that these are great. And doing that in concert with, um, with ServiceNow and with Nvidia, you know, NVIDIA's powering all of this, but also an amazing software development shop in its own, right.

Right. And so, so we really did think, uh, one of the expressions that Sami had as CEO of Jfr was, um, too integrated to fail, right? And I think for us, the, i the idea of, you know, we're trying to serve our customers, customers don't want silos.

They want something that just works. And that's why we were here. That's why Sonars here, and I think I speak for the others why they're here as well.

I love it, gal. You know what I, I stood I made like, just because I know you doesn't mean everyone out there helps you. Let me give you a chance to introduce yourself in your role at Jfr, and then we'll come back to what Tarek said and, and kind of show that up.

True. So I'm gal Mater, I'm the Chief strategy officer for J Rog. And, and the connection to what Tarek said is, uh, among other things, I'm responsible, uh, for the partnerships with other vendors, uh, within our industry, and we're honored to have Sona with us.

Thank you. That's one for this year. Absolutely.

Now, Shlomi said a few things up there. One, you know, that kind of struck with me. One was the singles record, single source of record, but two, what he was really talking about is when you look five years out, and look, I'm not crazy enough to think I know what's going to be five years out.

None of us are not the way things are going now. Right? Right.

Lucky if we could see what's gonna happen at the first of the year. But if you look five years out, it's, it's not gonna be one company that's your AI company, even Nvidia, for as great as they are. And, and, and their greatness is more, almost as much in their software as much as it is in their chips.

Yep. But even Nvidia, you are going to need, it's gonna take a village to run tomorrow's development shops to run tomorrow's enterprises that are AI powered, that are AI enabled, that are like turbocharged, if you will, with ai. And that's why I think it's important, breaking down those silos, right?

com. But this is more than just breaking the traditional dev ops security silos. This is really bringing the whole business together, right?

And that's why it's critical how beyond, you know, nice words up on a stage, where does the rubber meet the road? I mean, I think you have, at the end of the day, you need the people who are building the software to actually change the way they're doing something, right? And to actually, um, to, to actually understand the changes that are happening from ai, from all these things and really kind of adapt.

And so, you know, the, the, the notion we call, we, we call this sort of idea vibe, then verify that we are talking about, which is you use AI and you have to have the qual the assurance steps. You need to have the evidence, you have to have all these other pieces. That's the verification element.

And you're exactly right. This is not, there's some companies out there that are just saying, Hey, it's our platform and nobody else, right? And I don't, I don't believe that.

I won't speak for gout, right? But, but we think that, you know, we, we believe that we are really good at what we do. We invest a lot in it.

We've got several hundred people who do nothing but think about code quality and quality assurance and code security in these areas. And we think that that is great. I have zero expertise in artifact management, right?

Um, or in what ServiceNow does in the ITSM world of things like that. So in order to get the value, I think you have to go to the best of breed. And this is what we're hearing, is that people are trying to consolidate not to one platform, but to a series of best of breed, um, uh, capabilities that work well together.

So I I, I definitely agree, and I, I think that, you know, you said vibe, but verify. I, I love it, by the way. Thank you.

Uh, but, but there are, uh, different aspects of software development. And the first thing that AI did was around vibe coding. And, you know, uh, no one right now codes by themselves, right?

Everyone has agents, some giving it more responsibility, others less responsibility. But the reason that we even give it a chance, gave it a chance, it's not a chance anymore. AI is here.

Writing code is because we had this mechanism that allowed us to distill the verification process in it, right? You had this vibe coding, then some kind of a pull request to a, a, a giving. This is the control point.

And then a, a tool, a great tool like Sonar can come and verify the quality of the code and security and, and whatever it will be. But we know, we all know it's the beginning of the journey. You said we don't know where, where it'll end up.

But we already start seeing that journey extends not only to coding, but also to the release process. Yeah. So I have no doubt in my mind that we're gonna see pipelines like CICD pipelines changing to, um, include some aspects of AI making decisions or maybe orchestrating the whole thing altogether.

I don't know, as you said. But in order to allow that, we must have the same level of trust and control points in order to allow and delegate this responsibility to ai. And what I mean by that, that goes back to what you said as a system of record.

We have the system of record for coding. We now need to have the system of record for the release process itself. And I think this is what we're doing together.

That's right. Like, yeah. Sonar is, uh, giving us the, the, the, the results of their scanning, signing them, and then connecting them to the actual artifact where being there, the system of record to track these artifacts and verify at any stage all the different things that you need to, uh, verify.

So no one will get into, nothing will get into production if sonar, for example, says that the code quality is not, uh, high enough, or if any other criteria is Not met, is not met. And and I think that, just to build on this for one second, I think that this is becoming critical. We are past, you know, two years ago, there may have been a lot of magical thinking in the AI world, right?

Of this is all gonna be perfect and no one's gonna have to worry about it. And, you know, everyone will be out of a job and all this stuff. I think now we're realizing that, that the lack of trust, the lack of assurance, actually becomes an inhibitor to the adoption of AI inside of any responsible enterprise.

And that's why, why this is, we think so important. I I, I will tell you, you know, over the course of my career, I've seen a lot in technology innovation, the internet itself, probably much like you guys, we've seen the internet itself come Yeah. And cell phones and cloud and, and a lot of these innovations they focus on, uh, okay, I'm gonna code better or secure.

I've been in security 25 years myself. So we, I've seen a lot move from network security to cloud security to endpoint security. And we still don't do every of any of it.

Right? But, you know, but I've seen all of these things. Yes.

However, this ai, meaning this is so different. 'cause it affects from here to there, right? I, I really think Sachin at Microsoft said it best couple, maybe a month or two ago now.

And he said, we are moving from becoming software companies to intelligence engines, right? Where, you know, mark Andreessen famously said, software is eating the world. It ain't, the world got got a little digestion maybe, but it ain't the world.

But now we're moving from software factories, if you will, to AI factories, to intelligence engines. And to do that, you need no one, I don't care. As I said before, I don't care who it is, no one company.

I think this is going to, this whole AI thing is going to, it's, I was a biz deaf person for a lot of years, right? I, chief strategy officer, all those things. This is gonna be the greatest thing for business development and strategy, because who your partners are not comes back to what I started with.

Who your partners are are gonna determine who you are, right? You've gotta build, you've gotta build that and, you know, soup to nuts kind of partnership. Um, Tarik, I'd like to come back to you, talk a little bit about what Sonar is doing.

You know, I, I had your ct, we were talking off camera. We had, he was fantastic. You guys had some new news.

Thank you. What can you share with the, with the audience, anything since then? Well, there's a, I mean, we've been doing a lot.

The core of our business is code quality, code security, code governance, right? Yes. Really focusing on that.

One thing that really interested us, um, was, okay, if the models are the code are, are the brains of these AI agents, what kind of coders are they? Right? And what we found is that most of the, or what we believe is that most of the benchmarking that typically exist around coding models, whether GPT five or four oh, or clouds on at four, et cetera, they are all focused on the, what, what I call the IQ of the models, right?

Just can it solve this problem? Can it solve the math Olympiad of whatever, you know, things like this. And it kind of misses the whole question of what's the personality, right?

So you never hire a developer and say, they're really smart, they suck at security, they write really messy code, but it's really smart, so let's go ahead and do it. And we couldn't find anything that talked about that. And so one of the things we've done very recently is really do a really deep dive on, on the models and what are the personalities of these models.

And what you find is that the models are getting better. There's a little bit of a diminishing return curve that we're seeing, but, um, this question of functional completeness is only one dimension, right? And for example, the more reasoning that you put into the models, at least right now, what you find is actually you get not only diminishing returns, but you may actually start hurting things like security and maintainability, the mo not to overly, um, personify the models, but they kind of overthink the problem.

And so you think about this from a code standpoint, from a development standpoint, you're gonna end up with models that write more code. We've shown this quantitatively. They're very verbose.

Um, the cognitive and, and climatic complexity goes up exponentially as you have more model, uh, more sophistication in the model model. You've got more security issues, but the security issues are not the simple things you used to find. They're the hard things.

So you can be lulled into complacency. On the security side, same thing on the tech debt side. Same thing on the, on the bug side, right?

And all of this points to, hey, the bottleneck, the really hard problem. Now, one of them at least is going to be how do you review the code, right? Um, who reviews it?

How do you review it? How do you make it tractable? So that's something we've been spending a lot of time on.

Yeah. I mean, we saw this, right? 5 generation, we saw syntax errors.

Yes. Right? It doesn't make many syntax errors Anymore.

It will not make a spelling mistake, and it won't make a grammar mistake, really, right? Yep. Yep.

But the errors, it makes good, they could be pretty bad, right? And so, and that brings the human in the loop into the whole thing, right? And, and I think that's something we're still grappling with.

Yes. Right? Is, is where exactly is that human?

And is that human AI assisted or because it's, you know, the more code you generate, the, the more, either the more humans you need, or the faster the human has to be. Well, that's just, and and it is, it's more complex and it's more verbose. So the job a, I don't know any software engineer who went into software development to be a copy editor for ai.

Oh, right. It's just not the core skillset of these people, Nor what's gonna make them happy. No.

Nor what's gonna make them happy. And so you need the tooling and you need the trust, and you need then the verification. I've just done all of this hard work.

How do I stamp it? How do I make sure that now I'm shipping this, whether at the code level or at the artifact level, et cetera, so that you know that it's trusted. You know, it all, it all still comes down to one word I learned a long time ago about software and security quality.

Yeah. It comes down to the quality crap in it's crap out and bad quality makes for bad companies. Exactly.

And I, I kind of building up on that point, we create much more code and we have to somehow verify this code faster. So it cannot be manual, of course, otherwise it won't work. And this code, you know, it's not going directly to production.

It goes through our process. And this process should be scalable enough. Otherwise we'll just create a bunch of code.

But will, it will not get to it destination. So it means nothing. So we'll have to take the full process, the full software supply chain, and make sure we apply different practices, probably ai, agen, ai practices to scale the whole, uh, the whole thing.

And I think, again, this is exactly where things come, uh, uh, come together. If we focus only on the left, it's gonna stay on the left and not get, and that's Exactly right on the customer. I love It.

Hey, we're outta time. They're giving me dirty. Looks out there.

I'm sorry. Good job. I hope you've enjoyed this discussion.

It was a great discussion. T thank you. It was a pleasure meeting you.

You as well. Come back on Tech Drug team. Anytime, anytime, anytime.

Thank you so much, my friend. It's good to see you. Thank you.

Yeah. Keep doing what you're doing. You're doing a great job.

It's a great Show they're putting here. So Always, always. Okay.

We are here at Swamp Up. Check it out. We're gonna have a full day of coverage today.

Another full day tomorrow. Stay tuned. com.

Techron it Techron ai, uh, digital CXO Cloud native now, even Security Boulevard. We've got Swamp Up all over the place. I'm Alan Shimo.

We're back. Hey, everyone. We're back here with day two coverage of, uh, swamp Up.

I'm happy to have my next guest on here. His name is Yan Arbell. Yo Yan, I've, I've interviewed him before.

He's one of the unsung hero stars here at Jfr. He's a Frog. Yan, welcome to Tech Drunk.

Always a pleasure to be here with you, Alan, Pleasure to have you on. Before we jump into, looks like we just had a plane, Passover or something, huh? Yeah.

Before we, uh, jump into what you spoke about today, if you wouldn't mind, talk to the group, tell them a little bit about what you do with Jfr, where you've been, how you got here. Right. Uh, okay.

So, first of all, I've been with J Rog for the last nine years, uh, doing a lot of stuff. I started as a software developer. Uh, afterwards, uh, like few years later, I was a team leader at the infrastructure group, uh, where we kind of, uh, planted the seed for the platform, what we know today as what we know for today.

Okay. Um, and later on, um, you have Jfr CTO approach and say, Hey, Anan, you're so active on the networks, you wanna be, uh, dere and start, you know, communicate, bridging the gap between what Jfr does, you know, outside to the world. So, you know, kind of try to simplify things.

'cause Jfr, you know, we do a lot of stuff from the most left side to the right side. There is a lot to cover, and we need to somehow order all this building blocks, so it'll be easier to, you know, to grasp what we are doing. Excellent.

Now, so you moved into from software development to Derel, Right? Right. I moved in from software development to Dere, but, But always a, but Yeah.

Um, it's, so I'm, I'm, I'm so grateful that I can work with Joab because he is my CDO. But not only that, because, uh, giving me the freedom to also explore and keep coding, you know, once developer always developers. Right, right.

And, and when people ask me, Hey, you, you moved from a development to be a dev. I say, you know, it's like swimming. You, you, once you're a developer, you have the, the, you have the, you know, this infrastructure you can always get back on track and, and, uh, keep coding.

And I never stopped coding, actually. And in fact, jfr MCP server, our first open source experimental MCP server is server that IOP I created. And I, uh, you know, uh, uploaded to GitHub and we, you know, we started seeing some eye rolling on this, uh, on this project.

And, and it got, you know, lots of our customers got really excited about this MCP capabilities. Uh, we had PRS from Nvidia and even, uh, a guy from Cisco approached and say, Hey, we used your MCP, it was so good. We, we even take it as a reference for our MCP.

So it was, first of all, it was really flattered, uh, from it. Uh, but yeah, I, I keep my hands on the, on the keyboard. We keep, I keep coding.

And, you know, even today with AI is so easy. Yes, it Is. Everyone can code vibe.

Vibe code. Yeah. Vibe coding.

I, you know, we, we say vibe coding. I like the, the word I had sketch coding. Okay.

Yeah. Because vibe is like, you know, I, when I think about something, I, I, I give the prompt and I, I start sketching what I want eventually. If, if I think if it's something that I, I feel that is can, can have some maturity, I will take it to the next level.

And by the way, this is some, this is kind of how I build the MCP. It start with sketching. And then I created, uh, a tool, an MPY tool, and it opened another door.

And I like text, uh, text my manager bragging about each new capability that I just created. Hey, you know, that now you can ask about the production environment in runtime just from your id. Because I, I, it felt like a magic.

It felt That, you know, that is very close to how I use it too. 'cause I use it more for my stories and stuff like that and articles and scripts. And, um, I also, I, I never called it sketching, but that's what I do, is I, I sketch it out in an outline and then I keep putting more flesh on the bone.

Right. And then I ask it to polish it up. Yeah.

Um, I wanted to come back to, and, and talk a little bit about you speaking here. Well let, before we talk about that, let's go back to MCP server. Yeah.

It's crazy. In January, no one knew what it was. Right.

MCP server today, everybody has an m it's become the standard way we're communicating. Right. And, and you know, at first we heard MCP, like it was everywhere.

And now we hear it about it, uh, a little bit less and not because people, uh, are not using it because it, it became the standard. Yeah. It's built in already.

It's Not, it's built in. Yeah, exactly. Um, and, and it's amazing.

And we, we see the whole ecosystem around, um, you know, how the, it's not, it's not about only, uh, creating this new shiny model the strongest with the biggest context. It's now about how we as, as persons communicate with these AI agents and how AI agents communicate with another AI agent. So that's what's, it's now, now it's about the communication and interaction interaction between these AI to ai.

Yeah. AI to ai. Exactly.

Alright, let's turn to your presentation. Yeah. Tell us what you spoke about.

Yeah, so in, in my presentation, uh, I talked about the journey of, uh, what Jfr did from the moment, you know, AI came into our lives and, and like, like we talked yesterday and like gal mentioned today, we see every billboard out there saying how they are working with AI and what they're doing, uh, in order to be an AI company. Um, so ai, it's a tool come to help us to, to expedite the, the, you know, the processes. And I started by, by, you know, telling the audience about what we did, uh, at first with the jf, how it's like, it was kind of, uh, an, it started as an internal gig or experiment to see, uh, if we can reduce the friction between developers and documentation.

So if I'm a developer and I want to know how can I, uh, use the CLI in order to upload a new folder to Artifactory or, or add a property, I'll just do JFL and then I'll ask my question and hopefully the model will return me, uh, the right command with the right, uh, arguments. And so I can just, you know, alright, yeah, that worked for me, let's go. And that's where, um, our, it, it didn't do a lot of, uh, you know, a lot of splash out there.

Um, but yet it was a sign for us that, that the intersection between AI and DevOps does make sense. And we can, uh, you know, get these tools to help our customers and even ourselves, um, by just, you know, having the right models, the right train, uh, trained model and, and, you know, flow with it. And then we started develop more things and we see like, uh, and last year when, uh, I was on stage with, uh, with my partner in crime, Brian from GitHub, and we announced the, this, um, copilot extension Yes.

For Jfr, I remember. And, and it was great. It was finally our AI agent got a context of, of external service jfr in that case, uh, it has some caveats like the add jfr that you need to add at the beginning, like telling, listen, now you will take all the information from jfr.

Um, it worked and it worked well. Um, but the world needed something more. I call it like a smoother process.

So, and that's where MCP came in place and kind of, I don't wanna say made it redundant, but made it much more easier. I don't need, I don't need to tell it anymore. Hey, when I speaking with you about, I dunno, build repositories, release bundles, you should understand that you have the tools that's relevant, uh, to do this operation with jfr.

So LLM is smart enough today. It has a context there in the size of a absolutely in the size of a book. It, it, it understands the entire project.

Uh, if previously it got nervous from 500 characters, now it can understand a full project. You know what, it's every day it gets better too. Is is the thing about it.

And I think, you know, talking, we, so I talk to a lot of companies Yeah. And talk to a lot of the companies building the LLMs and the models and so forth. Yeah.

Even if they stop developing it now, it'll take us seven years to digest Yeah. What's available to us, right. Using the MCP servers, using, you know, what we have.

Right. It, it's, it's an amazing time to be involved in this. Yeah, absolutely.

Where can people get more information about the MCP server and about some of the things you're working on? Right. So, uh, alongside my MCP server, and once we see the hype around it, uh, we created a production grade, uh, on in SaaS.

Like we, we, we host it, we maintain it, we update it. 1. So, um, our SAS customers can already, um, with it.

Yeah. For now it's for sas and they can just, with, with two clicks, they can connect, uh, our MCP server with their environment, whether it's their IDE or an agents that they develop themselves. They can, uh, connect to our MCP and we still have the experimental, uh, MCP server, which just go and type, uh, MCP, GitHub jfr, it'll get you directly there, Objective out There as well.

It'll get you directly there. So, Mr. Devereux, where are you headed to next?

Well, you know, we, we always explore and we see we, we saw this, uh, great announcement during, uh, during Swamp Up that even, even I need to explore and understand more and, and, you know, put my hands and get into these trenches and start, walk with it. It looks super exciting and it make lot, lots of sense. The this, uh, you know, all the announcements about the, the partners and, and everything.

It's like, whoa, okay. We, we need to see it's, it's looks so amazing and we can get, we can do so much things with it. So, um, mainly explore, explore around all how we can, you know, work with this ai, integrate everything together so everything will fall into place.

I have no doubt it'll happen. Cool. Alright.

Pleasure seeing you my friend. Y you Yan our bell. Thank you.

Dere, but always a developer. Once a developer. Always a developer.

Exactly. Here at Jfr Swamp up. We're gonna take a quick break.

We've got a lot more coming. Your hardware based memory is going away. Well, it's not going away, but maybe we can make your models fit in much larger space.

Join me today for the Tech Field Day podcast. Welcome to the Tech Field Day podcast, where we bring together a group of IT technical experts to discuss a single idea about key concepts in the industry. This podcast features a variety of perspectives from members of the Tech Field Day delegate community, and it's often recorded in association with one of our, our events.

Tech Field Day is part of the Futurum Group, and this podcast is also published on Sister Company Site Techstrong tv. On this episode presented by Fon, we'll be discussing the premise that AI is gonna be without HBM high bandwidth memory. And that flash memory unification is changing training and inference before the discussion.

Let's meet who's on the panel today. Hi everyone, my name's Sebastian. I am with Fon.

I'm the CTO. It's a pleasure to be here and have this conversation. Uh, Fons really excited about ai, which is a little bit unexpected because we make storage.

And you would think, why is storage excited about ai? Turns out you can do a lot with storage and AI when you have the right mindset. Hi, I'm Brian Martin, VP of AI and Data center performance at Signal 65.

Spent my career around storage performance most recently ai. So I am excited about the confluence of these two technologies coming together. And hey everybody, I'm Max Montero.

I'm Chief Research Officer at Osteum Data Group, and I am, uh, absolutely bonkers about everything flash and, uh, silicon and, uh, storage. And of course I'm Alistair Cook. I'm an event lead here at Tech Field and the event lead for the Iion Infrastructure Field Day, uh, series.

And that's where I brought together these three people, uh, who are my guests today. And I think we've all seen this sort of transition, uh, with storage getting faster and faster. And we saw per particularly persistent storage attached to all kinds of new faster interfaces.

But I don't think we quite saw coming that Flash storage was going to replace the highest speed of RAM and high bandwidth memory that's sitting on the, on your GPUs. Uh, this is definitely one of the big cost elements for any AI deployment is buying vast numbers of huge, uh, GPUs. Uh, some of these big GPUs cost as much as a low cost car, and by the time you've filled out a server with eight of them, you've got a pretty expensive piece of hardware.

You start racks and yeah, you better have deep pockets in a long, well, a big payback on that. Sebastian, you've proposed this up. What does Unified storage with Flash and how's it taking away this high, high bandwidth memory requirement from ai?

It's not that it, it completely removes it because you're always gonna need, uh, some amount of dram and, and especially with the scale at which, um, high-end AI works, HBM is the, the right technology for that, that field. But there, there's kind of a, a balancing act where you do need a, a large amount of active memory, but then if you are careful with how you pipeline the requests, you can actually stream in, um, a any upcoming data that you will need before it's needed so that it's residency and HBM does not have to be a hundred percent of the time, but rather page in, page out, just in time so that you can essentially reduce that HBM footprint and focus on the part that you need for active calculations and then use an a less expensive, but still very fast memory for, um, for holding that data while it's it's pending processing. Yeah, no, a, a absolutely.

The uh, you know, and a lot of the work I've been looking at, uh, both, uh, in my day job working on H two hundreds, MI three hundreds, some of the bigger scale systems, uh, overflowing that memory and going to multiple GPUs to fill that up. I often run into this balance of time versus memory, or not being able to use the memory. I, I, I feel guilty sometimes when I'm using eight GPUs and they're running at 40% memory capacity.

I'm like, I'm not doing something right. There's something I can dial up. Uh, my home rig is an RTX 6,000 pro, uh, which I'm super happy with.

Uh, and at 96 gig of ram, I can do a lot with it. But when it goes, comes time to try to train, um, the models, especially as they get a little bit larger, eight gig, 70 gig, um, I don't have the GPU for it to have the memory. So be able to flex out to a high speed, uh, persistent storage for caching that I can trade an overnight run, uh, to be able to get that done, uh, in my home web.

And, and that's the interesting part, right? I mean, depending on who you talk to, when you initially present the idea, they're like, what? And, and that's the trade off.

'cause we haven't mentioned that before. When you are paging things in and out, um, you, you're trading off time for cost, but it's, it's actually linear scaling. So instead of like, so for example, if, um, I'll just make up nice round numbers.

Let's say that you needed, um, a hundred gigabytes to do, uh, a model and to do inference training. And that would take you one hour if you were to, and, and that, let's just say that that was two GPUs. So if you remove it and you reduce it down to one GPU, you have half the amount of memory that would be required.

So you actually have to do things in, in batches and, and the trade off is linear. So if one, if two GPUs would do the training in one hour, one GPU will do the training in two hours. Now, for some companies, they need just, you know, they, they need as fast as possible, and that's fine, that that is a perfectly valid use case.

But there are a lot of other use cases where it doesn't matter if it's, you know, if it takes four or five hours, it's fine for that particular use case. And so our, our proposal with what Fon has, which is our, our adaptive solution, which has a few different features, what we're basically saying is, look, if, if this is more, if you wanna balance your cost and you wanna balance, um, you know, your, your compute and the amount of memory that you need is, you know, without our solution, you really have no choice. Everything has to fit in memory or it doesn't run with our solution.

You now have a choice for, you know, scenarios where you, you need that speed, absolutely go to full GPU 100% HPM solution. But for other scenarios where it can take a little longer, this now gives you a lot of options for choosing how to right size your deployment to match your needs and your budget. Yeah.

And, and, and, and, and Sebastian, one of the things that I understood, because I was at AI in field day two where you were presenting, uh, adaptive, the, the solution is that if, if you are in a greenfield deployment, when you're just starting from the ground up, you have the opportunity to kind of architect everything so that it fits your need. But here, I think that the value comes from the fact that you can use that to kind of plug it into something which was already done before. And you can add in a very cost effective way, you can, you know, address those challenges of scalability of, you know, parallelizing and so on, right?

Yeah. Uh, another way to look at it is, um, if you have a scenario where, um, you already have your deployment and the, the primary use case we'll say, is an 8 billion parameter model, and then you have one department within your company that needs to, needs the, the complexity, the nuance that's available with a 70 billion parameter model, you don't have to buy it for the one use case that happens once a month. You can scale your equipment for your average use case.

And then when you have something that requires more capability, this is where adaptive comes in, unless you just basically offload that extra memory requirement to these, uh, uh, high endurance SSDs, we call ours the AI 100. Um, but the idea here is that, you know, you, you essentially scale for your, your primary use case, and then you, you, um, you leverage other memories to, to get to those, uh, special use cases, Right? It's, you know, with, with the GPU hardware, as you said earlier, it prevents customers from having to buy equipment to satisfy their absolute max use case and then run less than when they're not doing that.

And I think there's a, there's an important distinction here in training around model size, because when we're in the infant stage, the model size is the, the minimum, um, hardware based memory footprint for us to, to hold the entire model. So if you've got a 7 billion parameter model, it's gonna be roughly seven gigabytes of ram of hardware based memory. But when you're doing training, you need rather more than that.

And so if your use cases, primarily we're using the hardware to do inference, but once a month we're doing fine tuning training in order to get a, a faster response out of our, you know, normal rag system. This is a, a use case where that process for a couple of days to improve the quality of the base model might suit really nicely, but we don't wanna spend four times as much on the GPUs for that training folks. Yeah, the, the memory footprint required.

Um, so if you do full fine tuned training, it's generally 20 times or 20 x, uh, the model size. So for example, a 70 billion parameter model times 10 would be 700 and times two. 4 terabytes of memory.

That's a lot of GPUs. That's about 32 of the, um, well, not, so when the memory footprint was around 48 gigabytes, uh, for the RTX 4,006,000 range, I, I don't know if they've gone up recently, but if they were, if they were around, uh, 48 gigabytes, you would need 32 of those to actually do full fine tune. Now, there's another technique called LoRa, which is low ranking adaptation, and typically that's two to four x the models memory size that, that's the footprint required to train it.

And the thing about Lori is it either works or it doesn't. And, and it's, it's pretty, um, pretty obvious if it didn't work. Uh, and so you just go, okay, Laura didn't work, so let's def let's fall back to regular fine tuned training, and then that memory footprint goes up.

Um, but for, you know, and some, some people might say, well, it's not really 20. It is and it isn't. So in some cases it has to be 20 because Laura didn't work, and when it, but when it does work, it's two to four x, Right?

And even at the two to four, like, so the new, you know, RTX Pro 6,000 Blackwell is 96 gig of memory now. So a 70, 72 billion parameter model at four x, I still need two three cards for that. So the, the trade off and, and with, uh, you know, one or two of the AI 100, um, SSDs, you know, I can do either Laura or full fine tuning on a single GPU or a pair of GPUs, Right?

And, and our solutions actually, so we have them in M two form factor, which will fit into most workstations. And, and a lot of workstations have like two or three spare M two slots, so it's kind of nice. It, it fits in nicely, but our solution will also work on servers.

And so there are people that have, for example, a server with, um, 4, 6 8 GPUs in the, you know, RTX six range or the, uh, HB 200 range. And so even there, it has benefits because some of the other features that, that this offload capability adds is that we can, we can extend the KB cache to be much larger than what the memory footprint would allow. And that's important when you're supporting a lot of users and you're doing a lot of batch training, it allows you to service more users per, um, per box than you could otherwise.

Now, when I mentioned that people are, you know, one question that comes up, and I, I think I talked about this at, uh, a presentation recently. One of the things that comes up is, um, well, won't that be slower, because like even two gen four SSDs, that's 14 gigabytes per second, and on a workstation you've got a hundred gigabytes per second of dram. We actually did a test with two of our SSDs and an HB 200, and we had a workload of, uh, a batch size of 220 users with an input of 2000 tokens.

And, uh, respons of about 200 tokens, which is a typical sort of one shot kind of query, um, in an environment where you're supporting multiple users. Um, and what we found is that the number of tokens per second that you can service in, if, if all of the activities in HBM and, and the, uh, h HP 200 does have HBM, um, versus if you're offloading to the SSD, there's less than 10% difference. And what that really, because again, this doesn't sound right, but what it comes down to is that there are other bottlenecks in the system.

And so whether you're offloading to HB M or you're offloading to the SSD, you still get the same performance. But what that means is that if that HBM memory, and I, I don't remember the memory of an HB 200, but let's say it's a hundred gigs, uh, you can easily extend that to 1, 2, 3, 4 terabytes and have substantially longer context with no performance degradation. And without that, your only solution would be to have more and more and more servers.

So it's an interesting, um, it's an interesting feature that that actually adds a lot of value when you're not training. Yeah. Depending on the number of users you have, you know, a certain token rate coming back is more than enough.

You know, I can't read faster than 20 tokens per second. I love it when we come in at 40 tokens per second, I can just scan it, but, you know, 200, 300 tokens per second is, is overkill for me. So being able to balance that out, and I'm glad you mentioned the higher context size, because these models, the open source models are now often at 1 28 K and soon expect them to go larger.

And that context is key, uh, for good quality results. But even if you're not using the full 1 28 K, you know, you're going to 2, 3, 4, 5, 10,000 tokens, um, and you get a lot of that extra input either from a, a long session or a lot of input, reference documents that you're, you're putting into your, your query, even in that scenario. Um, it, you know, even if it's not the max, it turns out that you will quickly run out of memory if, if your only pool is the hbm.

Other thing that we allowed to do, so we talked about, uh, oh, sorry. So I mentioned context window, um, and they kind of work together there, there's the idea of context window expansion, which is to say that maybe with the memory pool that you had, it, it's only this big, you can make it longer. But parallel to that is KV cash eviction.

So once your KD cash is full, an entry gets evicted and then disappears, and then you have to recalculate it. And that's why you see in certain scenarios where the first query is pretty quick, second query is a little slower, third query is a little slower, fourth query is even slower. And what we've actually found is that when you catch those evictions and you put them on the SSD, and then the way that our middleware works is that it first checks the, the HBM cash, is it there?

No. Then it goes to check to the SSD, um, that whole flow, which intuitively would say, oh, yeah, that's gonna be slow and suck. Nope, that, that's what I was referring to.

There's less than a 10% degradation fetching those missing tokens from the SSD. Um, you know, versus had they always been resonant in the HBM memory. So, so what that means is that when you ask question one, question two, question three, question four, their speed is flat.

There's no slowdown in a scenario where you have a lot of users taxing the resource of the, uh, the GPU. So both of those go together. And there's another one that we can do in low, uh, resource environments, like typically what you see in Edge ai, um, and we can actually stream instead of, normally a model is broken up kind of in vertical slices across all available GPUs.

Another way to do it is to stream the model in one layer at a time and has about 30 layers. Um, and so yes, inference is slower, but again, in the case where you don't need maximum performance, and this is, it's for that particular use case, it's okay that it's slower. You end up with a scenario where you can actually run a much larger model without quantization than what would normally be possible without this kind of offload technology, which unifies, uh, the available memory, which is either, you know, dram, VRA or HBM or unifies that memory with the pool that's available from the SSDs that are dedicated to this task.

It's not like you're using the S SSDs to also do storage and os activity. It's, it's dedicated to the AI task, but it allows you to do a lot more with your hardware. You're clearly not super excited about this, are you, Sebastian?

Well, I think it's cool because it's great if you have infinite funds, and some companies do, a lot don't, and they're getting left on the sidelines or they're being sold, oh, go to the cloud. That's fine, as long as you are comfortable with your cloud provider. And, and the reality is, regardless of who your provider is, um, and we've seen recent stories where some of the biggest providers have had cases where information leaks, once the info is leaked, it's leaked, and you have better control when it's in your own environment.

But also, you're not, like those big environments have a giant bullseye on them, right? Because they're, they're juicy. If you're an, you know, if, if you're not a Fortune 500 company, there's odd, the odds are good that nobody's even attacking your infrastructure because they don't know whether it's there.

So there, there is some value in the anonymity of being a minnow in a flock as opposed to being a big guy right at the front that everyone's looking at. Yeah. So the interesting thing about what you, what you just said around, you know, the ability to, uh, do stuff locally versus in the cloud and avoiding, you know, being too much exposed is that even if you're in 14 500, or let's say even extend that to 14, 2000 organizations, you always have the ability to, you know, go on the cloud and do stuff.

But then again, uh, in some industries which are regulated, you have the risk of, you know, leaking sensitive data, right? So it could be, uh, if we're talking about the, the pharmaceutical industry, for example, it could be research data or kind of, you know, sensitive personnel, identifiable information and so on. And most of the time people come back, you know, to infrastructure teams and tell us, you know, we need, I don't know, 4,000, 400,000, $500,000, we need to buy some servers, this and that.

I think that is a really elegant solution, you know, to allow people to work on their work locally with their data on their workstations without having to go with the complexity of the cloud and the potential, you know, conflicts with, you know, uh, data, you know, data management, data governance, and with the, uh, let's say the high upfront costs. You know, and, and, and what you also said as well, the ability to, to run stuff, different models, it gives a lot, a lot of flexibility to organizations. So you can have a way to test things locally before you decide, yes, this is the right model, this is the right thing I want to use, and I can, you know, kind of industrialize what I'm doing here and bring it to the next level, perhaps.

Yeah. There, there's also, um, so some, sometimes people say, Hey, I can do rag RAG, I'm good. I don't need to do anything else.

And what we found is that Sure RAG is great because it gives you a concrete reference to where that information comes from. But if the model isn't trained on the data that it's referencing, it's essentially using its generic interpretation, which comes from training at large on the internet where wherever the model was originally trained, which is usually based on a bunch of stuff from Reddit, Wikipedia, and stuff like that. And so if you train, so if you combine both, if you have the rag reference so that you can get pointer to the specific document, and you combine that with a model that was trained to understand that document a little bit better, uh, through fine tuned training, you essentially get another 10 to 15% improvement on the quality of the, the responses that it's giving you.

So, you know, on top of all of the other benefits, th this is another thing, and, and our solution actually works on everything from Edge, you know, systems like, uh, something based on a Jetson all the way up to PCs, like an educational pc, which is another market that is heavily underserved, is, is just universities have the right tools for training their, their students. So this opens up possibilities, and we have partnerships with a few universities, um, and then, um, it goes all the way up to workstations and, and even servers. So our solution scales quite a bit and, and opens up possibilities that weren't there otherwise.

Just remember to put a really good heat sink on those SSDs. Um, yeah, in general, with these Gen four, gen five and soon Gen six SSDs, and, and we make them in all classes, um, it's generally recommended to have a heat sync, uh, because essentially the, the one workload that stresses the drive the most is, is the right workflow, which happens a lot and offload. Um, and so having a good thermal solution within your, your device is very important.

That's true. And one of the things that struck me about this also was that, that you, you mentioned middleware layer, but it really is a, it's a plugin to standard framework. It's not that I have to re-architect and rebuild my entire application.

This, this just plugged straight in. Uh, and I think that was, that's a, a vital part of any solution that's gonna change our workflow is you can still use the, the same processes that you'd use on your workstation with this if you maybe move into a, uh, production environment that doesn't have, or doesn't yet have the Solution in place. Yeah.

And in many, we, we interoperate with most open source tools. Um, right now we're on PyTorch in the PyTorch runtime, I always forget what it's called. It, it, it's not Run Torch, but it, it's something like that executor.

So we work with Executor, we work with PyTorch, uh, and, and just naturally collaborate with open source tools because essentially we're all leveraging the PyTorch framework, and we, we, we operate in our space, but we don't try to block anything else. So there's a lot of opportunities to mix and match. Thank you all for joining us today on the Tick Field Day podcast.

Before we go, where can people continue this conversation, join, uh, Ian synergy and enthusiasm, and learn a little more about this and about my guests? Well, we, uh, we have a, um, uh, a website and a blog that talks mo about our products. And of course there's a, a contact page there.

And, and if you want to reach out to me directly, uh, easiest way to do that is through LinkedIn. com and on LinkedIn as well. Yeah.

And here it's osmium group com and again, LinkedIn Rules Supreme. Yeah. And the fragmentation of the other social media.

LinkedIn seems to be taking over. You can find me Alistair Cook on LinkedIn as well. You can also find me periodically on the, uh, futurum group site, as well as on my own Demi test co nz site.

So thank you very much for joining us and listening to this episode of The Tech Fields, a podcast. And if you enjoyed this discussion, please subscribe on YouTube or your favorite podcast application so you don't miss a single episode. Do consider giving us a rating and a very nice review how much you enjoy the enthusiasm we bring.

This podcast was brought to you by Fon and Tech Field Day, the home of IT experts from across the enterprise and a part of the RUM group for upcoming events, more episodes, head to tech field day com slash podcast of your on text on tv. Thanks for listening, and we'll see you next week Once that cracking noise. You hear, it could be DevSecOps Foundations, you're watching Textron Gang.

Hi everyone, happy Monday. I hope you've had a great weekend. I'm Alan Shimel, and this is Techron Gang.

Um, man, I don't know, my weekend was much too short. I, I was looking forward to doing so many things and I, I had a honeydew list instead, so it was a bit of a honeydew weekend for me. And I don't mean the green melon.

Um, I'm almost happy to be here talking tech strong and tech with y'all. Let me introduce you to my compadres on the, uh, panel this morning, wearing his Indiana Jones hat Jack Poller guitar man, Mitch Ashley, and, uh, well, I don't know if he's still a Yankee fan, but he's chief content officer, you know, Mike Ard. Gentlemen, welcome to Textron gang.

Happy Monday to you, Mike. com and other sites, other tech drunk sites, non-text drunk sites, software, supply chain security, DevSecOps. You know, it's a, it's, it's a enough to shake to shake your confidence, you know, but it's the security pros dilemma.

We never seem, you know, we just never win. There's always something else out there. Let me, yeah, let me, let me go through the list of what we're talking about though.

I mean, shy Ude is attacking the CrowdStrike environments and other software supply chains. Andro is reporting that people are abusing claw the lodge attacks. White Cobra is a criminal group that's targeting, uh, visual studio extensions.

And then we see Oasis Security made a note of the fact that there's a lot of weaknesses in the cursor AI coding tool. Again, exploited and checks marks did the same thing and said, Hey, you know what, you can lie to these AI agents and tell 'em anything you want 'em to know, and then they'll give you access and do all kinds of weird things in your code. Mitch, none of this sounds good.

And, uh, Alan has a piece talking about how the foundations of our DevSecOps world are starting to crack. Do you agree? And what are we supposed to be doing about all this?

Well, I see many futures, many possible future about this may take place. So, and actually, there, there are, I mean, I think these are real examples, right? Of how AI can be used or is being used in an offensive standpoint.

And, and I don't know if you mentioned in, in the case of even anthropic and generating code, it's about extortion. It's about getting, you know, money from people. It's about how we craft now software that can go after not just large amounts of, uh, you know, individuals in a particular n and m kind of fashion, but also it can be done on a personalized individual basis.

So I think these are all good examples. I hope it, it sort of heats up the, the, the water, if you will, and ramps up the interest on getting more secure code generated out of software. But also more importantly is, is making sure that there are security guardrails that we can contain.

I did a piece on is regulation gonna save us? No regulation's not gonna save us. I think the market has to respond to creating and secure guardrails that we can control and manage.

But right now we're sort of seeing kind of if and every, any and everything that can happen with, uh, security around ai. Mm-hmm. Jack, let me ask you, d does it feel like we're being a little reckless from the security standpoint?

It seems like we're just letting these developers turn loose, do whatever the hell they damn well want, and it shows up in a production environment. And then guys like you are asked to clean it all up. Why should today be different than any other day?

I was just Gonna say that new boss, just like the old boss. Exactly, yes. We're being reckless.

I mean, there is a lack of fundamental cybersecurity hygiene, you know, uh, there's the AI issue, and then you look at something like the Chi MPM attack, the root cause of that attack isn't supply chain, it's actually poor password, it's passwords, right? You had people who were phished that then got access that enabled the bad guys to get access to the code base and then propagated from there. So why are we still doing password based authentication?

We have passwordless authentication, we have MFA, why are people leaving the front door unlocked? Mm-hmm. You know, and then, alright, So Go ahead.

I'm sorry, Jack. Oh, I was just gonna say, and the AI doesn't make, it doesn't make necessarily change the attack path. What it does is just enables people to act faster on the attack path, just like AI does for everything else.

It's an acceleration function. Okay? I think it's an acceleration function and it's, it's a breath.

You can do everything all at once. Anything, yeah, everything everywhere, all at once, that kind of thing. That's, that's one of the big differences.

It's not Just, it's a force multiplier for the bad guys, unfortunately at this point. Good way to say it. But, but let me back up a little bit.

First of all, I gotta give credit to whoever comes up with these names. I mean, I saw Shy Ude, it had me at Shy. I was, I was ready to go Blue eye and blue and blue eyes, and you know, I, my mind was running away with me, right?

You're ready to start the galactic war. What a great name for a worm. Why didn't I think of this?

Right? Where are names been all This time, right? White Cobra.

What was the movie where there was that guy, fat Cobra or something was his name. And he, it was one of these spoofs of like a James Bond thing, and the guy goes out and it's, the guy's name is Fat Cobra, but he, he's like a ruthless killer or whatever. I mean, what, who comes up with the names for these things?

I'd love to know, right? Spider? What's the other thing?

Spider? This gathered Spider. Spider.

Yeah. Yeah. These are great names.

This is like outta Hollywood. But, but seriously for a second, I, all our chickens are coming home to roost. We've known, we've known about our software, supply chain issues, security issues.

For some reason we thought putting SBOs in making SBOs mandatory would somehow fix this. No, it doesn't fix it. No, it doesn't fix it.

Number one. Number two, we've known AI is going to be a force multiplier that the bad guys, we're going to use ai. And that if we're gonna rely on these models that are taken from the bad code, we've been, been using all these years to generate new code, and we're surprised that the code it generates is insecure.

It doesn't stop us from using it. 60% of the code out there has, has AI's fingerprints on it. And, and we sit here and say, gee, I didn't see that coming, right?

This is, this is, you reap what you sow, right? Mm-hmm. This is, you reap what you sow.

We, this is, this is what happens here to us. And you know, and look, Jack, Mitch, me, Mike, we've all been around the block. We've all been in involved in the security game a long time.

We're always one step away from the ultimate calamity, but somehow, somehow we live on to the next episode, right? We, we, we squeeze through, you know, it doesn't turn out as bad as we, we it could be. So let me, let me challenge you on that, Alan, because I think we're in a, in a situation of fighting the next war with the last war's weapons.

Oh, no doubt. And we're, we're, we're Thinking about with Swift rocking and scanning and mm-hmm. Doing all these things that are passive after the fact.

SBOs, you name it. All, all good things, good Practice. It's a security management though.

Line. It, it, it is, actually, I'm Gonna write in, I'm gonna write an article about that Security management. You gotta, you gotta be over, you gotta be over 45 to know what that means.

But go ahead. The students of, but students of history know at that, are there any students of history left? But we're, we're, we're, the new war is, you know, this is, this is battlefield.

You know, we're, we're the, we're the, uh, red coats or something. Is it, it can't, we're not in a world of, it's about creating defenses. You have to fight fire with fire.

You have to fight AI with ai. And I'm not just saying that because AI is the answer to everything, but you know what, if, if they're doing calculus and you're doing algebra, you better or damn well learn calculus. Matter of fact, you better be really good at calculus.

And that's what we've gotta do. And so, you know, my wake up call to everybody, software developers, security professionals, all of this is get on the fricking AI bandwagon. Not just because it's the thing to do and it's popular.

And, you know, chatbots are that cool. Is that, is that's the new war. That's the new environment.

And if you ain't good at it, you know you're gonna get rolled over by the, the digital tanks of Tron taking care of you, taking you out. Right? I mean, to Mitch's point, I mean, I would love to be able to say to every developer in the world that thou shall have two factor authentication.

And you, you're being, you know, maybe subject to some penalties. 'cause you don't, but I don't think that's realistic. I do think when we need something that detects an anomaly, like instantaneously and then applies policies instantaneously.

'cause the amount of time for which havoc is wrecked is now measured in seconds. Right? And this is where the whole thing is moved to.

It's machine versus machine. And if we don't have machines that can fight the fight, we're gonna lose for sure. Well, I think the other part of it is we need to prioritize security higher.

Right now, a lot of what we do is productivity. We focus on productivity, both on output of developers and on making their lives easier. So you don't have MFA because it's friction.

You don't, you have automatic updates of your NPM packages because it's easy and quick, and you can press a a button and you get an update, but that bypasses a check of what exactly are you pulling in, in your packages, which is how this worm propagates, right? It automatically gets updated and it often goes to the next package. So if we look at our environments that we're building and the environment that we use to build things, we should be saying, yes, we've made it.

So it's super simple. It's PhD, it's push here, dummy, right? Everything's a push button.

You get an update, you get this, you get that, and let's sort of maybe dial that back and say, how do we layer in a little bit more checks to prevent bad stuff coming in, in one way or another? Right? And I, I used to talk about this in terms of we need to compensate, you know, we compensate developers on feature functionality and not on security.

The problem is it's open source. So how do you, you're not compensating them. They're doing this for free.

They're volunteers, so you can't penalize them and you can't compensate them for good or bad security. So we have to think about it in a different way. You can though, but I think you can.

I think you can. Okay. You know, here's, here's my take on the software supply chain issue.

The fact of the matter is, most developers are not conjuring up this open source software outta thin air. They're downloading it from a repo, from multiple repos, whether it's GitHub or, or NPM or Artifactory or Maven or, or whatever, right? Wherever they're grabbing this stuff from, the point where they download it from the repo, in my mind, has always been a choke point.

It's always been the point where you could say, wait a second, is this piece, is this script? Is this component, is this container secure? Is it the latest version?

Has it been scanned from malware? Right? Is it, and, and, and some, some things are still going to get by, right?

'cause we don't know that it had the vulnerability. It's a zero day or whatever. But so much of our aggravation here, of our security stuff could be solved at that border of the repo, uh, from the downloading from the repo.

Now, I know certain repo security companies have talked about sort of a repo firewall for forever. That if you're going to download their software before you get that software in, it's gonna be x-rayed, it's gonna be scanned, it's gonna be tested. But we don't, but developers don't do it.

We don't do it. We don't as a rule do it. We, that's the hygiene we need here.

We need Well, And that's, that's my point exactly. Is that's the, the, that's the slowing down of the development cycle. Don't do it automatically.

Have it go through some steps to check it in some way or another. But can, Why can't we make it automatic Jack? It's easy enough.

It's easy enough. Well, that's my, that's my point, Alan, is, is it should never be a next step is when security happens. It should be never be a, and then we scan it and then we go, and then we put it through a firewall, and then we do this right?

In an age of AI code to, to your earlier point, Mike, the code that we're getting out of, out of LMS today is all the insecure code that we put into it, right? We're getting back. We're, we're reaping what we sewed.

But so to solve that, you're not going to, you're not gonna improve it by just adding better steps at the end, after we've generated code, generate code that's created by ai, should already be secure by agents, ai, LLMs, all kinds of things can make it secure. Whether it got it through a good firewall or it downloaded from the, you know, crappiest, whatever. That software should be secured before it ever was presented to a repo or to a human or whatever.

Those steps can happen before we touch it, rather than, here you go, Mitch, here's some more JavaScript code. Try that. That now has some new things in it that, whoops, you know, I do, I need to run that through a scanner every time I test it on my system so that, you know, I don't get compromised.

Let's get real. You're not gonna create secure software by bolting on more steps just faster. You've gotta build it in.

You've, instead of shift left, you gotta shift in a security. You Know what this, this reminds me of? Like the Apple walled garden approach to apps versus the Google Play.

Not, not Google Play today, but the Google Play store earlier on mm-hmm. Where anybody could basically upload an app to Google and there was a lot of malware infested apps in the Google Play store where Apple, yeah. They were pain in the butt.

But they did it. They did. I think they did.

Anyway, check your apps before it was accepted into the Apple store to make sure it wasn't a security risk. And so we had much less security incidents from downloading apps. And I'm not saying it 'cause I'm a fanboy, but we had, you know, they did do a better job of, of testing them on the way in instead of just on the way out.

It maybe, maybe that, maybe that's the thing to do From, so, Mitch, let me ask you the question here. Yeah. We keep talking about best practices.

You should do the right thing. We should have some empathy. We should all lock arms and do the right thing, and yet we don't.

So at what point are we gonna get to where, you know, it's gonna take, you know, to use a phrase that's popular these days, we're gonna have to lock a couple of people up to make the point. Well, it could be lock the couple of people up. I think the, the real answer is enterprising entrepreneurial companies, people that solve this to make money, that's the way to solve it.

Because that's the Way we'll touch on. Yeah. Let me just say some putting regulation And penalties and all that, that's, that's nice.

That ain't gonna stop it. And, and guys, in all honesty, Mike, I gotta take issue with what you said. 'cause of the world we live in and the country we live in today, okay, we don't lock people up because they made a mistake with computer code.

People get locked up when they commit crimes. Intentional crimes, right? Let's, let's be clear about, I'd like, in all honesty, I'm, I'm not playing that game no more.

We're not locking people up for, for negligence or computer stuff. I would point out, I would point out if you drive your car and you know that the brakes are not working and you smash into something, you will go to jail. So that, so that, that is, well, not if you smash into something, you don't go to jail necessarily.

If you hurt something or kill someone, you do. Right? And they, they call that negligent homicide.

So, So, But, But legally there's a whole hierarchy there. I'm sorry I Jack, but I, But I, I think, I think you have a, both you and Mitch have a valid point where Apple created a walled garden and Apple was responsible for the trust. So you trusted Apple, apple enforced it, and you went from there.

Mm-hmm. Google's approach today is there's a Google walled garden. But if you want to install something from somewhere else outside of the Google Trust in search circle of trust, you can make the decision.

We've told you we haven't vetted this, right? Personal responsibility. Personal responsibility right now, software development in general is all on the personal responsibility.

It's up to you. And there's no circle of trust for you to go to. And enterprising company or entrepreneurs could go and create that and say, we'll, take the open source repositories, we'll validate it, and you come to our repository and you pay us to come to our repository because we've done this extra level of work to create this circle of trust.

That's one possible solution to this problem. It's not. And just with everything else in security, it is maybe necessary, but not sufficient.

It doesn't solve the entire problem, but it goes a good step forward. It, it's a, it is a step forward. I will tell you my last thing, and I'll put this back on the free market.

If people are selling you insecure software, don't renew the license. Got insecure software boycott, cancel your subscription to insecure software's. That's the mantra.

Hey, we gotta, we gotta take a break here. I'm gonna leave you with one thing on this. Fear is the mind killer.

You're watching Techstrong gang Discover Techron Group, the epicenter of tech innovation. We are your go-to for reaching IT leaders and practitioners worldwide. Our secret impactful content that sparks awareness, engagement, and top quality leads with us.

You'll access editorial websites, streaming videos, virtual events, custom content analyst research, and more. Join our satisfied clients. Let's revolutionize your tech journey.

Contact us today and tell your story to the world in the most powerful way with Textron Group. Hey folks, we're back and there's a war on in a place that we can't see it. It's up in space and involves all these GPS satellites and most of this issue seems to be emanating out of the conflict in Ukraine.

But Jack, you have an article about this over Security Boulevard. Educate us what's going on here? Well, to no surprise to if, unless you've been, you know, sleeping for the last 5, 6, 7, 10, 10 years, there's a war between Russia and Ukraine.

And no surprise Russia is playing dirty tricks in jamming GPS and Russia being Russia, they're sort of a little bit or maybe a lot indiscriminate. So basically just saying, we don't care, we're just gonna blanket the entire region with jamming signals and corrupting the, in the GPS signals. Uh, civilian GPS is insecure by design and is also used everywhere.

And not only by your cars and your phones, but much more critically by airplanes to navigate. And in fact, you can use GPS to pretty much land a commercial jetliner today, except when GPS signals are jam. Don't third use it.

We wish. Yeah. Um, but this essentially goes back to sort of a continuation of the discussion we had earlier, which is if you don't design security in from the beginning, you're going to have problems.

In this case, we have a very big GPS problem where the jamming extends out from the Ukraine to cover all of the Baltics. And in fact, even Sweden is reporting an increase in loss of GPS signals in airplanes and other things. And it's a very huge problem.

I mean, this is life critical, mission critical stuff. When a plane is trying to land and it doesn't have GPS anymore, it potentially lands in the field and crashes rather than lands on the runway 500 feet to the right or to the left, right? Yeah.

And there, there are other uses of GPS that there are pretty mission critical too, right? Yeah. It's all around emergency servers, as you name it, everything.

But let me Go ahead. I'm sorry. You go, Jack.

No, Well, I was just gonna say my, I I'm looking at both GPS, but I take this as look at the broader picture, right? Which is really going back to a little bit of the discussion in the, in the, a block of thinking about security, security at day one. Security should never be a bolt on afterthought, right?

It's, we have to design security into anything and everything we do. And whether it's GPS or how do you print, right? If you're developing software, hardware, it products, you need to think about security at the very beginning.

Okay? So I, I think this is more about something else we spoke about before, which is about fighting this year's, this year's war with last year's Tactics. Absolutely.

And I, and I think, look, the loss of life in this war is horrendous. I, I, I'm, I feel bad for the Ukrainians as well as the Russians and all their proxies who have lost lives in this war. But when we look at the history of this war, much like, I don't know how many of you're a fan of history.

I am, I'm a history major, you know, they say that the Franco War in Spain, the Spanish Civil War in the thirties and twenties was a precursor for a lot of the weaponry that was used in World War ii, especially by Germany and Italy, right? They were testing, it was a great test lab. And for tactics and technologies that found its way into World War ii, I think we're gonna see the same thing here.

I hope we don't have another World War ii, but we're gonna see the same thing here in the Russia, Ukraine War. Things like our over reliance on technology, which is not secure like GPS and the ability to take that off as a chip on the battlefield is, is paramount. Things like cheap drones, unmanned drones, autonomous drones make having billion dollar air aircraft obsolete with people driving them, right?

Oh, there's been a lot of new tactics and new, new technologies that are being deployed here on both sides that I think the folks in war colleges are gonna be digesting for years in terms of how do you fight the next war? Maybe, maybe, you know, look here in the US we pride ourselves, we make the best damn weapons in the world, don't we? No one makes a plane a fight, a plane like us.

No one makes a tank like us. Damn. No one touches our missiles.

But have we have, we made an over reliance on, on vulnerable technology in our defense or war fighting ability that leaves us vulnerable to very kinda low level jamming like this, right? And we sleep under the security of that blanket to quote, semi quote Jack Nicholson in a few Good Men, right? We sleep under the security of that technology and is the problem To, to, to use Minter's phrase, we are ironically literally fighting this war with both next year's weapons and last C'S weapons.

So Russia has chosen to degrade the Western technology capability by jamming GPS, which affects the high Mars rockets and affects drones. GPS guided drones, Ukraine is fighting back with. And, and sort of the, the impetus for looking at this is new technology, which is using laser communications to communicate to the drones, right?

Also, fiber optics. They had literally, a drone will have a 10 mile spool of incredibly thin fiber optics that it, that it spools out as it goes to attack the Russians from the Ukrainian side. And there are areas in the fields that this battle is bought that it now look, the, the, that it now looks like a spider's web of all these trails of thin fiber optic cables lying on the ground or caught in trees.

Russia, on the other hand, is using artillery pieces and tanks that dig back from World War ii, right? 1940 1950s tanks and artillery, because they aren't technolo, they can't be interrupted by GPS jamming. It's, it's a ballistic shell.

You fire it and it goes where it wants to go and that's it. And, you know, and, and the well, But they're doing that outta necessity, Jack. 'cause there are other stuff got blown up early on.

I, I agree with you. I'm not, I'm not saying well, you know, it's, it is what it is. Now the other part of this that gets interesting is the loss of life, which I think is abhorrent, but talking about history, Russia has always had a historical basis of treating its own civilian population as cannon fodder.

That's actually where the term sort of comes from, is we have 300 million people in the country and we don't mind if we kill 10 or 20 million of 'em in order to get what we want. It's okay. They're just peasants in the middle of, you know, the, the, the Siberian Peninsula.

Why, what do we care? Right? And so the, they, the more people they can throw at the problem, they're happy.

And the western world doesn't wanna fight a war that way. We don't like the death and the indiscriminate killing. And so we're trying, we're always trying to find other means technological rather than human to solve this problem.

Agreed. It's a shame. It is, it Is.

So do you guys think, though, ultimately, and, you know, coming back to this GPS thing, let's say that there was some sort of conflict involving another country to another country, is the GPS thing the first thing that's gonna go, because that's the communication. Yeah, no, I, I I think that is the new, you know, you, you want to black out your enemy like I do. I think the Israelis did this in attacking Iran.

Absolutely. Absolutely. You know what I, you know what I'm gonna go buy, I'm gonna go buy some R McNally maps, you know, the kind that used to stick in your pocket.

So you just Have the question is, once you open it, can you fold it back up, Mike? That's always the problem with those maps. Um, alright, well here's The 3D book kind.

It's okay. Oh, let's take a break. We're gonna come back on our C block here.

Mike took a little field field trip to Mongo. No, not the Congo, Mongo MongoDB. We're coming back at you.

com is the leading resource for news analysis and education on challenges facing the cybersecurity industry. com covers all aspects of cybersecurity, including data security, DevSecOps, cloud security, application security, network security, security threats, and more. com has the largest selection of security content featuring breaking news, blog posts, podcasts, and more.

com to learn more. com. Home of security bloggers network.

All right, to Alan's point, yes, I did go visit a friends at MongoDB. They had a developer event last week, and it was interesting. They were talking about three things.

The first thing wouldn't surprise you, they're gonna embed search and vector search into the document database, and that makes it easier to manage and you can have all this stuff in one place. The second thing though, they're also noting that you can now use AI agents to reverse engineer other applications faster. So you can do these, uh, application modernization projects that might have taken a year and a half can now be done in maybe a more reasonable four to five month window.

We'll see how that plays out. But here was the most interesting thing in my mind. They were also talking about this whole notion of context engineering.

And they're talking about the fact that one of the reasons that a lot of these AI projects are failing is because we're just slamming data into these things and there's not enough context to drive a workflow. Their argument is, is we need smaller chunks of data thrown into these LLMs and other, uh, processing engines that are, uh, more efficient. So that I'm not doing these massive amounts of processing, but I'm doing it in a way where there's context remains as I kind of do what they call a nested dollar approach as I'm processing things.

So these smaller, uh, chunks is technical term. I know, um, we can maintain the relationship between these things, AKA known the context. And so what they're really saying is databases are gonna be one of those shovels that drive AI and the, the goal rush associated with it, but we need a better shovel.

So Mitch, I think other folks are starting to talk about similar concepts and ideas, but do we need a different way about thinking about databases to make this AI thing really work? I think that's why, you know, you, you hear the term context engineering, right? Beyond just more prompt engineering.

And I think all of us who've worked with AI know that instructions are just part, part of the answer, right? It's now here's, here's the context in which I want you to perform this task or analyze this data or search this information. So if we review, if we review the data that AI has access to, it's just a giant repository of everything in the world.

Kind of like we think about how the model's been trained, right? They've been trained on the entire internet. Well, little, little more nuanced than that, right?

It's, it's contextualized in a much different way. We have to contextualize how we use ai. So a bot or a prompt or whatever agent, whatever it is, needs more more understanding of what you're trying to do.

And that's why you see even in the consumer products, right, that we do with chat, GPT has had, uh, memory for quite a while and now you see Gemini and also Claude, uh, coming out with memory, meaning it keeps, some memories are about your past, uh, actions, things that you prefer. Some of 'em are intentionally saved, some of 'em just your prompts, it's a combination of things. But that's all to add context to what you're doing.

So I think what Mongo is saying is, look, we can be the repository for your data. Anybody can do that. How do you do this in a much more intelligent kind of context aware way?

So that way, and, and also use structure when you need to use structure. So for example, they talked about JSON. JSON is is actually a really good thing for AI because it, it takes what we like to, to type in as natural language and it puts a structure around it, makes you really clear what it is, um, sort of a lexicon, um, some definition around it.

And it helps it, it gives more structure to what it's doing. So I think a lot of what we're doing today is experimenting with prompting, and we're gonna learn much more about how to do better contexting around prompting. Can I, can I, I just wanna make sure I got this right.

A company that sells small databases for unstructured data says what we need to make our AI models better, are smaller databases with unstructured data. Is that, that are processed, that are processed more efficiently? Well, that they process more efficiently.

If you use a smaller database with unstructured data, and by the way, we sell that, so, so you, Well, it doesn't have to be a smaller database, right? Document databases can get pretty large these days, and I string those together in a way that makes them, you know, feel like one logical entity. So I don't think size is what matters here, but yeah, Alan, you make, you make It sound, I've spoken like a troop, like a trooper.

I think there are people who would disagree with you, but I'll leave it at that Wishful in that. Jack, go ahead. I, I think it's all about the effort.

As someone once said, prove me wrong, but go ahead, Jack Conversation. Of course, Jack, take us outta here. Come on.

You make it sound like A, a conspiracy theory, Alan, but I believe that there, there really is some validity to what they're saying. So I'm, as you can tell from the hotel background, I'm on the west coast today and this week for a, uh, securities field day. And last week I attended a teleport event and one of the presenters there told a little story, which I think is applicable here.

And it's basically, you know, I used to go, you know, visit a town and you'd say, okay, I wanna find a place to eat. So you go and you hit Yelp or something, you say, go give me a list of my favorite types of restaurants. And you do a manual search and you hit Google and you look at reviews and you say, okay, now Google is this restaurant open right now.

And eventually make a decision. That's how we all used to do it. It was very manual intensive process.

Now you basically go to your little app, AI app on your phone and you just say, yeah, hey, I'm interested in a restaurant. Find me a restaurant. The app has context and history that says, I know what type of food you've chosen in the past, and I know where you're located, and I know what time of day it is, and I can take all of that information, that additional context to the query without you having to do that.

And it's those little pieces of information that you need to have give to the ai. The problem today is the AI databases are designed for a specific task. And that's the big stuff, right?

That's rag retrieval, augmented generation, where you have a very big set of stuff that you're looking at all of the data that's in a company, right? So you're, you know, gigabytes and terabytes of data. And here what we need to do is track little tiny pieces of information that just, uh, make the prompt and the AI work better.

So I think they have a point, maybe whether they're the right people for that. That's a different question. Well, Maybe I think it's prevagen for AI agents, if you think about it.

So, but Like, come on. You know, and I remember when you used your photo guide, you didn't have Google to make recommendations and all those things, but, or you just stopped it was ever was open and you took your chances and you found new places like that. It was after the adventure.

But let me, let me come back to our friends at Mongo. You know, they're a little late to the party because I think a lot of the people I know who were looking at operationalizing AI and using ai, you know, had the idea of creating small language modules or vector database kind of information, exactly what we're talking about, to give it more context, to give it more, um, subject matter expertise on a given narrow, uh, field that you won't get when you just use that big LLM. And I think for a long time, Mongo was sort of deaf, dumb, and blind to quote Pete Townsend, right?

Um, to this use case. And now here we are, and Mitch, you know this, you're shaking your head. You were at that AI operationalizing AI workshop we did those two years ago, Then you quoted Tommy, so you got me on both.

Well, you, you, you got you on that one. But now, two years later, all of a sudden they're saying, oh yeah, we should use that vector database or vector search. Well, let's let, let, let's be fair.

They had us, they had a vector database of their own or a vector search capability. They just didn't embed it in the database. They had it as kind of ancillary kind of thing.

It's not like they've been blind to the use case. And a lot of folks do use Mongo because of what Mitch was pointing out. It's JSON, the JSOI don't have a whole lot of conversion.

Most of those AI models are speaking JSO already. So to that end, I think, you know, as far as the convergences, yeah, that could have been done faster. Mm-hmm.

But I don't think they've been blind to the use case. They're, you know, if I go look at a lot of these AI projects, you're gonna find MongoDB in there. I think what's curious to me about, I invert What Announced, I'm sorry, Mitch.

Now what, what's curious to me, uh, Mike, and I'm not sure if I, if I caught enough to really put all the pieces together, but what Mongo also said is that repositioning databases from being a data store that you access data from is to, as an AI platform for workflow and for agents that combines like, you know, native vector search with semantic retrieval and, you know, lots of, you know, kind of nice language around, you know, technical terms, queryable, uh, queryable encryption, things like that. I wasn't quite sure that I got the, the transition from data to orchestration or data to AI platform. Um, did, did, did that jbe with you at all?

Can you fill in the gaps for me at all? I think that, you know, they will have to decide to what degree they wanna orchestrate those AI agents versus just kind of expose data more dynamically to them. Mm-hmm.

And there's a level of orchestration that's required to do that. But I imagine that they will probably expose their database to the orchestration layer APIs to provide more of that coordination, because there'll be the agent, and then there'll be the data that has to be accessed, and the data will have to be in something that feels like persistent memory. And I think that they're saying that, you know, you're gonna cache the crap out of a, uh, MongoDB database to provide that memory.

And that they're gonna provide, I guess I would call it the metadata around the memory so that they has that kind of awareness in the context. So be the data in whatever forms, memory, context, data source, all of those forms that, that would fuel an orchestration layer or workflows that agents do. Is, is that what you're saying?

Yeah. I believe that that's what they, and as well, I don't think they want be the orchestration layer for the agents themselves. Okay.

Right. That confused me. Okay.

Thanks. That, that helps a lot. But Yeah, I'll credit you in my next paper.

Yeah. But of course, you know, of course next week I'll turn around and they'll buy some orchestration framework and I'll Yeah, true. Yeah, exactly.

Interesting. Okay. Thanks for the edge and all, all kidding aside, you know, it is quite the, a its bit of an unknown story or an under-reported story.

Exactly how big a powerhouse Mongo is. MongoDB is in the ai Yeah. In the AI space.

If you, if you're a certain, if you're a certain database company owner who's recently seen his fortune go up by 34% because he's trading GPUs, may, maybe that makes Mongo a target for some of that money. Trump change. I I think that they're, all the database companies are gonna have to address this issue, and everything's gonna have to be in these kind of smaller, dynamically processed chunks.

I don't, so I'm not quite clear that one of them is gonna be, uh, better than the other. I just think that the way we consume that data needs to be changed, and we need to think that through, because that's what's holding up a lot of these projects in the enterprise, right? Yeah.

They're just not, I do think Mongo's the new MySQL, it is sort of the new default when you're gonna move to something beyond. And, you know, what happened to MySQL? Well, I, I wasn't trying to foretell that, but maybe what, um, but it, it is widely used and not just for ai, but it's used in a lot of cases.

You see sort of the default on lot of open source and a lot of different projects is in database. It's really the, the Mongos Mon Mongo, excuse me. Yeah.

And that's because developers are picking the database. And developers don't really like relational databases. They're big, hard, and clunky and hard to manage and document databases are easier for them to deal with.

Whether the IT team likes that when they get handed the app is a whole nother conversation. Hmm. Mm-hmm.

Excellent. Gentlemen, I think we gotta call it a, a wrap on this, Jack. Enjoy the West Coast.

Uh, I, I will be joining, uh, the tech field day, I believe on the 24th, isn't it? I'm, I'm not gonna be there in person. Uh, I'm gonna be remote, but, uh, I'll see you on there.

It should be fun. Yep. Um, I won't be on tomorrow's show.

I'll be observing the Jewish holiday of Rosh Hashanah, or is when I, my grandmother taught me Rosh Hashan Hashanah. But, uh, you know, it is the Jewish, uh, new Year and start of the Holly High Holy Days. And I'll, I'll be off for that.

But Mike, you'll, you'll drive the ship. I'll be here. And I'm, I assume you'll be dreaming up new fancy malware names.

'cause you know, now that you've got a new hobby, Mike, I'm gonna, I'm, yeah, I'm going, I'm going think about those. I'm gonna Come up with some good one. Mike, why the, why are the cat's away?

Just to plant a seed of an idea, let's come up with some intro theme music, like they do at the sporting events. And if Jack's on, on our Next show, I wanna make sure he gets Indiana Jones like they do at Yankee Stadium. When you come up, that's when they bring in Mariano.

Exactly know, it's funny. So I was at Swamp Up, uh, last week or the week before, and they, they, they had a gal or dinner, and they had a celloist as the, as the town, you know, as the entertainment. I'm saying to myself, a celloist, my God, this is gonna be boring.

What am I doing here? And turns out she's a very famous woman down in la She does a lot of movie soundtracks, Mitch, including Dune. And she played the theme from Dune on her cello, accompanied by some musicians.

And it was amazing. Amazing. Wow.

That sounds awesome. It was really cool. Really cool.

I'll find out about her. All right. Hey, enjoy the rest of your day.

Enjoy tech drunk TV immediately following this. And by the way, on that tech field day, you'll be able to watch it live too, right here on Techstrong tv. I'm Alan Shimmel.

We're out. Hey everyone, it's Alan Shimmel. Welcome back here to Techstrong tv.

Our next guest is Edan Ekman. Uh, first of all, we gotta give Edan a a Mazel t He just had his first child, a boy last weekend, so we are thrilled to have him join the, uh, the parents club. So, ed, thank you so Much.

Patience. Thank you. Thank you for joining us.

So very exciting times for you having a baby, your company coming outta stealth. You know, savor my, my advice to you, having been there, done that, right on both counts. Savor every second of this time because one day, 20 years from now, you're gonna look back and say, wow, those were the best days of my life.

Um, so good for you. Good for you. Thank you So much.

Yeah. We're not gonna talk a lot about the baby right now. We'll save that for when you're the family, but we're gonna talk about COI security, but even before we talk about coy security, let's talk a little bit about you.

You're a co-founder there, and CTOI believe, correct? Yeah. Give people a little bit of a sense of your journey, how you got to here.

So I started my journey in the, uh, intelligence first of the IDF. I did, uh, uh, cybersecurity there for about six years. And I was in charge of the security training for, uh, for the intelligence community.

And then most of, after I was released from the military service, I dealt with incident response. Almost my entire career. I was a consultant, one of the first employees in, uh, Signia.

And after that, I had my old firm. And I really, really loved the human aspect of it. Like working with security practitioners many times.

It was in a very, in very hard times for the organizations. 'cause incident is a difficult time, uh, but also other times, uh, preparing for the incident, trying to prevent it. And I think, uh, it was fascinating to see how different organizations think about security.

What are the priorities of, uh, different security teams and so on. And then, uh, a few years ago I started working in a startup called, uh, Canon Security that was acquired by Zscaler. I worked as the head of security research in Zscaler for a while.

And almost two years ago I founded, uh, COI Security with, uh, my two co-founders. Excellent. So, so you guys have been working at Coi Fru working at COI for two years already to get to this point.

Um, you know, even founding a company is not something one does lightly. Right. Talk to us a little bit about kinda what, what was the mode of, you know, what, what drove you, what was the passion in finding and founding Koi?

What, you know, every, I speak to a lot of founders. I'm multiple serial founder myself. Right?

What, what drove you, what was, in what way were you trying to kind of make the world better solve problems? So I think for me it's, uh, mainly two things. Uh, one of them is creating, uh, a great security product because, uh, during my career as a security consultant, I work with a lots and lots of security products.

And many of them feel like they're not, you're not even supposed to use them 'cause they're really hard to use. And the interface is, is low and it doesn't look so well. And it was really frustrating both for me and for the practitioners I was working with.

And I really had a, a passion of creating a security product that is almost like a B2C app. 'cause it should look good. It should be comfortable, it should solve a, a real problem.

And we knew there was a problem with, uh, software on, uh, endpoints that, uh, we were really passionate in solving, and we wanted to, to create a product that is loved by its users. And the second thing is that I really wanted to create a, a place that is a good place to work, because I knew that in my career when I worked in good companies, it was a tremendous experience. The people you meet and the experiences you make together, Make sure absolutely.

I, uh, you know, as I co-founded several companies and text Strum being the, the last one here, uh, I agree with you, right. Having a sense of we're all in it together. You know, as you grow, it's hard to keep that feeling of family, if you will.

Yeah. Right. But you want make it, and you want to make it a fun place.

You wanna make it a place where people want to come to the office, they don't feel like they're coming to a prison cell, you know? And Absolutely. So yeah, I, I try really hard with that too.

So when we talk about software on the endpoint, right? As a potential security mm-hmm. Potential security.

What, what exactly. I mean, there's a lot of facets to endpoint security, right? There's, there's looking at traffic coming in and making sure no malware gets through.

There's exfiltration, there's locking down the data, locking down identity, what, you know, what aspects of endpoint security is COI involved in? I think that when, when we think about software for the endpoint, like the third thing that comes to mind, uh, for anybody that's been in the industry for a decade or or more, is probably, uh, executables, uh, DLLs binary files. Like those are the classics.

You know, uh, like the, the, the classic attack is that somebody sent you a malicious file over email. And I think that many of the current, uh, solutions for endpoint security are still solving that problem. But this is something that happens less and less as time moves on, because we do make some progress, and now we have great tools that prevent this kind of scenario.

Um, but in recent years, we found that most software that actually runs on the endpoint is actually not, not what we imagined at all. It's not the binary software. It's a high level language software.

Many times the JavaScript, for example, Chrome extensions, people think, oh, it's just extensions. It's just, it's sandboxed. It's not something serious, but it's, it's software, software nonetheless.

And extensions for the IDE for developer environments, those stuff, they can, they have high privileges. They can touch any file on the computer. So the security world doesn't treat those kinds of software as serious threats because they're not binary, they're not executable, but they still are little pieces of code or not so little.

They run some logic, some business logic on the endpoint. And we found this to be a huge gap in the current, uh, security model for the endpoints. Absolutely.

Absolutely. And, and, you know, so a lot of it is browser based security, right? So that's what we're talking about.

Mm-hmm. Mm-hmm. Because the browser, more and more, the browser is the interface that users use on the endpoint, and it, it's, it interfaces with the software on the machine.

It interfaces with software, you know, up in the cloud everywhere now. So is that where COI plays, then? You're locking down sort of the browser, uh, browser extensions, browser interactions, monitoring web traffic there?

So we do provide, uh, security for browser extensions. Mm-hmm. Uh, our main focus is to secure the actual software that is installed on the endpoint.

Got it. Which can include browser extensions, but it can be, uh, NPM packages. And right now we have a huge incident in NPM Today with, did you see this?

That like 140 packages or something? I saw. It's crazy.

And we have an, uh, updating incident page in our website to help practitioners monitor, because new packages are discovered to be, uh, compromised, uh, by the minute. Uh, so this, uh, uh, a great example of how those type, the new type of software, NPM can be marvel for the endpoint, not only endangering the, uh, CITD of the production. You know, I was reading this morning before we got on here, um, evidently I think CrowdStrike is rolling back some of the packages to previous versions that didn't contain the malware.

Mm-hmm. But if, if you're watching this and you're not aware of it, you should get on top of this immediately. Absolutely.

Right. Make sure you don't have any of these packages in there, or maybe go use COI software for this. Right?

I mean, this is, this is exactly, you know, the postage shop for what you guys are, are trying to prevent, right? Yeah. And unfortunately, uh, we couldn't have asked for a better demonstration.

Um, and even people that are not, uh, using coil coil customers who can reach out to us, and we'll be happy to help, uh, understanding the threat and mitigating Yeah. This is a serious thing. And, and, and I'm not picking on CrowdStrike, right?

I, I know George for many years. I CrowdStrike's a great company, but when it happens to a security company, it's like a double, double whammy, you know? A double double.

Yeah, Absolutely. I don't think it's, it's definitely not CrowdStrike's fault. There are no, uh, sec not many security tools, uh, today that can stop this kind of threat because it's a new type of form.

Uh, but it just shows that if it happened to the people that know security best, then it can really happen to anyone. Absolutely. And it, And, and it also brings up the whole software supply chain.

And, and, you know, you've gotta know what, what's going into the software you're using, right. Or packages and so forth. And I, I think a lot of people, a lot of end users are shielded from that, right?

Whether you use Windows or Mac, you don't think of packages or Yeah, we don't even think much. I mean, we think of applications maybe, but you know, it, it, it's not like, it's not like you're installing Linux on a desktop and you're picking what packages I want in my in store, right? Where you're very, you could be very deliberate about what you want and don't want on the machine.

Uh, most of us, you know, we, we click the button and we download and we get what we get with it. And, and it, it's, it's scary. Um, it, and we'd be, you know, negligent If we didn't mention that Coy came out of stealth just recently now, and, you know, with a very, very big, a round seed round.

Why don't you tell us about that a little bit? So the way we started COI is we built the risk engine first, and then we started working with, uh, paying customers. And we got the Fortune 500 companies, uh, contacting us because we had this type of risk engine that nobody else in the industry had.

So it really helped us raise, uh, significant sums of money because the value was already proven when we, uh, stopped our seed round and then promptly after we moved on to raising the RA round, uh, uh, $38 million. Uh, that's the a who total funding is, uh, 48 million. And this is great for us because we really see there is a huge gap here and a big opportunity.

And we're very determined in, uh, building a product that mitigates this opportunity. But, uh, you need firepower to do that. 'cause it, this is a really big challenge.

Absolutely. Absolutely. Um, so congratulations.

You know, in today's world, it's not COVID times anymore. VCs have tightened up with the monies. They, it, they don't write checks like they used to.

So to raise that kind of seed, a round combination seed and a round is quite an accomplishment. So congratulations to you and your co-founders and the whole, the whole company. You know, it begs the question, well, what are you gonna do with this money?

Right? A lot of people out here have never been a founder, have never done the startup thing. They say, oh God, they raised $38 million.

They must be rich. Well, no, it don't work like that. Right?

We, I wish it did. But really, what did think of it as going to the gas station, right? And you're now filled up the tank with fuel, and the question is, where do you drive?

Where are you gonna drive? Where are you going next week? Exactly.

You got tank of fuel. Um, so where, where does COI go next here? So we, we started as a solution for a visual Studio code and Chrome ideas and puzzles, uh, which, which was great.

And we used the seed round funding to raise great l and d team, uh, founding engineers, really great guys, and, uh, of course a research team. And, uh, now that we have the A round money, our main focus is, uh, two things. One of them is, uh, creating the, uh, GTM team.

Uh, that is really important for our company to scale. 'cause now we know that we have a, a great product that, uh, we need to get out there. And the second thing is to, uh, to hire more people for r and d and start tackling, uh, the harder, uh, problems.

Uh, we already have support for NPM and Pipe E, uh, but those marketplaces are, uh, at different scale and we need to build, uh, we need to build for much, much, uh, larger scale because, uh, we are very ambitious in the, uh, problems that we want to solve. We want to be a single platform for managing really anything on the endpoint, any type of software That's ambitious. Let's talk about, go to market a little bit to GTM.

So is there, like, what's the model? You're gonna hire salespeople to go sell this, you're just going to create market demand from sort of outbound marketing. Is there a geographic focus?

Is there a particular verticals that you're aiming at? Right. How, how are you going to market?

Actually, something that is, uh, pretty unique, uh, at COI is that we get lots of, uh, inbound traction because, because of our risk engine, uh, we keep finding malware all the time in all of the marketplaces that we operate in, and we publish about it. So the risk becomes very apparent for many, uh, security, uh, professionals. And then they reach out to us.

So, uh, they marketing of kind of created itself just by the research work that we're doing. And, And I assume it's responsible disclosure and all of those things, right? Yeah, Yeah, absolutely.

So we report anything that we find, but since many times it's just malicious items, like malicious packages, malicious, uh, extensions or, uh, modules, then we don't need to responsibly disclose to the threat actor. Many times we do communicate with the threat actor, but Yeah, I get it. Um, that, that's fantastic.

How can people, you know, stay on top of your research? How can they, is there like a website, a, a service? How can they stay on top?

We have our blog at, uh, call security slash blog, and we have a very active, uh, Twitter or x uh, page. We have active LinkedIn page, so we publish all the time On There, usually every week. Very cool.

So if I had to ask you, what's the one, I'm sorry, what? One last thing that I really want to mention is that we all also have something called Codex, which is our community project and Codex, it's like the index by coi. And you can, it's like virus total, but for extensions and items, you can search any extension on Codex and get a detailed risk report.

Uh, even if you're not currently a customer of coi, you can use that. It's a research that didn't away just, uh, to help people get, uh, safer. How, how do people get to that?

What's, is there a URL something? How, how do they look? Guide up?

It's, uh, Dex, that's DEX, do co security. Got it. Excellent.

That's a cool, that's very cool. Um, we don't have a lot of time. One last question for you, Aiden.

What's the, what's the biggest problem you're looking to solve in r and d now that you have this a round money, let's say, over the next six months to a year? Oh, that's a great question. So, building, uh, a risk engine that can scan, uh, millions of, uh, binary software is something that we are aiming at and that we're actively working on.

And this is a challenge. Absolutely. Very good.

Edan again, congratulations on, on the baby. Congratulations on the raise. Congratulations on coi.

As I said, take a moment, maybe smell the roses, enjoy the moment, and, uh, come back and talk to us here soon. More on Text Drunk tv. Okay.

Thank you so much, Ann. All right. Yan Dart Dark.

Gonna make sure I get this right. Dot Kaman. Yeah.

Okay. Co-founder CTO at Coy Security. That's COI security here on Tech Drunk tv.

We're gonna take a break. We'll be back in a minute. Hello and welcome to the latest edition of the Techstrong that AI Leadership series.

I'm your host, Mike, er today with Brian Moore, CEO for Voxel 51. And we're talking about why a lot of these visual AI projects seem to be failing. Brian, welcome to the show.

Thanks for having me, Mike. We see these use cases all the time. I think most commonly people are seeing, uh, vision applications and everything from factory floors to cars, but a lot of the efforts underway seem to be still experimental and people are struggling.

What's your assessment of what's going on here? Yeah, definitely. So first of all, just to generalize it to all of ai, I think that's kind of the state of play in 2025.

You know, we've had studies from places like Harvard Business Review sharing that something like 80 to 90% of AI initiatives within enterprises are not yet reaching production. You could call that a failure. I would just call that kind of expected or par for the course.

You know, this is new technology. There's lots of rapid innovation, there's a lot of excitement to try new things and build proof of concepts. Unsurprisingly, uh, something that you cobble together in a few weeks or even months, is unlikely to meet the needs of the production environment that you need to deploy that into.

And that's perhaps, you know, uh, most poignant in something like visual ai where we're talking about deploying, you know, robots or vehicles, uh, or automations that have to act in the physical world and deal with all the different sort of nuances, edge cases, strange scenarios that might crop up. So yeah, there's definitely a need to invest, you know, kind of the typical 80% of the time to get that last 20% of the way to production. Uh, but the good news is that folks are aware of that, uh, and companies like ourselves are building technology to help assist, uh, you know, practitioners of visual AI address those key, uh, needs, which we can dive deeper into and get that model ready for everything that the production Yeah, the real world, uh, will throw at it.

Alright, well, to your point on that maturity curve, where are we when it comes to vision ai? Because, um, I guess there's some unique challenges there, but what are they? Yeah, so the interesting thing about visual ai, and by visual ai, I mean, anything that has to do with image or video or 3D uh, lidar radar data, um, that's an absolutely immense data source.

Something like 90% of all of the bits that go through routers on the internet today are actually visual in nature. Uh, so the vision AI challenge is at least two orders of magnitude larger than the challenge, uh, of building models that can, for example, process text, right? So it's kind of expected that it'll take some, you know, additional time and effort to get these things, uh, ready for production.

Of course, the good news is that with all the investment going into accelerated computing infrastructure, you know, data centers, power r and d, all the things you hear about the news, uh, those advancements are coming. The, the promise of being able to feed larger scales of data into these systems, uh, is also coming. And so I would definitely expect to see continued progress on some of the kind of bulletin board vision AI use cases that everyone's familiar with, self-driving cars, humanoid robots.

But maybe most, uh, exciting to us are kind of the more incremental advancements, you know, automating specific scenarios like maybe defect detection in manufacturing context, uh, or building purpose-built expert systems that can, for example, you know, example, uh, detect the fall, uh, of a, a human in a healthcare context, uh, or automatically, you know, process, uh, a camera feed to make a decision about whether a part is, uh, high quality or low quality. Those kind of things, uh, are much more short term. Uh, and we're seeing those types of technologies get to production, which is very exciting for the, the vision AI field overall.

Mm-hmm. I think everybody's excited about the use cases, but it seems to me they're also running into issues around, well, what does it actually cost to run something in a production environment when you add up all the infrastructure and resources required? So do we need to be smarter about what projects we're gonna pick with an eye towards what's gonna go into production sooner than later?

Yeah, I, I think it's a great call out. So, you know, uh, one of the exciting things that's happening in the AI space is the progress of these, uh, so-called foundation models. The large models, you know, the GPTs, uh, coming from hyperscalers and, and those models are, uh, have a broad expertise of knowledge.

Uh, however, large models are expensive to run. Uh, and so what you can expect to see is those models knowledge getting distilled into smaller expert models that are more efficient, uh, at solving, you know, specific tasks. Uh, and so, you know, that's what's actually getting, uh, into production, uh, in vision AI especially, is these distilled models that are purpose-built for specific use cases that can run at a much more cost-effective, uh, price point.

As you kind of sort that out, who's gonna build those smaller distilled models for organizations? Is that some data science team that they hire? Or are there specialist organizations that are emerging who are gonna basically make those things available as a service?

How does this kind of manifest? Yeah, so what we're seeing is that enterprises that, um, are having the most success in visual AI are ones that bring the development of these sort of fine tuned systems in-house. They treat, uh, their AI strategy as a core part of their company's competitive advantage.

Uh, and so they want to bring as much of that development in-house as possible. That definitely means using off the shelf models, uh, data sets and so forth, uh, to sort of, you know, turbocharge their, their development. Uh, but they see their ability to develop, uh, a high quality data set, uh, and model that's an expert in their use case, uh, as being critical to their, uh, company strategy.

Mm-hmm. I'm, what are the skills that are available as it relates to this? And I'm asking the question because, well, we're already having a hard time just finding your everyday run of the mill data scientist genius, and how many of them are actually cognizant of visual AI and, uh, what does the pool of talent look like?

Yeah, so for context, uh, a little bit about myself. So I have a PhD in machine learning. Uh, VL was founded by myself and my co-founder, Jason, actually over 10 years ago, uh, initially doing consulting, uh, in, back then it wasn't called visual ai, but rather computer vision.

Uh, and so computer vision as a field, it's actually been around for quite a long time. In fact, even Nvidia as a company, uh, got its start and spent many decades focused on computer graphics, the kind of, you know, uh, low level computer vision, uh, algorithms that are necessary to, you know, build graphics engines, video games, so forth, right? So there's a rich history and, and expertise in, in the, in the market that exists in computer vision.

Uh, and, you know, so that's the good news. There's lots of, uh, you know, capability out there. And then, uh, what we're seeing is that whenever there's advancements in sort of overall AI technology, uh, those models, those architectures can be deployed not only for language use cases, but also for vision use cases.

And so the visual AI field definitely benefits from all of the advancements that are happening in, you know, large language models. Uh, as an example, the, the transformer architecture that Google created a couple years ago, uh, was very important both in language but also in vision. Mm-hmm.

Of course, it takes a village to kind of build these applications and there's developers involved and, um, data engineers and all kinds of folks, but, um, how do I meld them together? I think a lot of organizations I talk to are struggling 'cause the data science folks have one culture and the developers have a different culture and they're not quite in sync with each other about how to not just build an app, but maintain it and update it. That's a great point.

So, you know, historically what we've seen is that some, uh, companies have decided to kind of create separation between what they call their data team and then their model or product team, which might result in kind of a separation of duties where the data org is responsible for building data infrastructure, maybe gathering data, and then throwing it over the wall to the product teams that then make use of that data kind of a one-way street type of modality. Uh, however, you know, especially when we're talking about model failures, what we're seeing is that the ability to overcome model failures and actually get a system into production, like we were talking about earlier, that comes really from the iteration cycle, being able to understand, okay, I built a data set, I trained the first version of my model, where is it succeeding and where is it failing? And inevitably, if it has a failure, maybe an edge case or a certain type of bias that you've discovered, you need to address that, uh, through better data.

It's not sort of a one way street, it's an iterative process, and you're in the best position to make that kind of, uh, action to improve your model, uh, if your data and model teams are working closely together. So, as an example, uh, our software 51 that we deploy to enterprises, it kind of puts the data at the center of the entire development process of visual ai, allowing data teams and model teams to collaborate together in one place. And when they see a model's performance or lack thereof, the underlying data is always one click away.

So with just a click of a button, a user who's trying to, you know, evaluate a model can understand, oh, I see this is why the model's performing poorly. I can see that there's, uh, something I didn't expect about my data. A bunch of it is low light or low quality maybe, or it's having problems in the self-driving yeast case, uh, you know, understanding, you know, sort of crowded intersections and low light conditions.

If I'm able to go gather more examples of those problem areas, that'll be the most effective way to improve that model's performance. Mm-hmm. How readily accessible is that kind of data?

I think, you know, you hear people talking about how all the data that's publicly available aren't even sucked up. So do we have enough of this visual data to train the models going forward? Yeah, that's a great point.

So, so that, that kind of, um, quote, uh, is most often used about large language models, people are saying that, you know, the reason that let's say GPT five had a smaller delta than one might've hoped over GPT-4 is that we've already ingested all of the data that's available on the internet. That's definitely not true of visual ai. Uh, like I said before, there's, you know, 90% of all of the bits that go through routers on the internet are visual in nature.

Uh, and there's definitely a vast amount of untapped data in visual, uh, it's visual in nature, uh, that is yet to be fed to all of these models. Having said that, uh, one trend that we're seeing with our customers is, you know, kind of by definition where you need to spend all of your time are on the edge cases, uh, or failure modes of a system. And those are rare, they're hard to acquire.

And so companies like let's say Tesla are in a, a good position, uh, where they can, for example, trigger, uh, anytime there's a, a hard braking event in a vehicle, they can capture that scene and feed that data back to headquarters and use that to specifically address that, you know, sort of failure mode. So being able to connect to your development process, to the products that you're putting in the real world, that's a great way to gather more data. On the other side, we're seeing, you know, synthetic data as an example.

You know, we partnered with Nvidia, uh, to make their, um, neural reconstruction, uh, models available to our customers. That's a situation where you can generate a synthetic version of the scene, and then you can play with things like, Hey, I've got this scene. Uh, what if I swapped out that FedEx truck for a UPS truck?

What would happen then? Or what if I took this sunny scene and I wanted to consider how the model would perform if it was instead snowy or rainy? Uh, you can perform those types of, you know, uh, you know, uh, synthetically generated tunings, uh, you know, uh, from a model standpoint, which obviously gives you the ability to plug those gaps that may be hard to acquire, uh, real data for now, I would say that that's kind of a, you know, uh, up and coming technology, uh, and there's definitely interesting questions to be answered about, you know, how do you evaluate how much real versus synthetic data you need, uh, to build, you know, a production ready model.

Uh, but it's definitely something that our customers are excited to, to tap into. It also seems to me the tolerance for being wrong in these apps is a lot less, shall we say, than it is in your typical, um, you know, chat GPT type of application where, you know, if the thing hallucinates on some sort of summarization, I'll notice, but, you know, if it's not the end of the world and all shrug, but it feels like with the visual ones, that those applications are a little more, shall we say, mission critical. Is that fair?

Yeah. I think what you're getting out there is that, you know, and this is kind of maybe obvious with hindsight, but the key to getting these systems into production is choosing the right use cases. And to your point, the right use cases, especially early in the development cycle, are ones where the, the, the system or the use case can tolerate a failure.

So yeah, if you're, if the task is to summarize, uh, some content, uh, for a human to take action on, and if it's not quite right, you know, there's a human in the loop already, and so maybe it's okay, right? Uh, on the other end of the spectrum, you could see something like a self-driving car where, you know, it makes kind of intuitive sense that it needs to be, at least in order of magnitude safer than a human driver in order for us to kind of accept, uh, any sort of failures that might happen. Uh, but the good news, like I was saying before, is that in addition to the sort of, you know, bulletin board use cases for visual ai like fully self-driving or fully humanoid robots, there's a lot of, uh, smaller more sort of focused tasks like detecting defects or, you know, uh, automatically processing, you know, user imagery for insurance claims where there's a lot of summarization and sort of constrained environment tasks that can reach production level and will, you know, uh, add lots of value to us while we continue to push towards those bulletin board use cases.

Kind of similar to however, everyone spends some fraction of their time talking about what a GI will look like when in reality, uh, those use cases like, you know, automating customer calls, uh, in service centers or, you know, summarizing, uh, knowledge work, uh, for enterprise. Those are the real value creation in the short term. And to your point about that, you know, everybody talks about, well, who moved my cheese and am I gonna get laid off?

But when you look at those use cases you're talking about, some of them are things that we probably would never have done in the first place, and many more of them are things that well, nobody really enjoys doing in the first place, and we typically don't do it all that well. So is that part of the thinking about where to make use of something like vision ai? Definitely.

Right. And, and just to put a point on maybe a macro trend that's happening right now and how it's impacting the visual AI space, uh, you know, we're talking a lot these days about onshoring manufacturing for various, you know, political reasons, uh, you know, so forth, which we won't go into here. But, you know, there's this sense that, well, uh, there's, there's certain tasks, sort of menial tasks in factories and so forth that may be, you know, some fraction of US workers aren't interested in doing, or it doesn't make sense to do, uh, at sort of like human level price points.

Perfect use case for vision ai, right? We can come in and invest, uh, as we're onshoring manufacturing in building automation, uh, and they, you know, building out factories that'll put us in a, you know, a competitive advantage compared to, you know, even our, uh, offshore, uh, competition there. Well, let me ask you this then.

Um, is this really a separate discipline in the sense that there will be a separate ecosystem for it, or, you know, are you at all concerned that the open ais of the world and everybody else who's in that space is just gonna, you know, just add this to their portfolio of services? Yeah, so that, that's, um, kind of what I was getting at before when I was making a distinction between, you know, what a foundation model, uh, can do, uh, versus, you know, what's actually required to get that particular use case fully automated and in production. Uh, you know, there's, there's interesting conversations happening right now.

Let's take on the language side for a second around, you know, hey, if, uh, if it is true that these large language models have quote unquote PhD level knowledge in all fields, then why do I need to go to a healthcare provider? Why can't I just go to chat GPT and have it solved, you know, provide my diagnoses and provide a plan of action? Well, there's not, it's not just as simple as providing the knowledge.

There's expectations around, you know, quality of care and certification and so forth that come out. And for that reason, it's not gonna make sense for a single company, certainly in the short term, to provide sort of expert level guaranteed certified services in all these use cases. Uh, and I would expect the same thing to happen in Vision ai.

You know, it'll make sense for vertical specific companies that deeply understand use cases and customer needs and so forth to take a technology, a general purpose technology off the shelf, and build a solution for a specific vertical. Uh, not to mention the, the, the point I mentioned earlier around how it's not cost effective to take that general purpose model and plug it in directly. That may be sufficient to build a proof of concept or show that something can be done.

But ultimately to drive margins up, costs down, you're gonna have to invest in building more expert, you know, fine tuned systems. And that clearly has to be done by a, you know, an entity that's focused on that one vertical and ready to make that commitment. All right.

Well, folks, you heard it here. There's a lot of things to be excited about in the AI era, but maybe all the cool kids are starting to hang out in the visual AI table because that's where the new and interesting applications are gonna be. Brian, thanks for being on the show.

Thanks, Mike. All right. And thank you all for watching the latest episode of The Techstrong that AI Leadership series.

You can find this episode and others on our website. We invite you to check them all out. Until then, we'll see you next time.

Hey, everyone, we're back here at our Swamp Up 2025 coverage, beautiful Napa Valley. We, we haven't started drinking the wine yet, so don't worry it's early. But let me introduce you to our next, uh, guest here on Tech Drunk TV to my immediate left, uh, kind of a VIP guest here.

I hope I get his name right, 'cause he's the VIP Tar Tarik Shark. Perfect. Thank you.

Great to be here. Tarik Welcome. Tarik is the CEO of Sonar, and if you watch Tech Drunk TV or read any of our sites in Sonar and Sonar Source and everything, he is a pretty well known brand and company we cover.

So thank you. But I don't think I've had the pleasure of interviewing Tarek before. To my far left, I've had the pleasure of interviewing him many times.

My friend, gal Marter of of Jfr. Gentlemen, welcome to Techstrong tv. Thank you.

Thank you. Great to be So Tark. You're the VIP guest.

We're gonna let you go first. You are up on the keynote with, with Shlomi this morning, along with, uh, folks from Nvidia and Service Now. Yeah, you know, I wrote a little article that's, I think it's up already, but, um, you know, my mom always told me, show me your friends, I'll show you who you are.

Right. And, uh, where to lived By. Yeah.

Yes. And, and so that was a great grouping of, of companies up there as the CEO of Sonar. Let's start there.

Talk to us about the relationship with Jfr, how you're working together, how you're working with Nvidia and some of these other companies as well, and why that's important for our listeners and readers and watchers here. Well, um, again, thanks for having, having us on. I think, you know, we were super, um, fortunate, very grateful, uh, to Shlomi for the invitation to join today.

And, and it really, what you saw on stage with Sonar, with Jfr service now Nvidia, really is in this AI world, it's like a complete life cycle of the software, uh, of software development, right? Um, from a sonar standpoint, we start with, Hey, you are writing the code. You're a a developer, you're writing the code, you're checking code in.

How do you make sure that that, um, code is high quality, whether a developer writes it or AI is writing it, or some combination you are then, um, uh, basically giving it to Jfr to build the artifacts and to secure them and to make sure that these are, you know, that they're rock solid for you. And to provide the evidence that these are great. And doing that in concert with, um, with ServiceNow and with Nvidia.

You know, Nvidia is powering all of this, but also an amazing software development shop in its own, right? Right. And so, so we really did think, uh, one of the expressions that Sami had as CEO of Jfr was, um, too integrated to fail, right?

And I think for us, the, i the idea of, you know, we're trying to serve our customers, the customers don't want silos. They want something that just works. And that's why we were here.

That's why Sonars here, and I think I speak for the others why they're here as well. I love it, gal. You know what I, I stood I made like, just because I know you doesn't mean everyone out there helps you.

Let me give you a chance to introduce yourself and your role at Jfr, and then we'll come back to what Tarek said and, and kind of show that up. True. So I'm Gal Mud, I'm the Chief strategy officer for J Rog.

And, and the connection to what Tarik said is, uh, among other things, I'm responsible, uh, for the partnerships with other vendors, uh, within our industry, and we're honored to have Sona with us Thank you, at oneo this year. Absolutely. Now, Shlomi said a few things up there.

One, you know, that kind of struck with me. One was the singles record, single source of record, but two, what he was really talking about is when you look five years out, and look, I'm not crazy enough to think I know what's going to be five years out. None of us are not the way things are going now.

Right? Right. Lucky if we could see what's gonna happen at the first of the year.

But if you look five years out, it's, it's not gonna be one company that's your AI company, even Nvidia, for as great as they are. And, and, and their greatness is more, almost as much in their software as much as it is in their chips. Yep.

But even Nvidia, you are going to need, it's gonna take a village to run tomorrow's development shops to run tomorrow's enterprises that are AI powered, that are AI enabled, that are like turbocharged, if you will, with ai. And that's why I think it's important, breaking down those silos, right? com, but this is more than just breaking the traditional DevOps security silos.

This is really bringing the whole business together, right? And that's why it's critical how beyond, you know, nice words up on a stage, where Does the rubber meet the road? I mean, I think you have, at the end of the day, you need the people who are building the software to actually change the way they're doing something, right?

And to actually, um, to, to actually understand the changes that are happening from ai, from all these things and, and really kind of adapt. And so, you know, the, the, the notion we call, we call this sort of idea vibe, then verify that we are talking about, which is you use AI and you have to have the qual the assurance steps. You need to have the evidence, you have to have all these other pieces.

That's the verification element. And you're exactly right. This is not, there's some companies out there that are just saying, Hey, it's our platform and nobody else, right?

And I don't, I don't believe that. I won't speak for gout, right? But, but we think that, you know, we, we believe that we are really good at what we do.

We invest a lot in it. We've got several hundred people who do nothing but think about code quality and quality assurance and code security in these areas. And we think that that is great.

I have zero expertise in artifact management, right? Um, or in what ServiceNow does in the ITSM world of things like that. So in order to get the value, I think you have to go to the best of breed.

And this is what we're hearing, is that people are trying to consolidate not to one platform, but to a series of best of breed, um, uh, capabilities that work well together. So I I, I definitely agree, and I, I think that, you know, you said vibe, but verify. I I love it, by the way.

Thank you. Uh, but, but there are, uh, different aspects of software development. And the first thing that AI did was around vibe coding.

And you know, uh, no one right now codes by themselves, right? Everyone has agents, some giving it more responsibility, others less responsibility. But the reason that we even give it a chance gave it a chance, it's not a chance anymore.

A AI is here, writing code is because we had this mechanism that allowed us to distill the verification process in it, right? You had this vibe coding, then some kind of a pull request to a, a, a giving. This is the control point.

And then a, a tool, a great tool like Sonar can come and verify the quality of the code and security and, and whatever it will be. But we know, we all know it's the beginning of the journey. You said we don't know where, where it'll end up, but we already start seeing that journey extends not only to coding, but also to the release process.

Yeah. So I have no doubt in my mind that we're gonna see pipelines like CICD pipelines changing to, um, include some aspects of AI making decisions or maybe orchestrating the whole thing altogether. I don't know, as you said.

But in order to allow that, we must have the same level of trust and control points in order to allow and delegate this responsibility to ai. And what I mean by that, that goes back to what you said as a system of record. We have the system of record for coding.

We now need to have the system of record for the release process itself. And I think this is what we're doing together. That's right.

Like, yeah. Sonar is, uh, giving us the, the, the, the results of their scanning, signing them, and then connecting them to the actual artifact where being they're the system of record to track these artifacts and verify at any stage all the different things that you need to, uh, verify. So no one will get into, nothing will get into production if sonar, for example, says that the code quality is not, uh, high enough, or if any other criteria is Not met, is not met.

And, and I think that, just to build on this for one second, I think that this is becoming critical. We are past, you know, two years ago there may have been a lot of magical thinking in the AI world, right? Of this is all gonna be perfect and no one's gonna have to worry about it and, you know, everyone will be out of a job and all this stuff.

I think now we're realizing that, that the lack of trust, the lack of assurance, actually becomes an inhibitor to the adoption of AI inside of any responsible enterprise. And that's why this is, we think so important. I I, I will tell you, you know, over the course of my career, I've seen a lot in technology innovation, the internet itself, probably much like you guys have, we've seen the internet itself come.

Yeah. And cell phones and cloud and, and a lot of these innovations they focus on, okay, I'm gonna code better or secure. I've been in security 25 years myself, so we, I've seen a lot move from network security to cloud security to endpoint security.

And we still don't do every of any of it. Right. But, you know, but I've seen all of these things.

Yes. However, this ai, meaning this is so different 'cause it affects from here to there, right. I, I really think satin at Microsoft said it best Couple maybe a month or two ago now when he said, we are moving from becoming software companies to intelligence engines, right?

We, you know, mark Andreessen famously said, software is eating the world. It ain't, the world got got a little digestion maybe, but it ain't the world. But now we're moving from software factories, if you will, to AI factories, to intelligence engines.

And to do that, you need no one co I don't care. As I said before, I don't care who it is, no one company. I think this is going to, this whole AI thing is going to, it's, I was a biz deaf person for a lot of years, right?

I, chief strategy officer, all those things. This is gonna be the greatest thing for business development and strategy, because who your partners are not comes back to what I started with. Who your partners are are gonna determine who you are, right?

You've gotta build, you've gotta build that and, you know, soup to nuts kind of partnership. Um, Tarik, I'd like to come back to you, talk a little bit about what Sonar is doing. You know, I, I had your ct, we were talking off camera.

We had Andre Weeks, and he was fantastic. You guys had some new news. Thank you.

What can you share with the, with the audience, anything since then? Well, there's a, I mean, we've been doing a lot. The, the core of our business is code quality, code security, code governance, right?

Yes. Really focusing on that. One thing that really interested us, um, was, okay, if the models are the code are, are the brains of these AI agents, what kind of coders are they?

Right? 0 or clouds on at four, et cetera, they are all focused on the, what, what I call the IQ of the models, right? Just can it solve this problem?

Can it solve the math Olympiad of whatever, you know, things like this. And it kind of misses the whole question of what's the personality, right? So you never hire a developer and say, they're really smart.

They suck at security, they write really messy code, but it's really smart, so let's go ahead and do it. And we couldn't find anything that talked about that. And so one of the things we've done very recently is really do a really deep dive on, on the models and what are the personalities of these models.

And what you find is that the models are getting better. There's a little bit of a diminishing return curve that we're seeing, but, um, this question of functional completeness is only one dimension, right? And for example, the more reasoning that you put into the models, at least right now, what you find is actually you get not only diminishing returns, but you may actually start hurting things like security and maintainability.

The mo not to overly, um, personify the models, but they kind of overthink the problem. And so you think about this from a code standpoint, from a development standpoint, you're gonna end up with models that write more code. We've shown this quantitatively.

They're very verbose. Um, the cognitive and, and matic complexity goes up exponentially as you have more model. I'm more sophistication in the model model.

You've got more security issues, but the security issues are not the simple things you used to find. They're the hard things. So you could be lulled into complacency.

On the security side, same thing on the tech debt side. Same thing on the, on the bug side, right? And all of this points to, hey, the bottleneck, the really hard problem.

Now, one of them at least is going to be how do you review the code, right? Um, who reviews it? How do you review it?

How do you make it tractable? So that's something we've been spending a lot of time on. Yeah.

I mean, we saw this, right? 5 generation, we saw syntax errors. Yes.

Right? It doesn't make many syntax errors anymore. It will not make a spelling mistake, and it won't make a grammar mistake, really, right?

Yep. Yep. But the errors, it makes good, they could be pretty bad, right?

And so, and that brings the human in the loop into the whole thing, right? And, and I think that's something we're still grappling with. Yes.

Right? Is, is where exactly is that human and is that human AI assisted or because it's, you know, the more code you generate, the, the more, either the more humans you need or the faster the human has to be. Well, that's just a, and, and it iss more complex and it's more verbose.

So the job a, I don't know any software engineer who went into software development to be a copy editor for ai. Oh, right. It's just not the core skillset of these people, Nor what's gonna make them happy.

No. Nor what's gonna make them happy. And so you need the tooling and you need the trust, and you need then the verification.

I've just done all of this hard work. How do I stamp it? How do I make sure that now I'm shipping this, whether at the code level or at the artifact level, et cetera, so that you know that it's trusted.

You know, it all, it all still comes down to one word. And I learned a long time ago about software and security quality. Yeah.

It comes down to the quality crap in it's crap out and bad quality makes for bad companies. Exactly. And I, I kind of building up on that point, we create much more code and we have to somehow verify this code faster.

So it cannot be manual, of course, otherwise it won't work. And this code, you know, it's not going directly to production. It goes through a process.

And this process should be scalable enough. Otherwise we'll just create a bunch of code, but it'll not get to its destination. So it means nothing.

So we'll have to take the full process, the full software supply chain, and make sure we apply different practices, probably AI agenda, AI practices to scale the whole, uh, the whole thing. And I think, again, this is exactly where things come, uh, uh, come together. If we focus only on the left, it's gonna stay on the left and not get, And that's exactly right on The customer.

I love it. Hey, we're outta time. They're giving me dirty.

Looks out there. I'm sorry. Thanks.

I hope you've enjoyed this discussion. It was a great discussion. Tar.

Thank you. It was a pleasure meeting you. You as well.

Come back on Tech Drug team. Anytime, anytime, Anytime. Thank you so much, my Friend.

It's good to see you. Thank you. Yeah.

Ill keep doing what you're doing. You're doing a great job. It's A great show they're putting here.

So Yes. Always, always. Okay, we are here at Swamp Up.

Check it out. We're gonna have a full day of coverage today. Another full day tomorrow.

Stay tuned. com. Techron it Techron ai, uh, digital CXO Cloud native now, even Security Boulevard.

We've got Swamp Up all over the place. I'm Alan Shimel. We're back.

Hey, everyone. We're back here with day two coverage of, uh, swamp Up. I'm happy to have my next guest on here.

His name is Jonna Tan Arbell. Yo yt, I've, I've interviewed him before. He's one of the unsung hero stars here at Jfr.

He's a frog. Jonna Tan. Welcome to Tech Drunk.

Always a pleasure to be here with you, Alan, Pleasure to have you on. Before we jump into, looks like we just had a plane, Passover or something, huh? Yeah.

Before we, uh, jump into what you spoke about today, if you wouldn't mind, talk to the group, tell them a little bit about what you do with Jfr, where you've been, how you got here. Right. Uh, okay.

So, first of all, I've been with JF Rog for the last nine years, uh, doing a lot of stuff. I started as a software developer. Uh, afterwards, uh, like few years later, I was a team leader at the infrastructure group, uh, where we kind of, uh, planted the seed for the platform, what we know today as what we know for today.

Okay. Um, and later on, um, you have Jfr CTO approach and say, Hey, Anan, you're so active on the networks. You wanna be, uh, Devra and start, you know, communicate, bridging the gap between what Jfr does, you know, outside to the world.

So, you know, kind of try to simplify things. 'cause jfo, you know, we do a lot of stuff from the most left side to the right side. There is a lot to cover.

And we need to somehow order all this building blocks, so it'll be easier to, you know, to grasp what we are doing. Excellent. Now, so you moved into from software development to Derel, Right?

Right. I moved in from software development to Derel, but, But always a, but Yeah. Um, it's, so I'm, I'm, I'm so grateful that I can work with yo because it is my CDO.

But not only that, because, uh, giving me the freedom to also explore and keep coding, you know, once developer, always developers, developer, right? And, and when people ask me, Hey, you, you moved from a development to be a devel. I say, you know, it's like swimming.

You, you, once you're a developer, you have the, the, you have the, you know, this infrastructure you can always get back on track and, and, uh, keep coding. And I never stopped coding, actually. And in fact, JF rogs, MCP server, our first open source experimental MCP server is server that IOP I created.

And I, uh, you know, uh, uploaded to GitHub and we, you know, we started seeing some eye rolling on this, uh, on this project. And, and it got, you know, lots of our customers got really excited about this MCP capabilities. Uh, we had PRS from Nvidia and even, uh, a guy from Cisco approach and say, Hey, we use your MCP.

It was so good. We, we even take it as a reference for our MCP. So it was, first of all, it was really flattered, uh, from it.

Um, but yeah, I, I keep my hands on the, on the keyboard. We keep, I keep coding. And, you know, even today with AI is so easy.

Yes, It is. Everyone can code vibe, vibe coding. Yeah, Vibe coding.

I, you know, we, we say vibe coding. I like the, the word I add sketch coding. Okay.

Yeah. Because vibe is like, you know, I, when I think about something, I, I, I give the prompt and I, I start sketching what I want eventually. If, if I think if it's something that I, I feel that is can, can have some maturity, I will take it to the next level.

And by the way, this is some, this is kind of how I build the MCP. It start with sketching. And then I created, uh, a tool, an tool, and it opened another door.

And I like text, uh, text my manager bragging about each new capability that I just created. Hey, you know, that now you can ask about the production environment in runtime just from your id. Because I, I, it felt like a magic.

It felt That, you know, that is very close to how I use it too. 'cause I use it more for my stories and stuff like that, and articles and scripts. And, um, I also, I, I never called it sketching, but that's what I do, is I, I sketch it out in an outline and then I keep putting more flesh on the bone.

Right. And then I ask it to polish it up. Yeah.

Um, I wanted to come back to, and, and talk a little bit about you speaking here. Well let, before we talk about that, let's go back to MCP server. Yeah.

It's crazy. In January, no one knew what it was. Right.

MCP server, right. Today everybody has an m it's become the standard way we're communicating. Right.

And, and you know, at first we heard MCP, like it was everywhere. And now we hear it about it, uh, a a little bit less and not because people, uh, are not using it because it built, it became the standard. It's Built.

Yeah. It's built in already. It's not, It's built in.

Yeah, exactly. Um, and, and it's amazing. And we, we see the all ecosystem around, uh, you know, how the, it's not, it's not about only, uh, creating this new shiny model the strongest with the biggest context.

It's now about how we as, as persons communicate with these AI agents and how AI agents communicate with another AI agents. So that's what it's now, it's now about the communication and interaction interaction between these Ai The ai Yeah. AI To ai.

Exactly. Alright, let's turn to your presentation. Yeah.

Tell us what you spoke about. Yeah. So in, in my presentation, uh, I talked about the journey of, uh, what Jfr did from the moment, you know, AI came into our lives.

And, and like, like we talked yesterday and like gal mentioned today, we see every billboard out there saying how they're working with AI and what they're doing, uh, in order to be an AI company. Um, so ai, it's a tool it come to, helps us to, to expedite the, the, you know, the processes. And I start by by, you know, telling the audience about what we did, uh, at first with the jf, how it's like, it was kind of, uh, an, it started as an internal gig or experiment to see, um, if we can reduce the friction between developers and documentation.

So if I'm a developer and I want to know how can I, uh, use the CLI in order to upload a new folder to Artifactory or, or add a property, I'll just do JFL and then I'll ask my question and hopefully the model will return me, uh, the right command with the right, uh, arguments. And so I can just, you know, alright, yeah, that worked for me, let's go. And that's where, um, our, it, it didn't do a lot of, uh, you know, a lot of splash out there.

Um, but yet it was a sign for us that, that the intersection between AI and DevOps does make sense. And we can, uh, you know, get these tools to help our customers and even ourselves, um, by just, you know, having the right models, the right train, uh, trained model and, and, you know, flow with it. And then we started develop more things.

And we see, like, uh, and last year when, uh, I was on stage with, uh, with my partner with crime, Brian from GitHub, and we announced the, this, um, copilot extension Yes. For Jfr, I remember. And, and it was great.

It was finally our AI agent got a context of, of external service jfr in that case, uh, it has some caveats like the add jfr that you need to add at the beginning, like telling, listen, now you will take all the information from jfr. Um, it worked and it worked well. Um, but the world needed something more.

Um, I call it like a smoother, uh, process. So, and that's where MCP came in place and kind of, I don't wanna say made it redundant, but made it much more easier. I don't need, I don't need to tell it anymore.

Hey, when I'm speaking with you about, uh, I dunno, build repositories, uh, release bundles, you should understand that you have the tools that's relevant, uh, to do this operation with jfr. So LLM is smart enough today. It has a context there in the size of a absolutely in the size of a book.

It, it, it understands the entire project. Uh, if previously it got nervous from 500 characters, now it can understand a full project. And You know what, it's every day it gets better too.

Is is the thing about it. And I think, you know, talking, we, so I talk to a lot of companies Yeah. And talk to a lot of the companies building the LLMs and the models and so forth.

Yeah. Even if they stop developing it now, it'll take us seven years to digest Yeah. What's available to us.

Yeah. Right. Using the MCP servers, using, you know, what we have.

Right. It, it's, it's an amazing time to be involved in this. Yeah, absolutely.

Where can people get more information about the MCP server and about some of the things you're working on? Right. So, uh, alongside my MCP server, and once we see the hype around it, uh, we created a production grade, uh, on in SaaS.

Like we, we, we host it, we maintain it, we update it. 1. So, um, our SAS customers can already, um, with it.

Yeah. For now it's for sas. And they can just, with, with two clicks, they can connect, uh, our MCP server with their environment, whether it's their IDE or an agents that they develop themselves.

They can, uh, connect to our MCP and we still have the experimental, uh, MCP server, which just go and type, uh, MCP, GitHub jfr, it'll get you directly there, Objective out there as Well. It'll get you directly there. So, Mr.

Devereux, where are you headed to next? Well, you know, we, we always explore and we see we, we saw this, uh, great announcement during, uh, during Swamp Up that even, even I need to explore and understand more and, and, you know, put my hands and get into these trenches and start work with it. It looks super exciting and it make lot, lots of sense.

This, uh, you know, all the announcements about the, the partners and, and everything. It's like, whoa, okay. We need to see.

It's, it's looks so amazing and we can get, we can do so much things with it. So, um, mainly explore, explore around all how we can, you know, walk with this ai, integrate everything together so everything will fall into place. I have no doubt it'll happen.

Cool. Alright. Pleasure seeing you, my friend.

Yo Tan, love you, Yan or Bell. Thank you. Dere, but always a developer.

Once a developer. Always a developer. Exactly.

Here at Jfr Swamp up. We're gonna take a quick break. We've got a lot more coming.

Your hardware-based memory is going away. Well, it's not going away, but maybe we can make your models fit in much larger space. Join me today for the Tech Field Day podcast.

Welcome to the Tech Field Day podcast, where we bring together a group of IT technical experts to discuss a single idea about key concepts in the industry. This podcast features a variety of perspectives from members of the tech Field aid delegate community, and it's often recorded in association with one of our events. Tech Field Day is part of the Futurum Group, and this podcast is also published on a sister company site Techstrong tv.

On this episode presented by Fon, we'll be discussing the premise that AI is gonna be without HBM high bandwidth memory. And that information, memory unification is changing training and inference before the discussion. Let's meet who's on the panel today.

Hi everyone, my name's Sebastian. I am with Fon. I'm the CTO.

It's a pleasure to be here and to have this conversation. Uh, FI's really excited about ai, which is a little bit unexpected because we make storage. And you would think, why is storage excited about ai?

It turns out you can do a lot with storage and AI when you have the right mindset. Hi, I'm Brian Martin, VP of AI and Data center performance at Signal 65. Spent my career around storage performance most recently ai.

So I am excited about the confluence of these two technologies coming together. And hey everybody, I'm Max Montero. I'm Chief Research Officer at Osmium Data Group, and I am, uh, absolutely bonkers about everything flash and, uh, silicon and, uh, storage.

And of course I'm Alistair Cook. I'm an event lead here at Tech Field and the event lead for the AI infrastructure Field Day, uh, series. And that's where I brought together these three people, uh, who are my guests today.

And I think we've all seen this sort of transition, uh, with storage getting faster and faster. And we saw per particularly persistent storage attached to all kinds of new faster interfaces. But I don't think we quite saw it coming that flash storage was going to replace the highest speed of RAM and high bandwidth memory that's sitting on, on your GPUs.

Uh, this is definitely one of the big cost elements for any AI deployment is buying vast numbers of huge, uh, GPUs. Uh, some of these big GPUs cost as much as a low cost car, and by the time you've filled out a server with eight of them, you've got a pretty expensive piece of hardware. You start filling racks with those.

Um, yeah, you better have deep pockets and a long, well, a big payback on that. Sebastian, you've proposed this up. What does unified storage with flesh and how's it taking away this high, high bandwidth memory requirement from ai?

It's not that it, it completely removes it because you're always gonna need, uh, some amount of dram and, and especially with the scale at which, um, high end AI works. HBM is the, the right technology for that, that field. But there, there's kind of a, a balancing act where you do need a, a large amount of active memory, but then if you are careful with how you pipeline the requests, you can actually stream in, um, any upcoming data that you will need before it's needed so that its residency in HBM does not have to be a hundred percent of the time, but rather page in, page out, just in time so that you can essentially reduce that HBM footprint and focus on the part that you need for active calculations and then use an a less expensive, but still very fast memory for, um, for holding that data while it's it's pending processing.

Yeah, no, a, a absolutely. The, uh, you know, and a lot of the work I've been looking at, uh, both, uh, in my day job working on H two hundreds, MI three hundreds, some of the bigger scale systems, uh, overflowing that memory and going to multiple GPUs to fill that up. I often run into this balance of time versus memory, or not being able to use the memory.

I, I, I, I feel guilty sometimes when I'm using eight GPUs and they're running at 40% memory capacity, I'm like, I'm not doing something right. There's something I can dial up. Uh, my home rig is an RTX 6,000 pro, uh, which I'm super happy with.

Uh, and at 96 gig of ram I can do a lot with it. But when it goes, comes time to try to train, um, the models, especially as they get a little bit larger, eight gig, 70 gig, um, I don't have the GPU for it to have the memory. So be able to flex out to a high speed, uh, persistent storage for caching that I can trade an overnight run, uh, to be able to get that done, uh, in my home web.

And, and that's the interesting part, right? I mean, depending on who you talk to, when you initially present the idea, they're like, what? And, and that's the trade off.

'cause we haven't mentioned that before. When you are paging things in and out, um, you, you're trading off time for cost, but it's, it's actually linear scaling. So instead of like, so for example, if, um, I'll just make up nice round numbers.

Let's say that you needed, um, a hundred gigabytes to do, uh, a model and to do inference training, and that would take you one hour if you were to, and, and that, let's just say that that was two GPUs. So if you remove it and you reduce it down to one GPU, you have half the amount of memory that would be required. So you actually have to do things in, in batches and, and the trade off is linear.

So if one, if two GPUs would do the training in one hour, one GPU will do the training in two hours. Now for some companies, they need just, you know, they, they need as fast as possible, and that's fine that that is a perfectly valid use case. But there are a lot of other use cases where it doesn't matter if it's, you know, if it takes four or five hours, it's fine for that particular use case.

And so our, our proposal with what F has, which is our, our adaptive solution, which has a few different features, what we're basically saying is, look, if, if this is more, if you wanna balance your cost and you wanna balance, um, you know, your, your compute and the amount of memory that you need, it's, you know, without our solution, you really have no choice. Everything has to fit in memory or it doesn't run with our solution. You now have a choice for, you know, scenarios where you, you need that speed, absolutely go the full GPU 100% HPM solution, but for other scenarios where it can take a little longer, this now gives you a lot of options for choosing how to right size your deployment to match your needs and your budget.

Yeah. And, and, and, and, and Sebastian one, one of the things that I understood because I was at AI in Infras Field day two where you were presenting, uh, adaptive, the, the solution is that if, if you are in a greenfield deployment, when you're just starting from the ground up, you have the opportunity to kind of architect everything so that it fits your need. But here, I think that the value comes from the fact that you can use that to kind of plug it into something which was already done before, and you can add in a very cost effective way, you can, you know, address those challenges of scalability of, you know, parallelizing and so on, right?

Yeah. Uh, another way to look at it is, um, if you have a scenario where, um, you already have your deployment and the, the primary use case we'll say, is an 8 billion parameter model, and then you have one department within your company that needs to, needs the, the complexity, the nuance that's available with a 70 billion parameter model, you don't have to buy it for the one use case that happens once a month. You can scale your equipment for your average use case.

And then when you have something that requires more capability, this is where adaptive comes in, lets you just basically offload that extra memory requirement to these, uh, uh, high endurance SSDs. We call ours the AI 100. Um, but, uh, the idea here is that, you know, you, you essentially scale for your, your primary use case, and then you, you, um, you leverage other memories to, to get to those, uh, special use cases, Right?

It's, you know, with, with the GPU hardware, as you said earlier, it prevents customers from having to buy equipment to satisfy their absolute max use case and then run less than when they're not doing that. And I think there's a, there's an important distinction here in training around model size, because when we're in the inference stage, the model size is the, the minimum, um, hardware based memory footprint for us to, to hold the entire model. So if you've got a 7 billion parameter model, it's gonna be roughly seven gigabytes of ram of hardware based memory.

But when you're doing training, you need rather more than that. And so if your use cases, primarily we're using this hardware to do inference, but once a month we're doing fine tuning training in order to get a, a faster response of out of our, you know, normal rag system. This is a, a use case where that process for a couple of days to improve the quality of the base model might suit really nicely, but where you don't wanna spend four times as much on the GPUs for that training folks.

Yeah. The, the memory footprint required. Um, so if you do full fine tuned training, it's generally 20 times or 20 x, uh, the model size.

So for example, a 70 billion parameter model times 10 would be 700 and times two. 4 terabytes of memory. That's a lot of GPUs.

That's about 32 of the, um, well, not, so when the memory footprint was around 48 gigabytes, uh, for the RTX 4,006,000 range, I, I don't know if they've gone up recently, but if they were, if they were around, uh, 48 gigabytes, you would need 32 of those to actually do full fine tune. Now, there's another technique called Laura, which is low ranking adaptation, and typically that's two to four x the model's memory size that, that's the footprint required to train it. And the thing about Laura is it either works or it doesn't and, and it's, it's pretty, um, pretty obvious if it didn't work.

Uh, and so you just go, okay, Laura didn't work, so let's def let's fall back to regular fine tuned training, and then that memory footprint goes up. Um, but for, you know, and some, some people might say, well, it's not really 20. It is and it isn't.

So in some cases it has to be 20 because Laura didn't work and when it, but when it does work, it's two to four x, Right? And even at the two to four, like, so the new, you know, RTX Pro 6,000 Blackwell is 96 gig of memory now. So a 70, 72 billion parameter model at four x, I still need two three cards for that.

So the, the trade off and, and with, uh, you know, one or two of the AI 100, um, SSDs, you know, I can do either Laura or full fine tuning on a single GPU or a pair of GPUs, Right? And, and our solutions actually, so we have them in M two form factor, which will fit into most workstations. And, and a lot of workstations have like two or three spare M two slots, so it's kind of nice.

It, it fits in nicely, but our solution will also work on servers. And so there are people that have, for example, a server with, um, 4, 6 8 GPUs in the, you know, RTX 6,000 range or the, uh, HB 200 range. And so even there, it has benefits because some of the other features that, that this offload capability ads is that we can, we can extend the KD cache to be much larger than what the memory footprint would allow.

And that's important when you're supporting a lot of users and you're doing a lot of batch training, it allows you to service more users per, um, per box than you could otherwise. Now, when I mentioned that people are, you know, one question that comes up, and I, I think I talked about this at a presentation recently. One of the things that comes up is, um, well, won't that be slower?

Because like even two gen four SSDs, that's 14 gigabytes per second, and on a workstation you've got a hundred gigabytes per second of dram. We actually did a test with two of our SSDs and an HB 200, and we had a workload of, uh, a batch size of two 20 users with an input of 2000 tokens. And, uh, responsive about 200 tokens, which is a typical sort of one shot kind of query, um, in an environment where you're supporting multiple users.

Um, and what we found is that the number of tokens per second that you can service in if, if all of the activities in HBM and, and the, uh, h HP 200 does have hbm, um, versus if you're offloading to the SSD, there's less than 10% difference. And what that really, 'cause again, this doesn't sound right, but what it comes down to is that there are other bottlenecks in the system. And so whether you're offloading to HBM or you're offloading to the ssd, you still get the same performance.

But what that means is that if that HB M memory, and I, I don't remember the memory of an HB 200, but let's say it's hundred gigs, uh, you can easily extend that to 1, 2, 3, 4 terabytes and have substantially longer context with no performance degradation. And without that, your only solution would be to have more and more and more servers. So it's an interesting, um, it's an interesting feature that that actually adds a lot of value when you're not training.

Yeah. And depending on the number of users you have, you know, a certain token rate coming back is more than enough. You know, I can't read faster than 20 tokens per second.

I love it when they come in at 40 tokens per second, I can just scan it. But, you know, 200, 300 tokens per second is, is overkill for me. So being able to balance that out, and I'm glad you mentioned the higher context size, because these models, the open source models are now often at 1 28 K and soon expect them to go larger.

And that context is key, uh, for good quality results. But even if you're not using the full 1 28 K, you know, you're going to 2, 3, 4, 5, 10,000 tokens, um, and you get a lot of that extra input either from a, a long session or a lot of input, reference documents that you're, you're putting into your, your query, even in that scenario. Um, it, you know, even if it's not the max, it turns out that you will quickly run out of memory if, if your only pool is the HBM, the, the other thing that we allowed to do.

So we talked about, uh, oh, sorry. So I mentioned context window, um, and they kind of work together. The, there's the idea of context window expansion, which is to say that maybe with the memory pool that you had, it, it's only this big, you can make it longer.

But parallel to that is KV cash eviction. So once your KD cash is full, an entry gets evicted and then disappears, and then you have to recalculate it. And that's why you see in certain scenarios where the first query is pretty quick, second query is a little slower, third query is a little slower, fourth query is even slower.

And what we've actually found is that when you catch those evictions and you put them on the SSD, and then the way that our middleware works is that it first checks the, the HBM cache, is that there? No. Then it goes to check to the ssd, um, that whole flow, which intuitively would say, oh yeah, that's gonna be slow and suck.

Nope, that, that's what I was referring to. There's less than a 10% degradation of fetching those missing tokens from the ssd. Um, you know, versus had they always been resonant in the HB m memory.

So, so what that means is that when you ask question one, question two, question three, question four, their speed is flat. There's no slowdown in a scenario where you have a lot of users taxing the resource of the, uh, the gpu. So both of those go together.

And there's another one that we can do in low, uh, resource environments, like typically what you see in Edge ai, um, and we can actually stream instead of, normally a model is broken up kind of in vertical slices across all available GPUs. Another way to do it is to stream the model in one layer at a time and has about 30 layers. Um, and so yes, inference is slower, but again, in the case where you don't need maximum performance, and this is, it's, it's for that particular use case, it's okay that it's slower.

You end up with a scenario where you can actually run a much larger model without quantization than what would normally be possible without this kind of offload technology, which unifies, uh, the available memory, which is either, you know, dram, VRA or HBM or unifies that memory with the pool that's available from the SSDs that are dedicated to this task. It's not like you're using the S SSDs to also do storage and os activity. It's, it's dedicated to the AI task, but it allows you to do a lot more with your hardware.

You're clearly not super excited about this, are you, Sebastian? Well, I think it's cool because it's great if you have infinite funds and some companies do, a lot don't, and they're getting left on the sidelines or they're being sold, oh, go to the cloud, that's fine, as long as you are comfortable with your cloud provider. And, and the reality is, regardless of who your provider is, um, and we've seen recent stories where some of the biggest providers have had cases where information leaks, once the info is leaked, it's leaked, and you have better control when it's in your own environment.

But also you're not like those big environments have a giant bullseye on them, right? Because they're, they're juicy. If you're an, you know, if, if you're not a Fortune 500 company, there's odd, the odds are good that nobody's even attacking your infrastructure because they don't know whether it's there.

So there, there is some value in the anonymity of being a minnow in a flock as opposed to being the, the big guy right at the front that everyone's looking at. Yeah. So the interesting thing about what you, what you just said around, you know, the ability to, uh, do stuff locally versus in the cloud and avoiding, you know, being too much exposed is that even if you're in 14 500, or let's say even extend that to 14, 2000 organizations, you always have the ability to, you know, go on the cloud and do stuff.

But then again, uh, in some industries which are regulated, you have the risk of, you know, leaking sensitive data, right? So it could be, uh, if we're talking about the, the pharmaceutical industry, for example, it could be research data, all kind of, you know, sensitive, personal identifiable information and so on. And most of the time people come back, you know, to infrastructure teams and tell us, you know, we need, I don't know, 4,000, 400,000, $500,000, we need to buy some servers, this and that.

I think that is a really elegant solution, you know, to allow people to work on their work locally with their data on their workstations without having to go with the complexity of the cloud and the potential, you know, conflicts with, you know, uh, data, you know, data management, data governance, and with the, uh, let's say the high upfront cost. You know, and, and, and what you also said as well, the ability to, to run stuff, different models, it gives a lot, a lot of flexibility our to organizations. So you can have a way to test things locally before you decide, yes, this is the right model, this is the right thing I want to use, and I can, you know, kind of industrialize what I'm doing here and bring it to the next level, perhaps.

Yeah. There, there's also, um, so some, sometimes people say, Hey, I can do rag RAG, I'm good. I don't need to do anything else.

And what we found is that sure RAG is great because it gives you a concrete reference to where that information comes from. But if the model isn't trained on the data that it's referencing, it's essentially using its generic interpretation, which comes from training at large on the internet where wherever the model was originally trained, which is usually based on a bunch of stuff from Reddit, Wikipedia, and stuff like that. And so if you train, so if you combine both, if you have the rag reference so that you can get pointer to the specific document, and you combine that with a model that was trained to understand that document a little bit better, uh, through fine tuned training, you essentially get another 10 to 15% improvement on the quality of the, the responses that it's giving you.

So, you know, on top of all of the other benefits, th this is another thing, and, and our solution actually works on everything from Edge, you know, systems like, uh, something based on a Jetson all the way up to PCs, like an educational pc, which is another market that is heavily underserved, is is just universities have the, the right tools for training their, their students. So this opens up possibilities, and we have partnerships with a few universities, um, and then, um, it goes all the way up to workstations and, and even servers. So our solution scales quite a bit and, and opens up possibilities that weren't there otherwise.

Just remember to put a really good heat sink on those SSDs. Um, yeah, in general, with these Gen four, gen five and soon Gen six SSDs and, and we make them in all classes, um, it's generally recommended to have a heat sink, uh, because essentially the, the one workload that stresses the drive the most is, is the right workflow, which happens a lot in offload. Um, and so having a good thermal solution within your, your device is very important.

That's true. And one of the things that struck me about this also was that, that you, you just mentioned middleware layer, but it really is a, it's a plugin to standard framework. It's not that I have to re-architect and rebuild my entire application, this, this just plug straight in.

Uh, and I think that was, that's a vital part of any that's gonna change our workflow, is you can still use the, the same processes that you'd use on your workstation with this if you maybe move into a, uh, production environment that doesn't have, or doesn't yet have this Solution in place. Yeah. And in many, we, we and our operate with most open source tools, um, right now we're on PyTorch in the PyTorch runtime.

I always forget what it's called. It, it, it's not Run Torch, but it, it's something like that executor. So we work with Executor, we work with PyTorch, uh, and, and just naturally collaborate with open source tools because essentially we're all leveraging the PyTorch framework and we, we, we operate in our space, but we don't try to block anything else.

So there's lot opportunities. Thank joining us today podcast. Before we go, where can people continue this conversation, join, uh, Sian synergy and enthusiasm and learn a little more about this and about my gifts?

Well, we, uh, we have a, um, uh, a website and a blog that talks mo about a products, and of course there's a, a contact page there. And, and if you want to reach out to me directly, uh, easiest way to do that is through LinkedIn. com and on LinkedIn as well.

Yeah. com and again, LinkedIn Rules Supreme. Yeah.

And the fragmentation of the other social media. LinkedIn seems to be taking over. You can find me Alistair Cook on LinkedIn as well.

You can also find me periodically on the, uh, futurum group site, as well as on my own Demi test co nz site. So thank you very much for joining us and listening to this episode of The Tech Fields, a podcast. And if you enjoyed this discussion, please subscribe on YouTube or your favorite podcast application so you don't miss a single episode.

Do consider giving us a rating and a very nice review how much you enjoy the enthusiasm we bring. This podcast was brought to you by Fon and Tech Field Day, the home of IT experts from across the enterprise and part of the RUM Group For upcoming events, more episodes, head to tofield com podcast on text on tv. Thanks for listening, and we'll see you next week.