Fastly VP Kelly Shortridge on AI Crawler Infrastructure and Security Challenges

Hey guys. Thanks Withdraw. We're here with Kelly Shortridge, who's vice president of Security Products for Fastly, and we're gonna have a little chat about how all these AI bots might be taking over the internet for better or worse.

Kelly, welcome to show. Thank you so much for having me, Mike. We've been dealing with these various bots, some of which are nefarious and some of the, which are for good for a long time now, but now the AI folks have come along and kind of taken that to a whole nother level.

And is that just overwhelming our websites? Are we generating a lot of traffic for no apparent purpose other than training AI models? And what's to be done about this, if anything?

All great questions. I don't think I can answer them succinctly all at once. I'll start with just the volume of bot traffic.

It is quite enormous, uh, for many organizations. It's not traffic they want, they wanna be serving ideally real users, or at least if it is a bot, it's a bot that supports their business. Um, I think search engine optimization bots are the classic example of generally wanted bots that most industries and organizations want to be able to access their content online, but there are a lot of bots that people don't want, um, to be accessing and they do want those real users to count and from their perspective, if anything, the bot problem is less so even maybe a security problem in their minds, but it's a fundamental business kinda like cost problem as well.

So I think it's, there are interesting dynamics that play both, you know, in terms of what's fair in terms of training LLMs, I think that's the question you were driving towards and you know, just more of the how do you optimize your infrastructure and make sure you're keeping costs down, you're serving, you know, traffic that adds to your business. It's, there are a lot of different considerations in this, so it's actually a pretty interesting problem. And the costs almost seem a little bit invisible to folks.

I mean, I have a website and suddenly there's a lot more, um, resource consumption, whether it's on the processing side of the server or the network or whatever. Where do all these costs manifest themselves and how do I figure out how much I'm spending to service bots versus real people? Well of course I'm going to say that using uh, a product like our AI bot management, which is part of our overall bot management solution, helps kinda streamline and filter uh, the unwanted bots from the traffic that you do want.

So that's certainly one approach you can do, um, arguably recruiter or you know, it's an additional defensive layer, more traditional rate limiting as well to make sure that, you know, you're only serving the traffic that ideally you want or at least not getting overwhelmed and having to serve like huge bursts of traffic, which we do sometimes see in particular with AI bots, we do sometimes see like large requests permanent. The way it manifests though, it could be anything from, you know, we are also a content delivery network that's part of our distributed platform. You could see some bills there because it's all about, you know, we deliver bits across the internet to real users.

You could certainly see it to your point around server side, it could be in your cloud bills as well. Um, if you're, you know, a little older school you're doing stuff on prem, certainly that could be you have to actually spend more CapEx and buy hardware to be able to support the sheer volume of requests that you're getting now. So there are all sorts of ways it can manifest, especially if you do, you know, highly personalized content where you have, you know, um, quite complex ways you set up your site where you know, there are a bunch of paths that you could have a bunch of like parameters you could have in that URL string and if a bot is gonna hit every single one of them, it can kind of affect costs across your whole stack.

Uh, which is why a lot of companies, even if they don't really care about the content scraping angle, they still care from a cost perspective too. Mm-hmm. Who's in charge of this?

Who's taking responsibility for this? And I'm asking the question because a lot of times the folks who are implementing the websites and the CDNs are not the security folks and yet the security folks might care about this. So who's stepping up here and saying, Hey, we need to do something about this?

I think there are quite a few folks in the industry or the community that are stepping up. I would certainly say we are one of them. Uh, part of the reason why, you know, I'm also a cybersecurity nerd as my background, part of the reason why I work at Fastly is because of this shift that's really happened over the past few years where platform engineering teams in particular who have to care about, you know, the infrastructure part, CDN parts, um, you know, the app optimization, they increasingly have to care about security because it affects their metrics too in terms of uptime, certainly cost efficiency, performance.

Um, it's all increasingly intermingled and it's the same, you know, with security engineers. They increasingly have to care about things like performance optimization. So really starting to see like much more collaboration, kind of like melding between these communities.

And they're all looking yes to providers like Fastly and there are a few others like us I'd say. They're also looking to, um, cloud providers more on the end of, hey, sometimes maybe you're ho hosting some of these bots, like you are the application infrastructure for some of these, you know, LLMs can you be helping too? And I think the great news is that the community overall, there's a lot of interest in solving this problem in a way that's fair to everyone.

Of course the downside is no matter where you are in the world, what problem area, if you're trying to design something that is fair for everyone, then everybody's gonna have strong opinions. Um, and that's where a lot of the trickiness comes in is because you know, you have to balance business interests on different sides. And again, you have so many trade offs that are at play as well.

And what we see with our customer base a lot of times is they have an agreement with, you know, some sort of AI company and they do want again to let that traffic through, but then they don't want the other AI companies, um, to access their content. Of course, what we see, and we have a threat research report coming out soon that's gonna talk about some of this data is sometimes the AI companies impersonate each other though and that adds like yet another layer of complexity to the problem. Mm-hmm.

The thing we've seen over the years, especially from a security perspective, is the CDM is kind of like, you know, A DMZ where it provides a layer of, um, distance between my organization and the wild wild west of the internet. Do you think as we kind of go along here with AI bots, that more people are gonna lean on a CDN to provide that kind of function because they will find that their sites are being overwhelmed by all this bot traffic, which as far as I can tell with the rise of AI agents is just gonna exponentially increase. Is that fair?

We've certainly seen it explode really this year in particular, uh, we released the AI bot management feature, let's say at the beginning of April. Precisely for that reason. We started to see that trend happening and it's really only grown, um, since we released the product.

I think you're absolutely correct. We do see a lot more companies realizing, you know, this isn't something we can really DIY and frankly except for very few companies across the world, they just don't have the visibility globally to be able to gather, you know, I don't particularly love using Bud buzzwords, but here it's maybe Applic a woman who's not threat intelligence, but certainly let's say bot intelligence where we're able to see the different ways bots can manifest their different techniques. Certainly, you know, we do sometimes see that the AI companies that are a little more unscrupulous adopt some of the techniques that your more traditional criminal bot operators use to be able to bypass detection.

And so coming to a vendor like Fastly or two point like other CDNs as well, we welcome more people trying to solve this problem. It is a way to kind of benefit from those economies of scale, like benefit from the fact that we host some of the major, you know, the leading publishers around the world and of course they're getting hammered right by bot traffic, especially AI content scrapers. And so having that global view and having that at scale and being able to like in real time be able to detect block, take whatever actions even deceive, that's one of the things Fastly does that other people can't is we, we love being able to deceive one appropriate.

That is something that is just really hard to create that logic and implement it in a way at scale that's still performing, that's not gonna interrupt legitimate users either, because you still want your business to be running. So it's, it is a really hard problem. We're talking about AI bots in the context of people who mean, well, they're essentially AI firms that are trying to train something using data that they find and that that may or may not be legitimate.

But as a security person, are you starting to see some of the more nefarious characters out there starting to think about how to use AI bots to train LLMs for purposes that, uh, are definitely malicious? Well, I think the one person's idea of malicious may not be another, and I, I definitely am not a geopolitical expert, so I'm not gonna wade into that territory, so to speak. I will say that again, we are seeing some of the less scrupulous AI companies adopt techniques that are more from that criminal bot world.

You know, where they used to try to bypass defenses so they could do things like credential stuffing and more traditional account takeover, you know, non-AI scraping as well was always a use case. So we do increasingly see that overlap at Fastly. We actually have an important distinction, which is based on, again, our heritage with publishers, entertainment companies, e-commerce companies that are getting the brunt of the, let's say AI onslaught, which is that they're fetcher and crawlers.

So what you're talking about is much more of a crawler where they're just hoovering up as much content as they can to feed into their LLM model, right? And certainly there are some, you know, companies that are willing to enter into AI companies willing to enter business agreements with content creators, which is more to your point above board. There are others who are trying to bypass that and would prefer not ever paying for any of that content into perpetuity, even if they, it means they have to either, uh, use gray or nefarious means.

To your point, kera, we also see fetcher, uh, which are the ones where you enter at some sort of query. You know, the example I give my favorite coffee shop in New York City, which is where I'm normally based, it's called Yanis. Um, hopefully they don't get overcrowded 'cause everybody floods to them, but they are the best in the city.

If I wanna ask, you know, what are the hours then there could be an AI agent that essentially fetches that content for me. They'll often attribute it to, which is generally we consider that good, be like, well, Yanis website says these are the hours. I'm like, okay, great.

And then I go, um, you know, contribute to that business. And that's generally for a lot of companies, they're like, that is okay, as long as there's attribution and it's contributing to our business. That's good.

So in some sense, crawlers are more of their concern in general. And certainly, um, when you think about something like e-commerce, there's also the gray area of maybe, you know, you're trying to expand into an international region and there's an upstart in that region who's like, well, I'm just gonna automatically like use some sort of AI bot that's gonna automatically scrape that, or even in some cases fetch that I'll just undercut automatically. Right?

And that's really harmful to a business. So you're seeing those competitive pressures and competitive dynamics as well. Um, it's all to say it's a bit messy and determining, let's say in a generalized way what is a good bot or a bad bot is nearly impossible.

Um, and so what we've done, and I would certainly encourage any other vendor who's, you know, entering this space to do is give more power to the organizations who understand their business context. Make sure, you know, we talk a lot with our customers, give them guidance like, Hey, you should probably talk to your business development team to make sure you know, you understand the agreements that are in place or not. And allow you to very easily like weave in that context into whatever, like logic, security, logic and rules that you create to make sure you capture that notion of what is good to you as an organization and what is bad, because it can be wildly different between companies.

So ultimately, what's your best advice to folks who are struggling with all these AI bots out there and trying to figure out what to do next? Because I think at the very least, the the whole thing's a little overwhelming. It Is overwhelming.

Um, especially I would argue for a lot of of folks, it's either they already don't know bot techniques, which is fair. Um, they evolve quite a bit more quickly than things like, I don't know, like kernel exploitation, that those techniques don't evolve more on the time horizon of years versus, you know, a couple months. Um, you know, you have to understand in some sense like how browsers work, certainly how clients interact with like any content you're hosting on a server, then you also have to understand, you know, what's the kind of content potentially that not only loons want, but also what matters to your business.

You have to have some level of business sense in acumen of what is actually gonna harm your business versus okay, maybe that's a nuisance and you know, it's more of a cost problem versus an existential threat, you know, to our ongoing health as an organization, of course I'm biased, I'm gonna say you should probably reach out to a vendor and get some help. Um, especially with bot classification, at least being able to get that, those signals around like, yes, this is this type of bot by this company. Like, and, you know, they're performing this kind of use case at a minimum, even if you aren't trying to classify those AI bots, at least being able to weed out, you know, ones that are using advanced techniques like headless crow where they're trying to bypass that detection and obfuscate who they are and, you know, other techniques that are at play that criminals normally use, it's only upside really, if you at least do that because you're wiping out the criminals, you're wiping out the less scrupulous AI bots and then you know, if it's really important to your business.

And I would argue for some organizations, it's just not, they have everything by a lot behind a login portal, so it's gonna be really hard for a bot unless they perform pretty blatantly illegal things like account takeover. It's gonna be hard for them to scrape any of that content. So maybe just stop bots, you know, from being able to try things like account takeover and you're good.

But for, again, e-commerce, certainly, um, I don't say any form of online retail, uh, entertainment and media publishers. You're probably gonna want to be able to have that granularity around what it kind of AI bot, is it what seems to be their purpose? Are they doing anything that looks a little shady?

And then having the control to say like, I wanna block it, I wanna allow it, or I wanna monetize it too, which is an emerging strategy. Hey folks, you heard it here. You know what, AI bots are not going away, but they're not all created equal either.

So you need to get smart about which ones are friendly and which one maybe you don't wanna engage at all. Kelly, thanks for being on the show. Thank you so much for having me, Mike.

All right, and back to you guys in the studio.