Half the Web Is Bots — Inside Fastly’s Threat Report and the New AI Crawler Problem

Hey everyone. Welcome back here to Techstrong TV. I'm really happy to have our next guest on here.

He's never been on Techstrong TV before, so let's welcome him. Marshall Erwin, who is the CISO over at Fastly. Marshall, welcome to Techstrong TV.

It's great to have you on here. Hi, Alan. Thanks for having me on.

I'm looking forward to the conversation. My pleasure. Marshall, as I mentioned, you're CISO over at Fastly, but people always want to know who's this guy talking to them, right?

Yeah. So give us a sense of how you came to be CISO at Fastly and maybe a little bit of your career arc. Yeah, sure.

So as the CISO at Fastly, there's two core areas of responsibility that I have right now. One is what we would think of as security operations, the day-to-day work of finding and fixing vulnerabilities, making sure that we detect and respond to attacks on Fastly, and also supporting our customers when they come under attack as well. And then the second part of the role is really focused on the proactive work, our security risk architecture work, to make sure that we are building and maintaining our actual services effectively and securely, so we can support our customers as well.

So I've been at Fastly about three years or so. Before that, I was the CISO at Mozilla for a long time, the browser maker, ensuring that we- Really? built the Firefox browser securely, which is an important piece of the web's infrastructure as well.

And I actually started my cybersecurity career back before cybersecurity was really a thing, almost probably more than 20 years ago now. Started in the US intel community, doing cybersecurity back before it was cool, and back before I think many people realized that it was going to be a big risk for all of us that we were going to have to deal with on a day-to-day basis. Yes.

I remember those days well. My background is in security. I've been in security about 30 years.

Back then, of course, we called it InfoSec. Mm-hmm. Or just security, right?

No one had the cyber thing, but- Yeah ... here we are. Marshall, Fastly's a brand that a lot of people have heard of, a lot of people are familiar with, a lot of people maybe not so familiar.

If you wouldn't mind, how would you describe Fastly to people? Mm-hmm. Yeah.

So many people have heard of Fastly, but I think what people don't realize is you're interacting with Fastly already on a day-to-day basis. We, as a part of our network services offering, we support a large part of the web's traffic today. So if you are engaging with your local media outlet, or The Guardian, for example, or if you're committing code on GitHub, you're often already interacting with Fastly.

Maybe you're using your preferred streaming provider. You're also interacting with Fastly. We are supporting some of the web's largest enterprises, making sure that their content gets to their consumers quickly and securely.

And then the sort of network that we built to cache content and get it to consumers also allows us to layer on top of that some interesting security properties because of its distributed nature. And so in addition to our core network services offering, we have a number of security products, DDoS, bot management, which we'll talk a lot more about today, which allows us to protect our customers' origins when they come under attack or when people try to siphon off their content. So I didn't realize, it's really so kind of in the Akamai, Cloudflare space as well.

Mm-hmm. That's right. Yeah.

We're one of the leading CDNs, I think, one of the bigger heavy hitters in the CDN space along with Cloudflare and Akamai. That's right. Very cool.

And of course, that puts you in really a cat's bird seat to see what's going on, right? As web traffic, the very nature and makeup of web traffic changes. I remember a couple of years ago speaking, I forgot who I was talking to already, but it was the first time I heard that a majority of the traffic on the web then was API to API communications.

Mm-hmm. Right? Which was, I think a lot of people were surprised at.

Of course, now with the advent of AI and agents and more bots than ever, we're seeing web traffic again kind of morph- Mm-hmm ... at the very, it's very fabric. Talk to us about that a little bit, if you can.

Yeah. So I'd say just at a high level, one of the fascinating things about Fastly and the view that we have across both our network services offering, which supports, like I said, a large portion of the web's traffic, as well as our DDoS, our bot protection offerings, our WAF product, that gives us a unique level of visibility that we have across web trends, generally, and also attack trends and bot trends. And we use those data sources to publish frequent blog posts, but also a quarterly threat report highlighting some of the risks that we see that our customers are facing.

One of the really big trends that we've dug into over the last year is on bot trends, generally, and also on AI bot trends specifically. And we recently published a threat report that really dug into the recent trends that we're seeing on bots as a general matter. So a few of the critical things that jump out to that, 49% of the requests that our customers see at this point are bot-driven traffic.

And the vast majority of that, 99% of that traffic, of the 49%, is what we would describe as unwanted bots, bots that really are serving some sort of malicious purpose that our customers do not want to service that traffic. It might present a security risk, it might be trying to vacuum up their content, but it's traffic that ultimately is only really downside for our customers that we want to block. Those are, I think, some of the most critical trends that jump out from that report standpoint.

So this is something we live with here at TextStrong, right? We operate a whole bunch of websites. Bot traffic is ridiculous.

Mm-hmm. The problem is the few pearls in with the mess, right? So for instance, if we shut off all bots, we're shutting off Google or other search engines from indexing.

We're shutting off now people who are saying 35% of searchers are going through AI, right? AEO, whatever you want to call it, GEO kind of things. And if you shut those bots off, you're shutting off your ability to rank in these AEO, GEO kinds of things, and so you're cutting off your traffic.

And so our conundrum here is that how do we get that, as you said, 99% of the bots probably are negative, but how do you recognize the ones that aren't? Yeah. So I'll just say the trend that you're describing, the challenge, is not unique.

It's a challenge that many of our customers are facing today. It's interesting to just take a step back. Security professionals are very familiar with this bot problem going back 10 years, because we would see these sort of malicious bots, bots just conducting what we describe as account takeover attacks.

" There's no upside to those. It's like a yes or no proposition. And so it was a yes or no proposition and a security challenge, and what we've seen now is a shift from this sort of bespoke, narrow security issue that I think folks like myself are very familiar with, to a broader business and strategic challenge that bots are creating for exactly the reason that you're describing.

Because not only do we have these traditional sort of security account takeover issues, but you have search engine bots, AI crawlers, AI fetchers, this sort of undifferentiated mass of AI traffic that is causing a whole host of problems, both security problems, but more importantly, strategic business problems. So the first thing to start with is visibility is really key, and I think many sites on the web today just do not have visibility into this set of problems. " Then you're going to have a problem.

Then you're going to be stuck in that yes/no proposition, where you're going to end up blocking legitimate traffic that you want and also missing some of the bad traffic that you don't want. And so getting a tool, like a web application firewall in place that has a good level of bot visibility, like Fastly offers, is really critical. And then from there, what you need to be able to do is look at bot intent.

Not just what is this bot and is it a search engine optimizer, but is this bot misrepresenting who it is? What traffic is it actually trying to look at, and what is it doing? And from there, you can start to use the granular controls that your tool offers to actually block what you want to be able to block.

Got it. Let's get back to this report you guys recently had. Any other kind of big items that stand out that we can tell people about?

Yeah. So another big item that jumps out from the report is that roughly 47% of that unwanted bot traffic is actually hitting what we would describe as cached content. That is content- Mm-hmm ...

that isn't hitting a site's origins, but rather is hitting their cache, hitting Fastly, for example, or your CDN provider. And that to us is super interesting because that is an even more hidden problem that websites are not going to understand and see. Typically, sites are focused on bots that are hitting their origin, and that's because that is creating immediate costs for them, and it's also- Mm-hmm ...

the bot traffic that's going to create those security problems that I mentioned earlier. But the issue with the cached content is a little bit different, because those are the bots that are actually vacuuming up content and monetizing it in other ways off of the platform. And so it's creating more of a business risk than it is a security risk or an operational cost, and that's why thinking about cached content is an important area as well for sites like yourself to focus on, because you want to make sure that that business downside isn't materializing quietly behind the scenes.

Part of me wants to-- Look, I consider myself tech-savvy. Mm-hmm. I've been in the tech space 30 plus years, as I mentioned.

I see this problem every day. But what about the rest of them? What about the rest of the people out here, Marshall?

What's an organization to do here, right? It seems like the rules of the game are changing, right? It's not so black and white blocking out the bad bots from the good bots, because even good bots are still taking bandwidth and resources and clogging things up.

What's the right answer? And I imagine this is to the heart of where Fastly wants to go. Mm-hmm.

Yeah, so it's interesting. You said that you see this problem every day, but I think there's still a large number of website operators that don't understand that this is a problem that they face. They see a growing volume of traffic, and growing volumes of traffic typically are good .

And they don't understand- Yeah ... And then there's folks like yourself that see that problem, but maybe don't have quite the right level of tools to really, one, gain the visibility, and then use those granular tools to dial up some traffic and dial down some other traffic. And so the right solution, to our mind, is a bot management solution, such as the one that Fastly offers.

There's others on the market as well, that first make sure that you give you that level of visibility that you need to distinguish between the AI bots, the search engine optimizing bots, the account takeover bots. And from there you can start to ask: Is this a bot that I want? Is it generating business value for me?

And if not, let's block it. And if so, let's make sure that I know what it's doing and I have the level of visibility that I need into that. Love it.

I want to make sure, is there anything else on the report that we overlooked or that we want to put in front of people? Yeah. So I'd say our reports, but the recent report, as well as our past more AI-focused reports, also dig into the trends regarding AI bots specifically, and you can see both growing volumes of both what we would describe as crawler traffic.

AI crawler traffic is the traffic that's sort of collecting data from sites such as yours to then train their models and get their models better. But more importantly, we also see a growing volume of fetcher traffic over time. Fetcher traffic is that traffic coming from AI companies that are augmenting their productized AI in real time, augmenting their results with your content as well.

And so we've seen really significant trends over the last few years in the growth of both the crawler traffic for training purposes and also the fetcher traffic for real-time consumer use of AI products. And that's a challenge that many of our customers are facing, again, because what those crawlers and fetchers are doing is grabbing that content and monetizing it off of the platforms that generate the content, creating real meaningful business risks for those parties. Which again, is a problem that I think many content creators today need to be more proactive about.

I agree. Marshall, where can people get information on this report, or if they want to maybe get a copy of the report itself? Yeah.

The report is available for free. You can navigate to Fastly's website and get it, as well as our other reports from previous years. You can easily find it and dig into it and ask us, reach out if there's any questions you have.

I love it. One last question. How can AI and agents help us with this?

Mm-hmm. Because they're part of the problem, a growing part of the problem. Could they be part of the solution?

Yeah. So, well, it's worth pointing out, though, some of that agentic traffic, in particular, I think a lot of it is going to be traffic that websites actually want. And that's why- They want ...

visibility and controls are going to be key, because while you might not want an AI company to vacuum up your traffic and monetize it off of your platform, you do want to allow agentic solutions to engage with your platform in a way that can generate consumer value. And so I just want to call that out. That, again, is getting to this challenge and why it's so critical to have meaningful controls that differentiate between different types of AI traffic.

To answer your question directly, I think I would say AI can be critical for companies like Fastly as we continue to improve our bot management solutions that allow us to detect and differentiate between the good AI actors and the bad AI actors. And so it's an important tool that we are actively integrating into the company to hone our actual bot management solutions for our customers at this point. I love it.

Hey, Marshall, I appreciate you coming on, talking to us today. As I said, it's a 15-minute interview. They go really quick.

But keep up the great work. We're living in interesting times with Mythos and all of this changing the whole vulnerability structure. I'm sure it's going to work its way down here.

We're already starting to see some of the impact of that. Come back and keep us posted. Of course.

Thank you. All right. " We're going to take a break.

We'll be back in a bit.