58. The Unknown Unknowns of Cloud Providers with Catchpoint – Tech Field Day Podcast Spotlight Series

It is the Tech Field Day podcast. With Catchpoint, we are going to be looking for the unknown and unknowns. Have you got links to external websites that maybe your Vibe coders have linked to?

Or maybe one of your developers found a really useful tool that links to some other website. Maybe it's a service that's running on a cloud provider like AWS. If you don't know it's there, it might cause you pain later.

Stay tuned for this episode of the Tech Field Day podcast. Welcome to the Tech Field Day podcast, where we bring together a group of IT technical experts to discuss a single idea around key concepts in the industry. This podcast features a variety of perspectives from members of the Tech Field Day delegate community, and is often recorded in association with one of our events.

Tech Field Day is part of the Futurum Group, and this podcast is also published on our sister company site Techstrong tv. On this episode presented by Catchpoint, we'll be discussing how your internet application is full of unknown unknowns. Before we get started, let's meet who's on the panel today.

Hi everybody. My name's John Meyer, chief content creator at Meyer Media. And, uh, my name is Eric Wright.

I'm the co-founder of GTM Delta and also the head podcaster at the Disco Posse podcast. And joining us from Catchpoint, fresh off their presentation at Cloud Field Day 22 is Eddie Dody. Welcome.

Thank you gentlemen. So, hi everyone. My name is me.

I'm the co-founder and CEO of Catch one, and coming to you live from New York. Excellent. Well, we're spreading all around the place.

Uh, we came up with this unknown unknowns because we were discussing a little beforehand, as, as you often do, uh, how incredibly interdependent we end up with with internet based applications because everything, every kind of service you can imagine is available over the internet. We end up building our applications to use bits of this and bits of that, and we may not even realize in the same way that on premises, we used to occasionally end up with a developer machine, which was running some vital service in our production application. Now, our internet applications can be dependent on all kinds of things all over the internet and medi.

This leads to people experiencing outages when they really don't think they should be experiencing outages. I like what you just said. They're experiencing outages where they should not be experiencing outages.

Um, so when you think about it today in 2025, an average web, an average website or web application has about two to 300 calls to, to objects that are, that make their webpage, right? So that can be ads, that can be CDNs, it can be tracking pixels, capture services, payment gateways, you name it. But on average it's two to 300 requests that needs to happen in milliseconds for you to have a good user experience.

That's true for mobile apps as well. Out of those two to 300 requests that needs to happen and fire up immediately, you're probably talking to about 50 to 60 different host names that are hosted by different organizations, whether it's cloud services like Amazon, Google, et cetera, or your own data centers. And those calls also rely on a DNS lookup that is hosted by either ultra DNS and and S one Dine Oracle, or your own DNS services and all of that rely on a spaghetti of networks, uh, of ISPs.

And all of this stuff needs to happen in real time. I mean, when you think about it, it's like a human body, right? My head is thinking it's talking, telling my toe to move, and I'm not even thinking about it, right?

All of this needs to happen in milliseconds. The problem is, uh, you, the company for example, have great service, uh, great services. You bought the best servers, you have the best third party, uh, hosting.

You have the best CDN, you have the best of everything. You have the best SREs, you have the best engineers, you're top-notch. But there is a third party pixel that is called by some unknown other pixel that is called by another unknown pixel that comes out of nowhere that is hosted in Cambodia or Vietnam.

Uh, and that thing is unre unreachable, and that is going to bring your house down, literally a house of cart. So, uh, so this is why I like the unknown unknown, uh, famously introduced by Rumsfeld during the Gulf War, for those that remember, but in many times, we're dealing with an unknown unknown. So what is the unknown unknown in the observability monitoring space is you don't know you have a problem and you, and worse, you don't know how to deal with it.

So again, you don't know you have a problem. So then you get to rely on down detector or our own version or, or something like that. Or maybe even worse, your CEO coming down, it's like, Hey, what's up with, with this?

Uh, I'm in a meeting and I couldn't close this deal because our website was done or our web application was done. So first you don't know you have a problem. So you're relying on Twitter down to, sorry, X down detector, uh, an angry C level person that is telling you what you should have known in the first place, which is you're, you're dumb.

Then you look at your playbooks and it's like, you've never seen this error, so you don't know how to act with it. And that is doing what? That is going to prolong the time it takes you to respond and get back in online.

So that's the unknown unknown we are dealing with. And, and believe it or not, it happens a lot more and we're not seeing that problem go away just because, hey, let's add the chat GPT uh, piece on our website, right? I mean, what can go wrong?

It's ai, it's self, self working, right? Or self debugging itself. Well, if you open AI is a problem, you, your website is going to have a problem, and then you're going to have angry customers.

So, uh, so we're constantly adding stuff to our websites and our web applications because we want to reach your things. We want to reach our applications, we want to reach our visualizations, we want to reach your experiences. Well, we, when we want that, we want complex and we, we add complexity, and the complexity just keeps feeding this unknown unknown little monster of ours.

One of the things that I hear a lot of times medi is, uh, you know, a PM is obviously like better telemetry, better instrumentation. We've got lots of people trying to solve the instrumentation challenge. So I'd love to let people know and like think of like, what, where does a PM end off and IPM come in?

Where does the interconnectedness of those things? So where are you winning by a PM and where does IPM really, you know, come in where a PM can't? Sure.

Um, so it's not one versus the other that I don't think that's the best way to look at it. So for the viewers and listeners, so I'm not, I'm not advocating one or the other. I think it's usually it's a stereo vision, so you need both.

So it's A plus B, and here is, uh, here is why. So again, going back to the example we talked about you, the company A has the best engineers, the best servers, the best, best, best. And you go and buy yourself the best a PM tool out there, Dynatrace, Datadog, you name it, they're all amazing tools out there.

That's great. So your house is protected, right? You, you, you have all the telemetry about that window in the north, that the door on the right, the roof, the water leakage, you have everything great, you're safe.

Except that, except that you invite two, 300 people every day to your house, and they're not going to necessarily put Dynatrace or Datadog or whatever you pick you, they're not going to put your smoke detector in their house. And so that's why it's important to have that, that end user view. Uh, that is, uh, the monitor, the monitors, basically, we, we are above all that stuff.

We're looking at it from a satellite view. So, uh, again, you can put your, your smoke detector in your house. I can look at it from a, uh, from space, and I can zoom in.

I can say, yes, your house is on fire because, uh, that third party that you, uh, uh, that you've let in is having a problem. And you, you don't know because you're not going to put an a PM tool agent on that third party vendor that you use. So, uh, the other thing is where a PM is very good at understanding your infrastructure, your systems, your application, your code, your, your relationship between that piece of code and the database lookup and whatnot, it's not very good.

That telling you, well, you know, in Boston on at and t right now, there is a BGP issue that is preventing all the users from getting to you. And you, if you just rely on a PM, somebody's going to call you and say, Hey, people in Boston having a problem, and you're going to go and blame the, you're going to blame the developer, you're going to blame the database guy. You're going to maybe look and see, was there a change?

Did we make a rollout yesterday that broke something? So you're going to go and spin your, your wheels in trying to maybe try to see if the problem is in your house versus the problem being, again, a third party. And again, the unknown unknown.

So you don't know you have a problem and you don't know how to deal with it, right? And that's again, where we come in. So it's not one versus the other.

It is just one covers the inside of, of your house, and the other one looks at it from a different perspective. And just make sure that even if, by the way, most of those a PM vendors that are great, uh, they're all hosted on the cloud, right? So by the way, if AWS goes down, you're completely blind because all of them are hosted on AWS.

We're not, we're not hosted on any cloud vendor on purpose, and I, it cost me a lot of money to not be hosted on AWS or one of those vendors. But we're on purpose because ultimately our job is to watch over those things. Medi, I love how you use the analogy of a house, and I wanna walk through that analogy just a little bit more here.

I have a house, right? And if I invite somebody into my house, I know where they're at, I know they're coming into my house, and I can validate that, hey, listen, they're into it. But here's the other thing that happens is that what if that person invites another person into a house and another person in a house?

I can't monitor them. I don't know. I didn't know that they actually invited them in a house.

And if, or something goes down, something happens, I have no way of tracking it. Now I translate that to your traditional, your IT infrastructure, uh, building an application where I am the, you know, IT manager, and I know that they built this application and the database guy connected to this. Now that guy left, and, uh, I didn't know that he connected it to, you know, say Facebook meta to pull in some data and APIs behind the scene.

Now I have tribal knowledge. And now they're pulling in that. And when that goes down, we're running around with our heads cut off trying to figure out what, why, why am I pulling meta information?

Why is that, wait, why did I build my application around it that is totally unknown that I didn't know my application was tied to this backend pulling this data that now critically handles it off to my customers. You know, those are some of the things that are totally unknown to an organization or an application until something goes wrong. And, and the other thing, John, what what we see a lot is, well, it's not my problem.

I didn't build it, so why should I like, Hey, that wasn't my problem. Don't yell at me, but I'll go fix it for you. Correct?

And, and that is, you see, in the unknown unknown equation, the biggest factor in my opinion is the, I don't care and it's not my fault. And, or, you know what, Hey, we have, we, we bought an a PM tool or we bought this other tool or whatever. So I'm covered.

And, and, uh, and so I think we have to rethink a little bit, right? When, when, when you buy a, when you buy a, a, a an an alarm system for your home, you want to protect your family, you want to protect your valuables, you want to make sure that if there is smoke, you can catch it before it goes everywhere. You want to catch carbon monoxide, you want to catch water leaks, and it all needs to work in tandem to protect you.

There is no, well, I don't care. Well, I didn't do the plumbing, or, Hey mom, dad, I'm, you know, I, I just did my bedroom. I'm not going to care about flushing the toilet.

It, it is just like, it, it's, it's almost like a sense of responsibility that companies need to, or the, the people that work in companies need to take on and say, we are all ultimately responsible for the assets that actually pay our paychecks, and we have to think about it holistically. And therefore I need to put in place the right monitoring system so I can catch things as quickly as possible. Because the sooner you can catch something, the faster and cheaper it is to fix.

Always. There's, there's that crucial thing of, of it. It doesn't matter how the business you are in fails, you are gonna be affected by it failing.

And so that it's, it's not, you know, I, I've done my bit, that's, that's, uh, it part of dealing with this, the solution isn't enough, and sooner or later you gotta make sure that the business is able to continue to operate. You've gotta have that system level thinking of, I'm contributing to this, but uh, the story is not complete if I just do my part of it. And having that awareness that we may be unaware that there may be dependencies, and particularly there may be dependencies on third parties who have their own dependencies on other third parties that we are totally unaware of.

Um, that Collection Scary Here, here is, here is the biggest first party, the biggest third party is your cloud provider. I mean, you, when you think about it, we have, we have decided to put all of our eggs in Amazon or GCP or Azure. You don't know how those guys run their stuff.

You don't, you we're taking it face value that, I mean, thank God they are right, but they also buy third party stuff. They also, you know, sometimes are not their data centers, their electricity is not from them. The water supplies from the city, the generator, the diesel comes from, I mean, Amazon does not have petroleum yet, right?

So, so it is just like, this is how people need to think about this to do risk, risk mitigation and understand what can go wrong versus this mentality that we've had since the cloud happened, which is like, oh, it's easy. I push a button and poof. And it is easy.

You can just consume a service. But what happens when the service isn't working? I mean, one of the, the elements in this is a lot of this dependency on third party is if you code your application for all of the loose coupling and, uh, being able to cope with delays along the way, then you minimize the impact on the end user application.

It's just that doing that is a lot of hard work for developers. And developers are usually incented to go and get new features out, not to spend all of their time ensuring the, the highest uptime. And don't get me wrong, there are plenty of developers who do an awesome job of that, but often the business driver is, we need to get this new feature out.

We need to get it out as rapidly as possible. And that's the sort of thing that leads to that scenario. Yeah, And you're so spot on, and we do a terrible job at explaining to the business that the number one feature, feature nu Uno is performance.

That is the most important feature in my opinion. I mean, there was this venture capitalist guy, Fred Wilson, you can Google him. He said that it was brilliant.

He said that in 2008, 2009, the, the most important feature is performance. And that hasn't changed today. It, it doesn't, that's why we keep buying new iPhones or new and why, because the processor is more performing because we want speed.

We are so driven by this damn speed and reliability. One of the things medi that I think more and more of us are seeing is the abstraction away from, like, the whole idea is like, can I get ideas to production? Whether it's no code, low code, we're seeing things like Versal and we're seeing vibe coding, and what you, we end up with these is a lot of, you know, less bespoke solutions, but they're also incredibly opaque.

Uh, so how, how are you seeing the rise of paths actually getting great success in my view? Like, you're seeing Heroku coming back to the for, again, where you're seeing a lot of that happening, but then what's the impact on application builders and, you know, where do you see your, your role in, in helping people to really get the most outta those platforms? I, I, so yes, there is a rise of all these applications, all these platforms.

Uh, I think we're going to see even more of that. I mean, when you think that chat GPT can start generating code and prototype for you and stuff like that, I mean, it's, it's, it's unbelievable the, the, the world we're, we're going to live in, right? Where it's just like the ideas are going to just like, it's explosion of ideas and creativity.

I'm, I'm, I'm, I'm jealous to some degree, right? Um, I do think that, uh, the opacity that you mentioned is good for us, uh, because at the end of the day, that doesn't take away the fundamentals of reliability. Again, it's still a computer, right?

It's still a network, it's still a service, it's still an infrastructure. It's still CPUs and memories and things like that. And that stuff still breaks.

And we keep breaking, I think, uh, where we are going to, and you showed that. You saw that, uh, uh, when we presented, that's why we keep investing on this internet stack map, being able to visualize and understand the dependencies. Uh, that's why we keep investing in some of our AI capabilities to be able to detect and connect the dots.

Because what this, what this is going to do is just like more opacity, more complexity, and then understanding what happened is, is going to be more crucial. That unknown unknown is still there. Bigger, worse.

We just need to shrink the amount of time to detect and, and, and help customers. But yeah, I don't see any change that just think that things are going to get worse from a complexity perspective. Medi, are we our own saboteurs when it comes to the unknown?

When you're implementing things and you're trying to be efficient, you're trying to get the most out of it and the performance, and we go in attached to a website or a database that's public out there, just because we wanna provide something to our customer, but ultimately not realizing that we don't know as much about them as we should. Yes. And I will add, I think we need to live around that, so we shouldn't stop that because I think that's stopping progress to some degree.

That's stopping creativity that, listen, I love the risk taking that people take, otherwise we wouldn't be here, right? Uh, I think risk taking is important. I think risk taking with some mitigation is even better, right?

So I think I, I don't think we should say that disconnect from the internet, you know, that's the safest thing. Let's unplug and, uh, you know, we can all sleep well at night, all that stuff. Uh, but I think we need to put the boundaries and the systems and procedures to detect and, and do all that stuff.

Uh, and, and even if you wanted to, uh, to certify third parties, this then becomes, again, problematic process red tape, and people go around red tape anyway, so it'll be a cat and mouse game. So I think what we should, I would rather do is like, hey, put in place the system to detect the smoke, uh, as quickly as possible. Uh, I know sometimes it's more expensive, but I think the, uh, the red tape and the risk, the risk aversion is I think very bad from a business perspective.

Your competitors might say, you know what, let John be risk averse. We we're going to go and do other things and kick his behind by coming to market with something better. I think, uh, as we're coming to the end of this podcast, and, and as, as always, when we gather together, it's such an interesting group of people.

We could go for hours at this, and I, for one, could probably do with a drink. Um, I think also one of the things I, I really pull out of this is that there is no nirvana state. Nothing is going to become perfect.

And so getting the ability to see the imperfect, to understand the imperfect understanding that there is a tension between business requirements at different phases and, and this conflict is natural and norm, not something to be avoided, uh, is, is an important aspect of not just this conversation, but any conversation where we're talking about innovation and building things. Since we are running out of time, it would be great if, uh, listeners out there would like to catch up with people some more and carry on the conversations. Uh, where can we carry the conversations on with you all?

com. Also follow it at Meyer Media and hit me up on all the channels, LinkedIn, Twitter, oh, sorry, X I'll never get used to any of that. But anyway, you can follow us at John Meyer And, uh, I'm Disco Posse and all social media.

com, find out what I do. org. One of my favorite tools that I promote to everybody.

Thank you. As Catchpoint gives back to, you know, in freeways to the community. It's incredible help for me as a developer to learn.

And, uh, but anyways, that was my last, uh, shout out. But, uh, with that, over to you. Thank you so much Eric.

org. So that's one way of, of getting to understand a little bit your unknowns. com.

We have a great blog. You should, uh, read it on LinkedIn, X, et cetera. So thank you so much.

com website. You can find me also writing various places my own, um, writings. co nz, uh, or in Z for those of you in America.

And, uh, you'll also find me on some of the other FU properties that are writing content out there as well. So thank you for listening to this episode of the Tech Field Day podcast. If you enjoyed the discussion, please subscribe on YouTube or your favorite podcast application so you don't miss an episode.

Do consider giving us nice rating and of review, let other people find us this, uh, great content as well. This podcast was brought to you by, uh, Catchpoint and Tech Field Day, a part of the RUM Group. com/podcast or review.

Uh, view us on Techstrong tv. Thanks for listening, and we will see you next week.