Incident Response for Developers | DevOps Connect: DevSecOps 2023

So I'm gonna talk about incident response for software developers and DevOps folks. So I was a software developer for a long time, then I switched into security and I did application security. And I was not on the incident response team, but every time a piece of software gets attacked, everyone would ask me to come help, right?

And throughout the years I saw many very, very, very well-meaning software developers and help desk folks ruin all my evidence and make a giant mess and make things way worse. And so just to be clear, like this isn't to say, uh, negative things about other people, it's about how we could all do better. And so there's five specific things I'm hoping you can take away today.

Um, so, oh, so I'm Tanya Jenka. Yeah. Um, okay, so next, so what is instant response?

So I'm gonna give you the super, um, very official thing. It's an organized approach to addressing and managing the aftermath of a security breach or an IT incidence. The goal is to manage the situation in a way that limits damage and reduces recovery, cost and time.

But I like to think of it as this. And so the woman on the, on the right hand of the slide, she's having a security event. She's like, something weird's happening and it smells fishy.

But then the woman on the left, she's having a security incident. I look like her during security incidents. And so when you have a security incident, you are certain that something bad has happened.

And so simple terms a security event is when you suspect something's gone wrong, a security incident is when you are certain something bad has happened. And so an example would be, uh, when I got, you know, a magazine rang me an email for a quote about how my company's data was for sale on the dark web, uh, for only $50, which made me feel really bad. Um, I tried to explain to my boss we're, we're so much more than that.

Anyway, he did not think that was funny. So who am I? I'm Tanya.

When I brush my hair, I look quite good. Uh, I am from, we hack Purple and we give training like this. Um, I wrote a book and I've done lots of tech for a long time and I feel like the key takeaway here is, uh, Tanya's a giant nerd and she really wants to help people make more secure software.

So just if you could take that away, that would be good. I have, this is the first time I've ever had to give a trigger warning with a talk. So there is a thing that happens and it's not graphic whatsoever, but the thing happens to kids and it's not nice.

So if you are a very sensitive person, uh, you might want to leave. I'm not gonna get into details, it's just implied, but I don't wanna make anyone uncomfortable. So there we go.

Okay, so your organization, so if you're at rsa, you already know this, your org needs your software to be secure, whether you are buying the software, renting the software like SaaS or if you're building the software. And so software developers, so I'm like, I'm Apsec, I'm in charge. But the software developers with 500 of them, for every one of me according to GitHub, I like, they are actually the first line of defense.

What they do is what actually matters every day. Like they outnumber me at such a gross scale. Now, there's no possible way that I could be like, I'm the first line of defense.

I am like right, like a really close second, like running behind them. But if they don't call me and tell me, I don't know. And so I wanna talk about security incidents and I'm just gonna talk about five things that, so the five things that I want software developers and people to know, we don't have very much time.

I'm gonna try to save time for questions. I have given this lesson where it's 15 minutes and where it's an entire hour. I have tons and tons and tons of stories, but I'm gonna try to behave myself.

So after I'm gonna be in the hallway and you can talk to me as much as you desire and I have stickers. Okay? So what do you need to know?

So your role during a security incident, and I would love to encourage all of you to steal this deck after, cuz I'm putting it on the internet and give this briefing to all your developers once a year. And I'm not kidding. And the first one is tell the security team, I will never be angry at you.

I will never be angry at you for a false alarm. I would so much rather a call in the middle of the night and then it turns out it's not actually something than you not calling me. And six weeks later I find my data on the dark web that feels very bad.

Um, who here has had like a major security breach at some point in their career somewhere where they've worked? Okay, so lots of you are fibbing. That's cool.

Um, so basically I try to tell them what it looks like. So I'm like, oh, is one of your APIs just like never responding and it goes down all the time and you just don't know why it could be performance, but after a certain point you should probably talk to this. Like if you can't figure it out, you should talk to the security team.

Because I've discovered, I'm like, oh, bad guys are are talking to it. Or in one case I was like, oh, you know how we charge for that service? You don't have a gateway in front.

Tons of our customers figured out they could just connect directly and just stop paying us. And by that it means 70% of the traffic and I so security, right? Okay, so number two, okay, actually just I can go slightly slower.

This is the most important one. Just getting them to say something and tell you and not be afraid of you. Whether you have to use donuts to reinforce this or some sort of candy in your office or whatever.

Just getting them to tell you stuff. You will cut down your triage time and like how long it takes you to detect things by a wat if you get them to do this. Okay, so next, don't leave.

Okay, so I was doing this security incident and there's like a, an active cross, a scripting that's being exploited at the time. And the dev went to just go catch his bus and left for the day. And I was like, what?

He didn't leave a phone number. His boss is like, oh, he doesn't do on call as part of his job. And so I couldn't, like I could have fixed the code but I didn't have the authority to push code.

I didn't know how to push that, not break everything. And so it just got exploited all night long. It was a terrible, very long evening for me.

So tell them if there's a security incident and we're talking to you about it, don't leave the building without giving me a phone number to reach you or something. Or even if you're just going into a one hour meeting for your team, just say like, listen, I'm going to a meeting, I'll be in this room if it's truly an emergency, you can interrupt me and embarrass me in front of my friends. But this sounds so simple, but I've had it happen a bunch of times now and just like, please don't leave.

Okay, so next, this is an emergency. So I've actually had developers say like, oh yeah, I'm totally gonna fix that bug next week. I just have a big release tomorrow.

I'm like, no, it's not like I found that you have a vulnerability. This is like, it's being exploited right now. I need you to drop literally everything.

I don't care if you're supposed to have a meeting with your boss, I will talk to your boss. Please help me. This is literally an emergency, um, a way.

So I'm really bad at, um, admitting errors. Like I don't like it. I find it terrible and it turns out a lot of humans feel this way, but I had a boss and he's like, they keep hiding things from us and not telling and trying to like cover it up so they look good.

And so he told this team who had been totally uncooperative during a previous security incident, uh, their manager actually said, Tanya, get the F away. And like was yelling at me to leave that area of the building cuz I was wasting all his devs time. And so he gave them this big talk and told them like, oh we had these three security incidents this month.

And one of them was where a logger got installed on six employees computers. And then I had like this lady that's a grandma and she was like crying as I'm trying to help her change her banking passwords. And he is like, when we screw up, Tanya has a very bad day.

And then he told them about it was their incident, but we didn't say that until the end. He's like, and then Tanya got a quote from the media. People are asking this and our data's over here and all these things.

And and they're like, wow, it's terrible. He's like, that was your app and Tanya came to you and asked for help and you literally told her to f off that sucked. Um, and we need you.

And so explain that it is truly an emergency. Sometimes. I mean only use that if it's true.

Just to be clear. If it can wait until morning, let it wait till morning. Okay?

So follow need to know. This sounds so obvious to all of you probably cuz you all work in security. However, I have had executives walk around and tell their admins all about this super cool incident that's happening.

I have had like given briefings out that are quite sensitive and then I see them in a pile in their suckling bin and then I have an aneurysm. Um, and so I've, I've started in meetings. I'm like, Hey, do you, you have a vault that you're gonna put that report in after cool, then give it back.

I can't trust you. And I remember the executives being like, mom has spoken, but, but seriously. So explaining need to know like does your boss need to know the specifics of the security incident you're helping us investigate?

Or do they really need to know? Okay, so Tanya has this security incident, she's a stress ball. She said she needs me for the rest of the afternoon.

She said to talk to her if you need details. And then I say no. I've had lots of managers be like, well you have to tell me I'm their boss.

I'm like, actually, I literally don't. Uh, and I'm the CISO and I'm above you. And so just go to your room or whatever you're gonna do.

Go be angry. We'll talk about this tomorrow when I fix this. And I feel like when you do incident response, it's hard to make friends with other teams.

Um, but explaining need to know means I don't get into an elevator. And then I hear about that. Uh, I, I was late for work one day like on purpose.

I was at the dentist, I like having clean teeth, this is fine, right? And I told my boss, I come in at 10 or 10 30 instead of my normal time and my team's like, go, go to the big room. All the executives are there, everyone's yelling and freaking out.

And I'm like, why? And they're like, there's a security incident. I'm like, but Eric isn't in that room and only him and I do that.

And they're like, yeah, some dude from help desk is managing an incident. So he decided that one of our buildings had malware. This was a concrete building.

It was not a smart building. There was no intelligence, there's no internet connected. It was a dumb old concrete building.

And he's, he's telling all the executives like, yeah, this building has malware, people in it aren't safe. Everyone's freaking out. Uh, and then all the executives looked at me and they're like, how come you weren't here?

I'm like, no cavities, no one was interested. Um, and so I explained to them like, that's a dumb building. It can't possibly any, they're like, well Brad told his name wasn't Brad, Brad told us the truth.

You always hide everything. And so it turned out what was actually happening was it was winter in Canada and the Olympics were happening and it was figure skating and every single person in the entire building was streaming figure skating to their desk. So the network slowed down until it stopped.

And I was like, yeah, it's just figure skiing. It's just Canadians being Canadians. That's all it was.

And, and then seriously, for weeks after I heard, I wonder if they got the malware out of that building. Ah, so please don't tell people I know. It's ridiculous.

Okay, so the last one is do not try to manage it yourself and be a hero. So that help desk dude thought he was doing a good job. He was like, I was trying to help and I had to have like a little talk with him after.

And then that's how I started making this training. And so me and Eric, so Eric was my mentor. He's a wonderful human who's amazing.

Um, and we still talk and he, he told us a story. So this is his story, not mine, but he's like the last place I worked, one of the help desk people was like, I'm gonna be a hero and I'm gonna help. I see the security teams busy all the time.

I'm gonna take one for the team and I'll help with this one. And it ended up being, um, a computer that had images of child abuse and you know what I mean by that? And he ended up ruining the entire chain of custody and seeing images that I would not wish any person to ever see.

He ended up in therapy for years that criminal went free and that was him trying to help. And so us security folks, us instant responders, we agree to this risk when we take this job, we agree that we might be exposed to criminal elements, that dark web things that are terrifying that no like a load to carry forever. And so he didn't know that you can just like scan it and do other things.

He didn't know that doing these things would create a break in the chain of custody and it would not count as evidence in court. He didn't know cuz he didn't have the training and he felt such incredible guilt for a very long period of time. And so explain to them, this is not your burden, it's ours.

We agreed to carry this burden. It's not for you to carry. And so like a lot of the help desk people are like, oh no, but we're helping.

Like it's like you think you're smarter than us. I'm like, no, I'm more prepared than you. And so if you can get these five things across to them.

So this is the picture slide. So if you were gonna take a picture of a thing and then tell it all your devs, which is what I'm hoping that you'll do, like that's literally what why I made this. This would be the one if you could communicate this to your Hope desk, to your software developers, to your DevOps folks, your operational folks, like please don't be a hero, please follow me to know this is truly an emergency.

If you could do this for me, that would be a wonderful favor and we would have a lot faster, cheaper resolved incidents with less damage. And so with that, um, oh, I'm so good. I'm way, way, way on time.

Awesome. And so if you don't know ask, uh, if they don't know at work, um, I have an online community that's free called We Hack Purple. You can ask there.

So we have like this huge team of volunteers that like go and like help people with things from work. Um, but we want everyone to know come to the security team. We are here to help.

And so with that, I have some free resources. So first of all, I have a podcast because it turns out if you say you have a podcast, you can just go up to famous people and be like, Hey, I have a podcast. Would you like to be on it?

And then you get to hang out with them. It's really cool. I wish I'd known this years ago.

Um, and I just talk about securing software. Um, every Monday on Twitter and on Mastodon, the InfoSec Exchange server. Um, I do this thing called Cyber Mentoring Monday where I pair people with professional mentors.

And when I say I pair them, I'm a terrible matchmaker. What I do is I make this thread and let people meet each other and decide who's actually a good mentor for them. And so every Monday you can check that out.

If you are someone with five or more years experience in InfoSec, can you put your hand up for me? All of you would make spectacular mentors. So don't think you don't know enough cuz there's someone that would love to be able to wear your shoes.

So consider taking a mentee. Um, lastly, I have a community and I would like to invite you and especially your developers to join. We do nerdy things like hack secrets and give each other workshops and every single thing inside of it is free.

And then the last resource is me. I am awe on the internet. I have a YouTube channel.

I got a TikTok recently and you may make fun of me about it. Um, and with that I just wanna say thank you. Thank you very, very much for having me.

Thank you for the DevOps Connect people and Text Strong. Thank you. Okay, well I have a question for all of you.

What was your favorite talk so far today? You could just yell it out and you don't have to say this one, just be, this one's not, this one's not included. So was the other favorite talk You had the security champions talk?

Yes. And you all the great, oh, so he said every single one's been great. Like DJ's not gonna come out and get you.

I know he is right there. He's actually very friendly. Thank you.

Thank you very, very much for having me.