Sandra Cavazos – Product Security at Scale: Lessons from Comcast

This is texturing TV. Well, I'm very excited to be here with you all today. I wasn't sure if I was going to be able to make it to the trip because my son is actually graduating from high school my youngest and there's just a lot of events and things that sort of lead up to the culmination of a high school career and a graduation.

We've been handing all these events and my husband and I have been kind of talking about all the things that we've taught my son in these 18 years and then also some things that you know, we've kind of missed So, you know my son, he's I have to give it to him. You know, he is an accomplished young man. He's done a great job.

He can pick up a viola and he can just like play it like a virtuoso or he can pick up a golf club and he can swing and he can hit the ball and it goes straight down the Fairway. Well most of the time. He can get up he can sit up on a bicycle and okay, I've got to be honest with you if my son gets up on a bicycle.

He's gonna fall down flat on his face. Because apparently I was supposed to teach him how to ride a bike somewhere along the way and I completely missed it. Okay.

So, yes, so my son he comes to me last week and he's like Mom. Hey, you know my friends and I really want to go on this trip this summer. Can I go?

And of course, this is a trip that involves bicycle riding. So now all of a sudden he's motivated he wants to learn how to ride a bike. now, of course, they don't make training wheels for adults, do they It's with knee pads and some Neosporin at the ready or he's going to have to Trail his buddies in this like little kid bike with pink sparkly training wheels or something.

So I have to say if only I had taught my son to ride a bike when he was say like five or six years old not 18 years old. As parents we do our very best to try to prepare our kids for whatever the world may throw at them. But by the time graduation comes around it's a little harder or maybe a lot harder to you know, learn major life skills those types of things are much better built in versus bolted on at the end and so with parenting and products security.

It turns out that proper planning and preparation are essential to launching secure products into the world. Hello everyone. I'm Sandra Cavazos the vice president of product security and privacy at Comcast.

At Comcast we are building just amazing technologies that connect millions of people with moments that matter but with that comes a tremendous responsibility when it comes to cybersecurity after all we can act hundreds of millions of devices with millions of customers and also hundreds of thousands of employees and partners. So cybersecurity at Comcast is a major operation. Now for me in my role at Comcast as you can probably imagine, I spent a lot of time with leaders of business units talking to them about cybersecurity.

and there are a lot of topics that are just difficult to talk about difficult to explain right or difficult to convince them of the importance of but what I found is when it comes to building Security in versus bolting it on or moving security to the left. I find that this topic resonates with just about everyone. Like the story with my son and the bicycle this is something that people can just relate to there's so many examples of it in everyday life.

and What people have really come to recognize is that the old way of doing things which may have been where a product team built a product then? They threw it over the wall to the security team to kind of do some assessments find out what's wrong with it. Throw those findings back over to the product team to fix that does not cut it anymore for at least three major reasons.

The first one is that the old way of doing things involves inherently rework you build a feature insecurely and now you have to go back and re-architect and redesign it and rebuild it the right way. So now you have to you have to expend more energy and resources. It's inefficient.

So that's the first reason the second reason is because you have Time pressure so, you know when you're about to launch a new product that is about the worst time to figure out that there are security issues with it. That's when there's incredible time pressure to try to get a product launched on time and finding out that there's a critical security vulnerability could mean a delay to your product launch. And finally the types of Assessments that are done by humans like threat models and Pen tests since they're done by humans are inherently not going to be complete you're gonna miss some things.

So there are a lot of reasons why the old way of doing things isn't sufficient and why building Security in moving security to the left is effective in reducing your development costs. Would you sing vulnerabilities and ultimately your risk of incidents which is the end goal. When we talk about the importance of the secure development lifecycle.

We talk about the business drivers in terms of protecting our brand reputation people trust Comcast as a brand and we have to make sure we protect their information and that we're true to that. We also talk about the increasing sophistication of the types of attacks that we see. That something that was Secure a year ago or five years ago is no longer considered secure today.

And then of course the increasing regulation pressure, the new laws that are coming into effect that require us again to up our game and to have higher and Tighter controls than we had to have in the past. So, like I said, it isn't difficult to convince people that building Security in is the right thing to do. It's the smart thing to do.

It makes good business sense. But doing that at scale is a whole nother matter. It's much easier said than done at Comcast.

We have over 700 development teams building all sorts of different products on disparate technology stacks. And so the question is how do you do this effectively and efficiently with a small dedicated team? How do you do it at scale?

Well, that's what I'm here to talk to you about today. So I'm here today for you. All From The Trenches to tell you about what we've learned at Comcast that works what's working for us?

And what I'm going to do is I'm going to step you through our whole secure development lifecycle process how we've built it and what we do and then we're gonna take five steps along the way where I'm gonna zero in on a key lesson that we've either learned or even we're still learning and you know really kind of give you that inside look at what it looks like to do product security at Comcast so that you wherever you are if you're at a large company a small company no matter what industry, you know, these are Universal things I think will apply that you can take with you and apply to wherever you are. So I hope you take away some things that you can apply wherever you go. So when I talk about our secure development lifecycle program at Comcast, I always like to start with a guiding principle since these underpin everything that we do this is what we believe in.

I won't belabor the point. There are three of them. We've really already touched on building Security in over bolting it on these look a little bit like an agile Manifesto by the way, that's intentional.

Right? So this is something that we talk to people across the company about about how we designed our program implementing features securely over just adding security features. And then iterating and learning continuously over gaining decisions that throwing it over the wall story.

I told you. Another key principle for us is empowering the development teams over relying on a few security Specialists. Why because of course it doesn't scale and because when development teams take ownership and and for the security of their products, it's done better and it's more effective.

Growing a culture of secure practices over policing enforcement. This is really important too because although policies and standards have their place and and I'll talk about them a little bit. Ultimately.

We want to build a culture of continuous Improvement and building security into the culture of the way that the team works. That's more effective way of doing it. So now what I'd like to do is focus in on our first lesson.

Branding a company-wide sdl program and then presenting it with a taxonomy a language even visuals has been really important at Comcast for solidifying and making an effective sdl program that people have grabbed on to so at Comcast we have done a lot of corporate Communications. We've published articles. We've published a white paper.

We have a lot of materials and we've done those Road shows and that branding that we've brought to our sdl. You know, what we have we've had an sdl security problem lifecycle for many years and even a formal one for five years, but what we didn't have was this branding around and what we found is that this is really really helped our company to make strides in terms of the effectiveness of the program. So this is how we show our secure development lifecycle program at Comcast.

We have 12 practices and we show it in an infinity loop diagram because we are showing the dev devops, you know, Loop here where the left side is larger because we want to emphasize the security practices on the development side versus the operation side that's moving security to the left. So with each one of the development practices, we have an Associated security practice and by showing this continual, you know, it shows that obviously the products are constantly iterating and changing. So we're trying to do this in in a place of constant change not really a linear flow.

The first few practices deal with preparing and planning and I'm going to walk through each one of them. The first one is really about our policy standards and guidelines which in some ways really underpin our entire program. Policies are high level statements and then the standards are really the requirements that teams need to comply with and the guidelines are how do you do it?

So for us when we work with development teams, the policies are too high level, but the standards are extremely important having making sure development teams are aware of what their standards are is important. And the other thing that we've done at Comcast is we went through an effort two years ago to completely overhaul our standards. We rewrote them completely and we made them simple statements that are very tangible and measurable and and they're not that lengthy so that that teams can really wrap their arms around them and they can know exactly what is required.

So I really recommend that for organizations that are looking to up level in this area. The second practice is artisanship. Again, really important to train your developers and other roles within your team all around cybersecurity and we have a program that goes through levels sort of like a martial arts program white belt.

Is that your typical we call it be cyber Savvy training that's where people learn not to click on links and emails and you have secure Wi-Fi or whatnot that is Corporate mandated required for every single employee. No brain or no problem. But the rest of the belts are the ones that we deal more with our yellow belt training is a custom Training that really informs people about the security development life cycle and our practices at Comcast.

What we realized was that most people who develop technology in any capacity and I'm talking about here whether you're in development or operations or QA or even product those people need to understand what how to build security and to products and so we make sure they understand this is not off the shelf training we've developed It so that they know what our processes are and how they engage with us. The orange belt training is secure coding training for Developers. This is not something that people can just opt into this is something that is mandatory required.

If you develop code and whatever languages you develop in you're required to take the training in so we track that so, you know, we found that this to be also very important to be to make it a requirement and then we have levels beyond that. So the green belt training requires a lot more expertise. We like to have one green bell on every development team.

That is the person who can do the more complicated feature reviews and code reviews. So they have a higher level of skill. And then we we also reward those who go even higher The next practice is called initiative onboarding this is all about the planning right avoiding the bicycle story.

I told you about so with initiative onboarding is where every time a team or an organization at Comcast is building something new or making a major change that affects the the attack surface of of the product. They need to take it to this initiative onboarding process where we determine what steps need to happen for security and when they need to happen and they get planned in to the product development timetables and roadmap so we found this to be really important otherwise things end up getting missed and so we've developed a whole process around this and we've been educating all of the teams about the importance of doing this planning properly. The next set of practices deal with the building Security in so now that you've prepared and you understand what you need to do and you've done the planning now comes the really tough part of building Security in how do you do that?

So the first part we can't ignore all the vendor dependencies that we have right we deal with so much third party software. And so what we do is, you know, we really vet all the vendors that we work with the Comcast. And so no matter what all come into this pipe.

I show this as a funnel diagram because everything goes through the top of the funnel but then depending on the risk of the engagement and the vendor and what we find from the risk assessment that will determine what type of Assessments we need to do. So the very smallest part at the bottom is really where it intersects with our security development life cycle the most so whenever there's an engagement that involves a lot of risk if there's sensitive data that's being shared. We always take it through a threat model and a prsa which I'll talk about which is our End test process so that we can make sure the integration is done securely.

Such an important way for getting the development team together synchronously. With an architect to be able to map out. What are the types of attacks?

Why would they happen? How would they happen? And how do we stop them?

And these types of threat models always end up with an architectural diagram documentation and at a plan for remediation based on what is found? We always did these around a whiteboard that was kind of the rule when we started. You know, you had to have everybody in a room, even if it meant traveling across the country and then we had to get really creative with covid and we did and we adapted so I'm gonna stop for lesson number two right now to talk a little bit about our threat modeling program.

So we have had to innovate and change partially like I said because of covid we've gotten really smart about how to do whiteboarding sessions on Microsoft teams how to use the technology to be able to interact with the architectural diagrams in such a way that we're able to do what we could do all gathered around a whiteboard, but now virtually which is great. The other thing that we're doing is we are looking at other ways to again scale and improve on our threat modeling program. So we've brought privacy into it.

So we're before we just we really focused on cybersecurity and then we went into Data protection which is sort of that intersection point now we look at even more things in privacy space like the ability for customers to often and opt out and whether data retention is proper and and all of those sorts of things. So now more of these things are coming in scope for us, which has been great. The other thing that we're doing is we're working on new ways of automating bringing data to the the architect who's doing a threat model so they don't go in blind.

They have a lot of information about the team about how they work about the platform about the servers that they're running on. They have all this information at the ready so that you know, they're better able to go into the areas that may be may have issues. Practice in our sdl is privacy impact assessments.

So this is where we work very closely with our legal team to understand the Privacy implications of what a new initiative is doing. So there are a couple of important things that come out of a privacy impact assessment. A lot of times.

We'll realize that there are security assessments that need to be done. We also understand what needs to happen to be compliant with regulatory requirements and the contractual language and make sure that that's in place and we make sure that we're compliant with our internal policies. The next practice is definitely one of the most important ones this is secure coding.

So if you're developing an application, how do you make sure that as your coding it? It's being done securely. We use secure design patterns so that once something is solved once it can be applied to many places.

We have code scanning and we break this down into two categories first party scanning and third party scanning since of course, we use so much open source and third parties libraries. So we require it to have first party and third party code scanning for all of our code. We encourage all of our teams to build this into their pipelines so that they're getting into a pattern of resolving these issues every time they merge so, you know, we talk about only merging secure code is one of the practices that we work with teams and we coach them toward and then finally the peer reviews because tools can only catch so much you also need humans involved and so we train and we upskill someone on the team to be able to conduct those Security reviews.

The production ready security assessment is our pen test and you know, we call it a pen Test Plus because we look at source code. We look at configurations. We look at everything from the infrastructure all the way to the application.

This is an iterative process that our assessors are undertaking to look for any sort of way that somebody might be able to compromise an application. In order to do this. Well, you need people with really strong skills in this area.

And so what I'm going to do now is pause for less than three, which is really around our pen testing team how you keep them engaged and motivated. And also how you scale pen. Testing does pen testing is very expensive and you know, when you maintain as many applications as we do at Comcast, you can't possibly pen test like absolutely everything so so how do you do it?

So one of the things we've learned is that our assessors love doing hackfests and what they call continuous penetration testing, so when they find something in an application, they find an issue then they go and they look for that issue all across the environment and they really enjoy doing that. So they'll all get on a bridge and a Microsoft team meeting and they'll just kind of hack away and they'll look for this all over the place and they'll find all sorts of things. So, you know, that's a really it's it's great for scaling.

It's also one of their favorite activities to do it makes them, you know, Some sharper and then tooling so we're always again innovating and and our pen testers are rarely creative about finding ways to automate the things that they do, which is really also another way that we scale. So we have both we have a team of full-time employees dedicated. Who and those are the ones that do the hack Fast and The Continuous pen testing and even the tooling but we also use third parties to do large numbers of Assessments.

And those are more just like, you know, transactional contractual type of you know, we have very specific, you know requirements of what they need to cover and then they do that. Yeah. It's good to have a mix.

Actually. They work really well together. So Security reviews, this is the bookend to the planning.

Orange one I showed you of initi. Ative, so it's so important that after you've done all of your assessments. You've gone through your plan.

You're about to launch your product. You take the time to have a full-on security review with the stakeholders. And this is a lot of times involves very senior folks in the company who are interested especially again a high profile product, you know, you may even have you know, svps and EVPs in the room.

So what we do is we bring all this data together with of course a recommendation that would say maybe there's a critical vulnerability that we say you really should resolve this before you launch this product to end users or customers and then we have others that we might say. Let's get this done in 90 days. Let's get this done in 180 days and you map out a plan.

So effective because all eyes are on this product. All eyes are on this launch and everyone wants to do the right thing and they want to reduce risk. And so that's the really the best time to have these types of Security reviews and bring people to the table where they care the most and you can get their commitment and then you just have to follow it through Bull, you know think about security development is the left sdl is the left side, but we look at the full devops cycle.

And when we talk to teams, we talked to them about their security end-to-end including the operational aspects. So we talk about technical controls process controls patching configuration management, all of those types of things that are of course essential to making sure that a product is is secure. We have a piece art program that runs bug bounties with external researchers who look for vulnerabilities on our products has been great.

We have playbooks so that the development teams know who to call when to call if they notice an anomaly in their logs. And then we we also conduct tabletops so that people are prepared. They know what the plays are before they get into an incident type scenario.

Finally our last practice is called reporting and when we meet with teams, we don't just talk about what they should do and what the practices are, but we also show them their data. So my lesson number four is called gamify and what we've done at Comcast is we've created a fico-like score, you know the FICO model for your credit rating. It's a score like a FICO score for every development team out of a thousand points.

So the first thing they get to see is their score and they get to see exactly what it's made of it gives them a single pane of glass for all of their security data and helps them to know how to prioritize the things that they need to do because they're all weighted. So, this is our Comcast X cyber score at a glance. We bring all of the data from the 12 practices in and we wait them based on the csos priorities for a given year.

So, you know the ciso, you know will sit down and balance these based on the types of incidents or tax. We're seeing or what threat Intel would show and then calculate a score for each team based on that. So it gives them a prioritized list of things to do to maximize their score based on ordering by opportunity points from highest to lowest.

RX cyber score has some important attributes. The first one is that it's fully transparent teams could calculate their own score because we provide them full drill down capability to understand how it's computed. It's action oriented.

So it's based on what is open today. Not their performance over the last year that way any Behavior changes. They make will have an immediate effect.

It's also Motivational because we don't base it on a compliance SLA date or you know a due date. So a lot of times we'll say a vulnerability of a certain type has to be fixed in 30 days 90 days 120 days, but X cyber score gives you more points. The faster you resolve your vulnerabilities versus cutting off at a particular date that way teams are motivated to continuously improve the rate at which they close their vulnerabilities.

And finally, it's growth minded every one of the numbers that's calculated is connected to a practice that they're working to get to culture so that they're making it easier and more the way that they just operate as a team. So we always focus on on that the cultural mindset and building this into the way the team works. They'll leave you with.

At Comcast we developed a program many years ago, and it's flourishing today of having an sdl coach meet with a team every 90 days. The coach will sit down and show them their ex cyber score make sure they understand how it's computed and talk to them about how they can build a culture of each of the practices that they need based on the Technologies. The team is building and what the data would show So what we found is this sdl coaching model is is very effective in working with teams.

We onboard a team this this is just kind of showing you that we have a process of onboarding and then a continuous cycle of coaching every 90 days why every 90 days well because When the coach sits down with a team they come up with a plan for what to do in the next 90 days. Like maybe three people are gonna take their orange belt training and maybe they're gonna reduce all of their vulnerabilities coming from first party code scans. That plan gets Revisited in 90 days.

Do we do what we said we were going to do let's check in. What can we do better? And then they make another 90-day plan and by having that continuous cycle and touch point the team really is able to grow over a given year or two years in a way that you can see reflected in the numbers.

Yes, we do. We see all sorts of uniquenesses and differences at Comcast and we've been around for a long time. And you know, we've also acquired a lot of companies along the way so they're you know, there's just nothing homogenous about our environment, but I think you know what you see is that the coaches get to know those nuances that's part of the relationship that they build and so they know where the team needs to lean in they know what the challenges are.

And so it's just it's just not a one-size-fits all so it's a really good point. So I'm gonna leave you with this today. You know, I'm gonna go back to my kids.

We're just a moment. My older daughter she a couple years ago. She decided that she wanted to try to learn how to ride a bike.

So, you know her friends said they were going to teach her and they got her up on a bike. They went to a park and they're all like you can do what you can do it. We're gonna teach you how to do this.

But unfortunately my daughter also she ended up just with a face full of gravel and a wounded Pride as well. And so these are the five things that you know, I wanted to share with you today is lessons that we've learned at Comcast and I feel like if you really apply this security development lifecycle to your program that you're going to have be able to build more secure products. And also you're going to be able to avoid a lot of skinned knees.

So thank you very much for listening.