John Willis – Investments Unlimited: Fiction and Practice

you Hey everybody. My name is John Willis. This is a presentation based on a book that recently published actually here in September 2022.

And I wanted to go through a little bit of the sort of book itself because it is a fictional story sort of like the Phoenix project. It's sort of a derivative of the Phoenix project produced by it Revolution, but I wanted to talk about like where it came from. The real practice was even though it's fiction.

It was based on a lot of practicality and actually real world examples. So anyway, for most of you that probably see me around doing this stuff. You know, this is my quick resume.

I probably most known even today for being co-author DeVos handbook. Although I've done labor love project with Gene Kim on beyond the Phoenix project and then we'll talk a lot about Investments unlimited and I'm actually in my third draft of 10 year. Um labor of love, you know sort of white whale which is a book about Edward Deming.

So it should be actually out pretty it's in the third draft. So the third draft reviews looking pretty good before when we get to investors living. I'm going to talk about that green book on the upper top, which that was the book that sort of drove the conversation to create this novel called investment.

I've worked basically everywhere done a lot of startups and recently and a new company left red hat and It's called ergonautic. It's a startup and I'll tell you a little bit more about that. There's kind of interesting but just his names go, you know, it's hard, you know that the hardest part of building a new company is finding a name, but we really like this one and applied science concerned with designing and arranging things people use so that the people and the things interact most efficiently and safety.

I like that. There you go. It happens to be founded by for of the smartest people I've ever worked with in my career Andrew Clay Schaefer a little idea.

He was a co-founder of puppet lab. She built an incredible organizational structure. There we go.

All right. Yes did it. Let's go back.

That's it is our first break Andrew Clay Shafer. He built. The pivotal Labs incredible story and he created a team that well, the four of us were at red hat called the global transformation office.

And now we're actually independent in the studio Venture me John Willis. We start by Kevin bear was the co-author of the Phoenix project and Jay Bloom is had been working with Kevin for years and he's getting a PhD in design transition from Carnegie mail, and we kind of joke. He's the smartest of all four of us.

And so the the book The this Investments unlimited so September 13, it dropped. There's actually nine authors, you know, the Basically on Amazon they usually list three and then you have to select the button to get the fourth one in so I'm not even listed on the the main page that you look up but it's actually I've been out since September 13 and it's actually on a couple of SLS like a 14 in the morning category you have there's always crazy categories of business books, but but it's getting great reviews. And and so that the thing is and I'll talk about how we we wound up with this book.

But in order to do that, I have to talk about three people. Um, I'm the guy on the right the lower, right? But Topo pal and I in like 2000 towpower was the first fellow Capital One.

He had done tremendous devops stuff and he early Pioneer and kubernetes and containers and just a whole amazing career in general really brilliant guy. We started having conversations as part of Gene Kim's Gatherings about like the mess of audit in it and it risk and audit and you know how this devops and audit thing or just still not gel. And in fact, there's still not telling me today in my opinion.

And so we we decided in 2019. We did a couple of things together and audited about Center for Summit but 2019 we decided to write this paper. It was that green book and I'll go a lot more into that.

I showed you on that one page and I invited John and the rest of toski. We just come John rest simple and he's okay with that we because he was working at The PNC and so he was running big devops initiative there again part of Gene's tribe. If you will and things I say don't get over nine peoples that worked on this but Tob and I literally sort of invented the idea that only winds up in this book and John went back after we wrote that book in 2019 and implemented and created implementation of it for like two or three years and ran the bank just we call it devops automated governance or modern governance.

and and so like if you think about the like Knowing the past to the Future like that, you know for those you like probably everybody who's listening knows this but you know, John Osborne Paul Hammonds 10 deploys a day. We had a very heavy technical influence from Netflix in devops. So microservices Adrian cochroff and then you know, it's just one Milestone but like Amazon reinvent, you know sort of it.

It was the whole deaf secops conversation. I've done plenty presentations on that and Amazon doesn't deserve the sort of credit for Jen leech. Actually Manifesto.

You should follow her work. She's brilliant and but in 2019, we tried to put it all together. And in fact the that green book in the upper right was a reference architecture specifically die designed around could we prove out this idea and I'll talk more about the idea in a minute of and by the reference architecture was designed to be a microservice sort of a Java microservice get customer ID and could we create evidence all the way to and you know?

Doesn't have to be kubernetes, but the admin controller and be able to gate a delivery of software and again that turned into two years later a novel called Investments unlimited. Yes. So the problem, you know, so the problem is anywhere it's in highly regulated.

So what happened is, you know devops took off we obviously short obviously with the cloud Titans, you know, the Facebook the Twitter is that you know, they would say they were doing devops before devops existed. Basically, everybody can see that has been doing this for more than about 10 years. But but in a lot of ways the big Banks even though they did a lot of devops things Banks Healthcare highly regulated environments still we're not getting that sort of non-regulated sort of short burst of you know, sort of delivery.

There was still a lot of like sort of especially for the customer facing at least in financial, you know, you sort of cliffs that they still they can do a lot of things in that box, but there was just certain things, you know is you know, some of my friends say in a bank like, you know, the the death penalty in the bank is you lose your banking elections. Therefore you're not a bank so Go back this another one of those stupid things. So the and then is working with IQ Evolution and Gene Kim's organization as far back as 2015.

I was I've been over the years I decided not to be an author. Sorry, there was more of a collaborator and a lot of papers and so we got the 2019 paper but it was a paper in 2015 and a collaborated on you know, basically an unlikely Union devops and audit and then in 2018 a really fascinating paper culture order which was an apology letter to Auditors. But but I'll show you it was a lot more than just a tongue-in-chief apology letter and I talked about what happened in 2019.

We actually created a reference architecture and ultimately 21. We created a novel, you know, sort of fictional story of house how a company implements this idea. So dear auditor was fascinating.

It's actually out there as a GitHub project. It's really fascinating. It's worth checking out.

You know, it really wasn't apology to the Auditors and say, you know, like we will bring you along. We're sorry that Did this will be more transparent. So it was really sort of developers and operations.

You could Loosely say devops folks. But but in there that like I said the you know, the first three pages so is an apology letter, but then there are about 30 or 40 pages of What's called the devil's risk control Matrix RCM and you know, and I think that what's important on this is anyway to understand and you gotta remember this is back in you know, five or six years ago some of the things that you know, I wasn't wanted to core authors of this but I worked as advisor, you know, the things that like we're still struggling to get right right now early I said audit is still it risk and the the connects between it risk and and we'll just call devops, you know developers and operations. It's still a mess and most organizations and so identity management Secrets management all that stuff as part of the pipeline.

I know a lot of people use vault in a very successful at that, you know. Breaks due to human error. This is sort of making sure this is really I would say sort of the infrastructure is code genre, you know now it's more containers but like everything in source code, you know, everything configuration, you know your bills everything in there and we'll get into like how devast automated governance the DSL for the language.

It drives. The governance also is a artifact this goes in there your material Miss station. Yeah misstatement of financial data, that's basically a former segregation of Duties, right intellectual property and license violation.

This is where we sort of want to introduce again the ideas that this evolves into automation software composition analysis, even you know, like espombs have been real popular this year and the end of last year but mostly issue. But like we were having those conversations, you know, five six years ago about software building materials the You I'm sorry, just the tokenization. So basically data breach or encryption at rest the deployment strategy.

So that's really the number six there or I'm sorry seven getting a little bit confused here the business continuity but deployment strategy so The Unwanted customer impacts. Sorry a blast radiation. It's just a lot to put remember, you know, but I was trying to remember all without having to look at my notes, but the deployment strategy blue green feature Flags those kind of things but eight is interesting because the eight sort of part of the RCM was the Divergence of order evidence.

So this is something we're terrible at and it was really one of the goals of that original Green paper, which is could we do better at or and I got more coming on that and then gdpr and data jurisdiction and then finally, you know thinking about red teams. And in fact, one of the things they talk about each earlier like she's built massive 70 80 people red team And financial institutions, she's run. And so so I talked about I've talked a lot about this green book The devast automated governance reference architecture.

We a lot of us called dag the devast automated governance. And and so the when toppo and I was sitting around 2018 and we were thinking about what kind of paper we wanted to write in 2019 as part of jeans, you know sort of event structure because of the foreign papers group 50 or 60 of us meet in Portland to come up with these ideas. So good idea Revolution is 70 or 80 books out there.

But the the what was we wanted to think about could we increase the efficacy of audit? It risks development. Well reducing the talk right that that's that is the problem in many organizations today, like most of the what the evidence is again going to how the how is turned subjective?

So how we would do it? So the how we would do it is turn the subjective evidence into objective evidence. So what we mean by that is, you know today a lot of people create their evidence for order and it risk you search now records and sort of subjectively say, well we did this we did that and then maybe have pointers to logs and and that's a little more objective but it's still decoupled from the process.

So so the the toil is that most organizations spend 30 or 40 days dealing with Audits. And and for those you've been through the it auto part, it's just a mess. You know, it's Email exchange is and Screen prints.

It's why does the you know this certain log not match that log, you know, why does a Sony Cub? Not right? That's just on a tight law, right?

It's just it's very nonsense. So could we let the computers do the work through some sort of blockchainish like mechanism. And so we did is we sat down to tonight 2019 and we we thought about and and on the 2019 was a you know, just an incredible group from Nike Microsoft PNC Capital One Marriott, and we thought about we collaborately tried to discussed like what are the requirements for audit?

And what did the gates you know, what is the evidence in the gates? And we should we looked at could we decompose the software supply chain into you know in this case seven, you know different stages and you see dependency and artifact is sort of its own sort of flow within the flow of the software, right? And so we did that and you know, we early on we came up with a lot of great example not all of them originally in the first book, but they need to presentations I've done and the people involved we use but the idea could we automate Some of the things that that are required as part of IQ risk or evidence for you know GRC governance risk and compliance, you know things like code quality unit test coverage maybe a percentage change size psychromatic complexity may not be a direct, you know nist or or a PC DSS a pull request review.

It's almost everyone every compliance structure. I know requires some type of evidence of reviewers a branching strategy clear dependencies. And then in the build stage, you know things like to build ID the version the configuration linting SAS scan all these things instead of humans trying to document them in a change record and then saying well go look a little telling the order or IQ risk internal order go you figure it out because here's the lock right?

Like it's 20 just gonna be twenty twenty three come on people, um, you know, and then just in the package stage artifact versioning code signing, you know, I I had the opportunity to go into solarwinds to do talk to the executives about this automated governance process after they hack and you know, I did some research on crowd strikes analysis of using the miter attack framework and like it was clear that they hijacked the code in the build and compile process but like a simple, you know, I hate to say, you know, this is what they call counter facts all mean to say sit here now and say if you only did this and that's not what I'm trying to say, but the point Seeing there was no code signing of images and that's table Stakes for devops automated governance. And then finally apart Prairie pod pod you things where you might have threat modeling just that the point being is you can create now a holistic approach anywhere from a dependency assassed a dash an SCA to really could be like it needs to have three rabbits running in a wheel that runs the engine. I mean, I'm joking but the point is like cyclomatic complexity or test coverage.

They might not be directly part of a specific anist or psidss or HIPAA requirement But the point is what is the point and the point is is to protect the brand right? Well, you know, why do we have audit 19 risk, right. It's to protect the brand.

And so, you know, one of the things that we didn't get enough time in the book, which was to prototype this idea. If you had all of these we call attestations but evidence and possibly's for gating. Could you create a yamo-based infrastructure that could be interpreted by the machine and create a blockchain like?

Output which was non-tampable. And so the reason genres Tasker John Reza. So important in this story is he went back and built a system, you know, here's just a simple example of a particular artifact that maybe required a sort of a particular component structure for the name.

Required a particular component structure for the name. You make sure the verification build server artifact server, but you can see down there in the bottom unit test coverage. So like literally typing in or using a variable substitution like model to say it had to be over 90% test coverage or 80 to encourage very very powerful.

And so the book itself the Investments unlimited is what we like to call a core chronic conflict. And so if you know the story of the Phoenix project, you can go back, you know, 20 30 years early, but Elliott gorette wrote a book called a goal. And and he has this concept of a core.

It's a novel editor. It's like the best way to teach people is put in novel format and Jean and Kevin bear and team decided to create a modern day rewrite purposeful modern day rewrite of the goal with you know gonna give it away. But everybody at this point should have read Divine project.

So it's in you know, in instead of enclaves and automated robots. It's a Java stack and a systems, you know, it's just the program or you know, but so investors unlimited follows that thread we use the very same a very cold rating our type structure. So this book is you know is about a company thousand employees 20 billion market cap.

And here's the cake kicker right? I won't give too much away about the book but they there's a thing in financials called Matt is requiring immediate action and matters require an action. And so so the CEO gets a heads up from the OCC which basically regulates banks in the US.

They're going to get an MRI some banks. This is very dangerous. This is really this is failing in order basically and she just goes to the devops group and says wait a minute.

I thought we were doing devops like I thought we were doing good. We presented, you know devops Enterprise Summit and get cheers and they're like, yeah, but not security. He's like, okay and then that becomes a dialogue of understanding what they didn't do what they needed to do and like sort of what devops kind of got them in a little bit of trouble thinking they were doing all the right things.

And then there is we we created our own dear auditor letter where in the book there's an apology to the Auditors within the organization. And so if you followed the goal or certainly Venus Project, there's always this like Socratic like dialogue which I love right? And you know, it was Alex and Jonah in the goal.

It was Bill and Eric in the Phoenix project. So we we wanted to you know, say true to the the gold ratting architecture. And so we we have adjacent cobart.

Who is this sort of Jonah character or the Eric character in Bill Lucas who is in the goal. He is the Is the Alex sorry and then in Phoenix particular rules to get to all the remember all this but in the Phoenix project it is Bill. So we make sure we created that we had all these great characters again a very Phoenix project here, right like and and there's a scene in it like in the book like Sunday, they'll make the movie.

I don't know. I don't know who played Jason but maybe we'll see, you know, maybe I don't know the great actor Tom Cruise with Playstation program. So he says to the CEO and they're sort of and certainly Bill he says, Your devops has failed you and like and that sort of breaks down this thing.

Like I don't know never outside come we're getting mrias. You know, what, you know, are we at risk of losing our banking license? And and so they sit down and they start thinking through this idea.

Like, okay. What are we gonna do and it's a great story. And here's the thing that makes it great.

It's nine authors from all, you know, I I basically have driven this, you know, I'm not trying to sound egotistical but it was my idea to have that conversational topos my idea to create the first team to create 2019. It was my idea to to create this project which originally was not gonna even be it was just gonna be another version too are reference artificial but we realize it was so boring. We created to a novella Jean came in his team editorial team decided.

Why don't you make it a book we spend another year or created a book. So the story then comes from all you know, Jason Cox It Disney and Topo pal now at Fidelity. What was the capital and John reset at PNC and Helen Beals work some of the largest banks in the US and and build that and I can go on and on about all the different people.

So when we read the book you feel like this is real stuff because it's coming from you know, I think I Actually 200 years of experience. And so here's the interesting thing. Right?

So you look at I love the you know, so go back here that you know the promises like we have like non-documented software at least process how many people like in you know realize that that's their situation but I love this one because I see this all over I did qualitative analysis with lots of organizations, you know, there's I do qualitative analysis with a lot of organizations large organizations and and like you see this all the time. It's like insane devops Paul, you know, like there some people are doing this but they're not documenting. Everybody does a different but here's the kick right in our book before I'm gonna show you the next slide which is sort of, you know, fact becoming fiction or fiction becoming fact right with so are those we're gonna have like these A 15 MRIs manages requirement not as requiring attention.

And those are like the OCC saying hey, you know that I don't like this. I don't like this and if you stick around too long and you and the reason why the CEO is Susan Jones I think is like so freaked out is when you get the MRA they're saying hey, you know, you've been ignoring us on these NRA's and literally while we're doing this we were able to talk to somebody from from the Japan's Mitsubishi North America Bank, basically that got a cease and desist because they had about 15 MREs outstanding for about 18 months like it's like wow. Yeah, we come up with official story and then we wind up finding that there's a real life exact store now in their case unlike Investments unlimited.

It sounds like we don't know exactly what happened. They ignored the Mr. IA and I I don't own a bank but like that sounds like a really stupid thing to do is ignore.

MRI A and so you so that you'll get to kind of join the Journey of how the team very much like the Phoenix project or even the Unicorn project was jeans sort of second book where you know, they they try to discover this problem and the teams start working together and they become teams and they're they're sort of discovering through collaboration. It's a great book. I mean, you know again if I look at my co-authors the other eight cars, they're brilliant people and just it's getting really good reviews.

It's just fun. It's just fun and I'll say this I think it's you know, one of the things I've been telling people is You know in the early days of devops, you know, Gene had written to Phoenix project, right and and like a lot of times I give a presentation about devops and somebody come up to me see John this is great. But how do I convince my manager?

And I'd say tell you what buy a physical copy of Phoenix project give it to your manager. Now you imagine what we read it right away. You're gonna have to pester him or her a bunch of times but at some point they'll feel guilty in the read it and then they'll be oh I see what you're saying.

This is gonna be that book for security. That's my opinion. You know, let's see if I'm right in a couple years, but I really think this is kind of book that you know, you're struggling to try to help your leadership get better collaboration with it risk and internal audit and even external audit I think external art is gonna take this book and you know, you're gonna see all the big guys big people the big corpse and you know, the big four big three big eight whatever they are today.

They're all gonna basically use investors unlimited book as they're like, this is what we do, so Yeah, so you're gonna I think this book is going to be the kind of book that. Explains to your leadership the things that you're hearing at devops and DevSecOps stays. And one of the things also I love in the books.

I've done a lot of this work and it's one of my sort of major cards that are fair amount of contributions. Although the really big contributions were the the companies chasing Cox at Disney John reset and PNC teleport Fidelity. It was the people that really been doing this Helen been doing this and Bill Benson, you know, I just put the right people in the room, but I really been gracking over this idea of the three lines of defense and how it gets us in trouble.

So we try to do is explain very much accept separation of Duty or segregation and duties. You know, that that unlike that first book back in 2015 ish. Um, you know, like you look at it to it you like I can't live a software in a devops lady because of that somebody Or by the way, I actually can because I can because of devops I can do it and it's the same thing like this discussion happens in the book where it's it's Jason is trying to tell the bill.

Um, you know, by the way, you can kind of have your cake you need to do with the three lines dance. It's just the way you're going about it where you're purposely separating or segregating or creating. I would say like a virtual firewall not a technical firewall between your internal audit and you're sort of owners of the risks.

So and you know, that's that sort of in between group that tries to show to translate and by definition they're separated, right? So there's a great discussion and I've got a lot of passion about fixing this problem and you know, one of the things that you know in the book is sort of black and white but you know, we did a mock up prototype of like like you put all this together. Definitely get rid of that Brian.

All right back, three two one. So we put all this together. And so when you start seeing the stuff I had in those Matrix is early right with all those things that you can create this whole list of view.

You know, what some of the companies that have implemented this already is you can put this in your Jenkins or bamboo. Do these badges this idea of like showing the developer. So instead of like telling developer.

Hey your build broke if you want to find out why go look at these four locks. Right as opposed to there's a DSL that's actually in you know associated with the artifact that delivered it that when I showed you earlier that says these are the things and like for example some a component test failed, right, you know, there's some sort of brown or yellowish open source scan didn't work, right. So so there's sort of some gates that will stop the building some will just sort of notify but all the other things like, you know unit test coverage at 92% right like, um, you know, like and if that one failed, okay, well looks like You know Sonic you gave me a lot of you know, and nicely about this system.

It's all web hook-based. Right? So like you're not changing the tools that you use, you know, any of the sonotype Jay frog, you know, x-ray sonar Cube.

I mean I go on and on there those tools all exist. It's just built on a like instead of those ones just riding the logs and telling the order. It's go look at large.

They're getting controlled by the computer. Remember that it's subjective to objective. They're getting told by the computer like what evidence to write based on what they did and what had a gate and just the end the story which is sort of like I love this part of the story which is so they get through the whole thing and it's the big demo and like the everybody's so excited.

They know they've been collaborating and getting oh there's that kind of Phoenix project moment and and say run through a demo and everything works and everybody in the room is like, okay big deal and they're like, you know, have you ever done that where like you do a demo and like you do the demo where like everything sort of works and they're like, okay. Well so like shouldn't have happened and then the one one person that you women or whatever says, oh wait wait, let's try this. Let's purposely break it.

Um, you know, and and like and in the book the story is again, that's the expertise. I didn't come up with this story. But somebody I forget who did the expertise is that they've been through this example and you'll just see that in the book over and over of like these are not fixing.

These are stories that the names have been changed to protect the innocent. But soon as they did the thing where they purposely broke it like the sort of the the code signature everybody room like oh I get it. Anyway, you know, like this book is not gonna put my kids to college or anything like that.

I did this was a labor of love it. Really I think you're gonna enjoy this book. You know, my name is John Willis spotskaloop on Twitter yet.

We'll have the slides to show the new organization Andrew Kevin working together reach out. And you know, I love to have conversations about this. I think this is an important conversation.

That we need to have sort of a Last Mile problem. If you will, you know, we thought we solve the dev the Ops we saw the devops and and I think a lot of people today think app sack if you if you don't solve the app SEC problem you solve the security problem. You've not secured solved the problem until you can get your it risk and your Dev secops on the same page because it risk or internal audit protects the brand that's the connection to business.

It's a last it's another Last Mile problem. Anyway, thank you so much everybody. I appreciate your time.