Sleeping at Night | Talking Software Supply Chain Security with Bren Briggs

In this episode, Bren Briggs of Hypergiant joins host Kim Weins and Josh Bressers to discuss software supply chain issues that keep them up at night. They touch on SBOMs as an inventory tool, DevSecOps by definition and the practice of software supply chain management.

Transcript

Announcer:

You're listening to Upstream: The Software Supply Chain Security Podcast. Brought to you by Anchore.

Kim Weins:

Hi there, and welcome. I'm Kim Weins, Host of Upstream, a podcast for those curious about the security of the software supply chain. In each episode, we talk with experts, practitioners, and [inaudible 00:00:26] leaders about concrete ideas and approaches to improve software supply chain security. This podcast is for everyone, both inside and outside the world of security. On this show, we highlight ways we can work together to protect the software that we all depend on.

Kim Weins:

It's the first week of February 2022, and the focus on software supply chain security is growing. From the White House summit on software security in the wake of the Log4j attacks, to major investments from Microsoft and Google together. In the open source security foundations, new Alpha-Omega Project. To new research from the Linux Foundation, showing growing plans to use SBOMs. Software supply chain issues seem to be keeping many of us up at night. Josh Bressers, VP of Security at Anchore, is that statement true for you?

Josh Bressers:

It's easy to say keeps us up at night, which is true in some ways, because many of us are becoming more and more aware of this, but I am extremely hopeful right now because I am seeing so much forward movement in this space at a rate that quite frankly, we just don't see very often in the industry. So while you could say, it's keeping me up at night a little bit, I'm also extremely optimist, which is very exciting.

Kim Weins:

I like to see Josh optimistic. Well, it's most certainly true for our guest on today's show, Bren Briggs, VP of DevSecOps at Hypergiant.

Josh Bressers:

Thank you so much for being our first guest on Upstream, Bren.

Bren Briggs:

It's great to be here. Thanks for having me.

Kim Weins:

So Bren, you're a VP of DevSecOps. What does DevSecOps even mean at your organization?

Bren Briggs:

I like to joke that DevSecOps is about 90% branding by weight. But it's doing the things that... I feel like DevOps and security always should have done, which is to integrate security and to bake it in from the very beginning. And at Hypergiant, we consider everything that is not a machine learning model and not customer facing application code to be infrastructure. So this means IT, this means cloud services, this means behind the scenes tooling, it also means data engineering for us. And so we actually separate the data engineering and data science disciplines. One is infrastructure, and one is more product and business logic. So we roll all of this up into the DevSecOps vertical, and we operate together as a unit and we tightly integrate with each other. And so that's how we organize within the company, and we try to automate and bake security in from the very beginning. We coordinate roadmaps with each other. And so that's in a nutshell what the DevSecOps practice at Hypergiant means.

Kim Weins:

So are you pretty tightly glued with your development and engineering teams?

Bren Briggs:

Yes, absolutely. So I actually embed with the development and engineering teams frequently and probably not advisable for managers to do, but given the size of our company, I can and do this. I can actually embed with these teams. I can commit code. I can do these reviews and I can see what's going on, on the ground and participate at times as an individual contributor. And I'm writing these workflows that we're using in helping teams threat models. So very tightly integrated with all of our development teams.

Kim Weins:

So Josh, is this different than the average company?

Josh Bressers:

I think every organization is unique in their own special way. From my perspective, this is one of those instances I think we're in, and Bren, I would love to get your opinion on this, but we like to talk about DevSecOps as this singular thing, but the reality is, it just depends. You have to do what works at your organization, because if you try to go against the grain of your culture or your development models, failure is the only outcome. So I feel like what works for you is what is best.

Bren Briggs:

I absolutely agree there. It's a shame that DevOps and DevSecOps have been reduced to job titles. And these job titles are synonymous with a cloud CIS admin, when DevOps is really, you never advertise a position for an agile developer. Likewise, you should probably never advertise a position for a DevOps engineer because DevOps is a process, it involves a lot of people. It shouldn't be incumbent on one person with four years of industry experience to carry the banner for DevOps for their whole company. Right? So I regret that it is a job title now, and I guess words or how you use them, so I have to concede that point. But really we need to be thinking about it as a process, like you said, and there's no one singular right way to do it.

Bren Briggs:

There are principles that make it work, and there are some things that you need to have to make it ontologically true that you're doing DevOps or DevSecOps. But you're right, it's not one single blessed path to get to DevOps and there is a maturity process. I don't know anybody that is perfectly mature and has hit all of their goals and targets when it comes to that process. So we have to acknowledge that it is a journey and it's a process.

Kim Weins:

I like the term that's coming out now of developer experience. That's an interesting one, and we're seeing it even especially, I would say at larger organizations where they're trying to centralize a team that does tool chain and other aspects that will help make developers more efficient and more effective. What do you think about developer experience?

Bren Briggs:

I love that idea. I love that term. I think within the DevOps ecosystem sphere, you can pick out several subspecialties. So there's the operations expert, there's the SRE, and they all have a significant overlap in this person, this developer experience person. There's elements of developer relations happening, where you're going out and you're engaging with your developers and you're showing them how to use the tools and you're finding out their problems. But then there's also... Is DevOps even old enough to have a traditional term? But the more straight ahead DevOps where you're writing tools and you're automating things, and you're thinking about things in terms of systems. So I love that idea.

Bren Briggs:

And that's actually something that we're trying to figure how to define and measure as a metric here for us, where, what is developer experience and how do we measure it and how do we improve it? And I think it's probably the number of angry Slack messages I get is a really good indicator of, or how many people are complaining about their [inaudible 00:06:34] is going down or something. That's a pretty good indication of developer experience right there. And when those go away, you have to figure out more nuanced ways to measure it.

Kim Weins:

Are you getting invited to the developer parties or being left out and shunned?

Bren Briggs:

No, that's a really good question. I need to see if I'm still on the happy hour, zoom invitation next week.

Josh Bressers:

Nice.

Kim Weins:

All right. So let's pivot a little bit now that we know about your world of DevSecOps, how are you thinking about software supply chain security?

Bren Briggs:

Constantly? You mentioned it, it keeps people up at night. It definitely keeps me up at night where any given application is going to have anywhere from a handful to dozens of direct dependencies and every single one of those direct dependencies has dozens of transitive dependencies and on for four or five more layers. So there's a lot of software that we're pulling in and trying to get your head around it and think, well, how do I audit every single one of these packages? And I know what I have in my inventory. That's terrifying, especially when you consider that repo's change owners and you have malicious file uploads or changes happening where you see entire organizations like Colonial Pipeline, ransomware attacks started through supply chain. And so I think it's on everybody's mind. At least I hope it's on everybody's mind. And I'm definitely looking at ways to improve our visibility of our packages and software inventory.

Kim Weins:

You wrote an article for venture beat and you described that finding vulnerable dependencies is a daunting task. So why is it so daunting?

Bren Briggs:

Well, like Josh mentioned in a previous interview that I'd listened to, not all languages behave the same, not all of them even have... Well, I guess they all have similar concept of dependencies, but not all of them do. They don't all map one to one. Every language is unique and it has its own particular ways of linking packages and they have their own tool chains. And some of these tool chains aren't as it exhaustive as others when it comes to declaring and finding dependencies. And so let's say that you use five different languages in your company's software, and then you import others from, open source tooling that you've downloaded.

Bren Briggs:

That's minimum five different ways, unless you use Java, then you have three or four more on top of that just to go audit dependencies, just to go figure out what you have. And then once you have it, now you have to go check versions. You have to see if it's patchable, you have to go check against your organization's particular like exposure and threat model. Like, okay, this is a medium, but the way we use it makes it a critical for us. So you're basically having to do the supply chain thing for every single unique tool chain or language that you have. And that alone makes it a challenging task just in terms of ours.

Josh Bressers:

I want to build on that because you mentioned something not too long ago about talking about a developer experience. And I think when you talk about this daunting task and how broad it can be with all these languages, you also end up in a situation where I think as humans, our first instinct is to try to pull back and say, you can't use it. You have to... Here's a list of things I'm going to approve. But I think that is an unrealistic expectation given the current universe and that whole, what we're calling developer experience right now, where developers have a certain way they want to work and they have a certain velocity they have with our existing tooling and open source. And so I'm curious of your thoughts about the first instinct of trying to throw a rope around some of this and pull it in versus saying, okay, I have to exist in this universe, as it is today.

Bren Briggs:

Developers bring stuff to us all the time where they want to use a new tool or a library or something like that, and doing application reviews for these things. So GitHub actions are probably one of my favorite case studies for this. So there are good actions and there are bad actions. And then there are things that should never be touched, and it can be hard to tell unless you know how to go and audit and look for these things. It's more than just, can I find a vulnerability in the repo? It is how many people are using it, who is maintaining it? How active is this repository? And you have to go do this check for everything... In a perfect world. You could do that check for every single thing. So instead of that, one of the things that we do is one is we're looking at the obvious metrics.

Bren Briggs:

So is this a 2000 star repo? Is this a two star repo? Okay, well, let's prefer the 2001 most likely because it seems like it's probably going to be better maintained. And the other is if a developer wants to use a tool, they likely already know how that tool works. They can show me the ins and outs. I don't have to spend all this time going and learning the thing and whatever. So I think the first thing that security teams should always be doing is instead of, like you said, being the gatekeepers and know all the time, let's collaborate and let's say, yes, and how can we make this work? So when a developer brings you something, let's explain it. Let's talk about how it's going to be used, learn from them, what they know. And then beyond that start enabling safe paths to things.

Bren Briggs:

So maybe you don't need five different ways to build Docker images. Maybe you just need the one. So we find one that works generally for everybody, or generally enough, we make this a template. We push this out and now they can self-service and start applying GitHub Actions. And you can repeat this process in various similar ways, across a number of different tool chains. It doesn't have to be actions, but I think that collaborating with your dev teams first, having this nature of approachability, where they don't think that you're just automatically going to SWAT them down, they know that if they need something done, it will get done. Whether it's with the thing that they ask for, or some other process. I think building that trust and giving them the experience that builds that trust is important.

Kim Weins:

I think it's interesting that the example you gave was GitHub Actions because that's tooling, essentially developer tooling.

Bren Briggs:

It is.

Kim Weins:

As opposed to the actual components that are going inside the software. And that's becoming just as important as the stuff going in the software.

Bren Briggs:

Well, the tooling is written. It is software and it's not in your SBOM. It's not in your SBOM, but it's doing things. And it has... Tooling, like actions always has incredibly highly privileged access. I volunteer with a cybersecurity competition called Collegiate Cybersecurity Defense Competition, CCDC. I volunteer for the Southeast region. And one of the things that we do is not only do we attack applications, but we attack infrastructure for these students. And we set up environments and we make sure that they're not just thinking about the business app, but they're thinking about all things that support it. And we've set up build servers before and you can swim like a fish through build servers, because they're never secured, ever, ever Jenkins is like go on Showdown and go find a whole bunch of Jenkins servers. You can have thousands of shells by the end of the night. So it's incredibly important for developer experience to improve tooling, but also do it in a very safe way. The three are inextricably connected.

Josh Bressers:

I think this is a really important point, Bren is you're talking about infrastructure, but I think when we talk about supply chain, sometimes we put a huge focus just on the dependencies, but your build infrastructure is absolutely a part of your supply chain.

Bren Briggs:

Yeah. If you want to think in what software gets delivered way, if you're going to inject vulnerable code, or if you're going to swap out a bad dependency as a malicious actor, what better place to do it than at build time. It's a lot more difficult see, but then you can also think about, well really any component that exists in your infrastructure should be part of supply chain concerns. So we're not talking about just build systems. We're talking about the IDE that developers are using and all the packages that they're using in VS code and things like that. All of these things are part of your supply chain and part of... And not just what you're delivering to customers, but what you're using internally, you are a customer and a consumer of that package and you need to be concerned about it.

Kim Weins:

So in the last several years, we've really made a huge shift towards cloud native, typically built with containers and microservices with containers, is this dependency problem worse or better or just different?

Bren Briggs:

Well, you've multiplied, you're a tax surface. So every container effectively has its own Linux going on. They share a kernel, but everything else beyond that is isolated and duplicated. So let's say that you're running three different images that are all based on Ubuntu. They could be based on wildly different versions of Ubuntu. As base images, they could have different layers and packages and different versions of packages. So for every single application that you run, within a container, you're multiplying the supply chain problem. And so it doesn't just make it worse. It makes it fantastically worse. It's so much more difficult. And to boot, a lot of times these containers are handled like black boxes, even though they can be inspected. A lot of people just aren't and that's one thing I think that there's a two pronged approach here.

Bren Briggs:

One is to use container scanning tools. They're abundant. They're everywhere. They're not hard to use. Go pick one up. It's fairly easy to do. The other is you should be minimizing what is in the container. The only thing that should be in your container is what you absolutely need to run your application. And depending on the language that you use, a lot of times, that can be nothing. You can compile a Golang application and use a scratch container. Distro list is also a great choice for a lot of this stuff. And you run nothing in your container. So you're not bringing in a bunch of extra dependencies.

Josh Bressers:

Can you explain what a scratch container and what a distro list container is? Because I suspect not everyone has heard of those.

Bren Briggs:

Yeah, absolutely. So when you're using Docker, you from a base image or a base layer, and this is usually literally just a completely blank layer, and then let's say you want to develop something and use an Ubuntu image. Well, Ubuntu has created this container that looks like an Ubuntu Linux, but with a lot of things stripped out, it typically has about 85 to 95 tools that are installed by default. So you bring this container in, you build your app, it behaves just like Ubuntu Linux and everybody's happy. Well, you don't need any of the rest of that build chain once you're done creating your application. So Docker has this wonderful. I say, Docker, the technology, not Docker, the company or Docker, the very specific tool here it's overloading of the name is really makes things really challenging. Let's imagine that you've built this application and normally you would just build it like you're building inside of a Linux box and you would just run. Well, if you don't need any of that other stuff, there's a couple things you can do.

Bren Briggs:

One is you could just go back in and rip everything out. And that creates extra steps and layers that you have to manage or Docker has this cool thing called multi-stage builds. So basically what you do is you have one Docker image that builds an artifact or a package or whatever for you. And you have a second stage that starts over instead of from some particular base image, as your first directive in this multi-stage build, it would be from scratch. Scratch is literally a blank container.

Bren Briggs:

There's nothing in it. Then you copy over this artifact that you need. And now you can run this artifact with no other packages, no other files. Distro list is very, very similar, except that it contains just the bare basic files that most applications on Linux are going to be looking for like Etsy shadow and password for user and group management and the time zone data and SSL cert chains and things like that. So if your application happens to use TLS or happens to be a limited privileged user instead of root user, which you should be doing, then distro list can handle those things for you. And so those are both things that I recommend for reducing the attack surface and making supply chain inside of your containers, that you develop much easier.

Kim Weins:

You're listening to Upstream: The Software Supply Chain Security podcast brought to you by Anchore. We're speaking with Bren Briggs VP of DevSecOps at Hypergiant. So you mentioned a little earlier Bren the word SBOM and there's been a lot of talk about SBOMs and there's some interesting new data that just came out this week from the Linux Foundation saying that 47% of organizations they surveyed are using SBOMs today and 78% will use SBOMS in 2022. That's a big growth. So what are you doing with SBOMs today and how do you think SBOMs will be used going forward?

Bren Briggs:

First of all, that's fantastic news. I did not realize that the adoption rate was that high. I'm very impressed. This is really, really good. The only thing that can make me happier is hearing that we've got a hundred percent MFA adoption than all companies. No, this is really, really good news. And so how are we using SBOMs? If you look at the SANS top 20 critical controls. Controls number one and two in this order are inventory of hardware and software. It's the most basic control you have to start with inventory and SBOMs are really just a form of inventory. So I consider them as a fundamental control that you should be developing and maintaining with your applications, or be able to generate on the fly with ease. There shouldn't have to be a person going in and doing LS in a directory and looking at all the repos and then going into a spreadsheet and then adding dependencies and packages and stuff that we shouldn't have to do that we can write tooling and we have tooling.

Bren Briggs:

So I think that the first way to describe it is we use it as inventory. We need to know which packages and dependencies we're running. We need to be able to put these somewhere that's searchable so that we can say, Hey, were we affected by Log4j? Absolutely not. We can show you why. And so inventory really, it's a fundamental control. Now, if you are a government or defense contractor, if you under FAR or DFARS regulations, then you're going to have to provide SBOMs anyway, just because you're working with the government. And that's also... We have to provide that first software that we develop and deliver to the government as well. So yes, we use SBOMs and we are working on ways to improve how we use them, but I consider them first and foremost, an inventory tool for us.

Kim Weins:

Josh, a lot of times will say, "It's one thing to create an SBOM." And of course, that's the first step. If you don't have an SBOM, you don't have that inventory, but now what do you do with it?

Bren Briggs:

That's a fantastic question as well. So this is, I think, where we're starting to run up against the limits or the edges of what we have developed as an industry, both as far as tools and processes go, where we have the tooling to generate SBOMs and it's improving the developer and user experience, there is getting better. But what we don't seem to have are comprehensive ways of quickly searching all of your inventory that you have generated. And so I think that's next. I think whoever really nails that problem, they're going to be a very successful project. And so ideally what you do with this is you have some sort of a searchable index as the MVP, and then the next step would be, it's not only searchable, but it's automated in proactively letting you know about vulnerabilities or problems in your infrastructure or in your packages.

Bren Briggs:

So that when you build an application with, well, for instance, if you're linking a vulnerable version of log4j. You shouldn't be able to do that even. So there's several steps in the process where this should be caught. One would be when in your manifest file, for whatever language you're using your go.mod or Pega language, it would spot it right away and say, no, you can't do this. And then if it somehow happens to make it through the build process, and you have an artifact that's generated, there should be an SBOM sign and tied to it. And we should be able to automatically say, no, this is a vulnerable version. So either stop the build or flag the security team. And then a lot of times software ages like milk. And so you have already built all of these artifacts that are sitting and waiting for people to download. And a new zero day comes along. You need to have this intelligence happening very quickly. And so making SBOMs searchable, I think is the next step.

Josh Bressers:

I agree completely. I think the challenge we're facing and we have all the SBOMs, we've been doing all this work and now we have to start doing something with them because an SBOM is not the end state. And I think that's one of the things that I won't say, people are confused by, but I feel like there's all this talk of SBOMs, but I'm not hearing all the talk about what do we do at this. And SBOM is the beginning. Now we have to go figure out where do we go from here? And I like what you're talking about, where, just searching it and then vulnerabilities and having gates on all this. And I think we're getting closer and I think we have potential, but then there's also now here's, I'd like to your opinion on the other piece. So let's say I'm generating SBOMs. I have to give you an SBOM if you are my customer. And so that's going to be one of the other challenges, I think.

Bren Briggs:

Yeah. And there are formats right now for generating these SBOMs and there's some standardization that's happening. I think there's like two main competing formats right now. And as we start to ingest these and make them searchable this imaginary tool that I've got in my head that maybe I'll start building this weekend. I don't know, but that's like this thing that I've got picturing in my head. So you have your internal stuff. And then just like your governance and compliance team would have a list, a searchable list of all of the applications that they're using inside of an organization or policies or controls. You're going to add SBOMs to that. And it's going to be... I think that you're going to have both the GRC team and the security operations team have a heavily vested interest in the contents of those SBOMs and managing those and keeping your information fresh and up to date.

Kim Weins:

So here's my hypothesis is that as we start to focus on this we'll evolve this process practice of software supply chain management, just like we've had it for traditional supply chains for many years. And that will include many of the things that you've discussed. So obviously we have to generate the SBOMs. We have to store them in a repository. We have to be able to search against them. We have to be able to provide policy based controls against those. So we can create gates in our development process or in our deployment process. And we probably have to do reporting against those. We probably have to manage those over time because we might have different SBOMs that become obsolete. As we have new versions, we need to monitor them after we deploy software, whether we deliver to a customer or deploy in production. So I think those are the capabilities that you've laid out that are starting to evolve that I think added together will create that software supply chain management practice.

Bren Briggs:

Yes. And we've already seen the role of a software supply chain specialist emerge, and we're going to continue see it evolve I think. Especially in very large organizations like Google, can you imagine the number of packages that they're linking to and importing and producing for other people, you absolutely have to have a specialist at an organization of that scale to be able to cut through the jungle of dependencies and inventory that they've got going on. At small organizations, I think that it's going to be a function of one person's job or something like that but you can see a class of security engineer emerge that thinks really deeply about this. And then that's where the tooling is also going to come from is the people that have spent a long time thinking about this problem at larger scales.

Josh Bressers:

I want to focus on the tooling as well as a concept, because when you're talking about a lot of these applications, we're not talking about tens of dependencies, we're going to be talking about thousands of dependencies and no human can deal with that scale. This is where, we truly need the robots to help us out on this one.

Bren Briggs:

Yeah. We already have the robots doing this at build time. It doesn't seem like much of a stretch to have of the same robots do this on the SBOM as well.

Josh Bressers:

Absolutely.

Kim Weins:

Well Bren Briggs. Thank you for a great conversation. And for joining us on the upstream podcast and thanks to everyone listening, you've come to the right place for discussions of the big issues impacting the security of the software supply chain subscribe and never miss an episode, Upstream is available everywhere you get your podcast. And if you learn something new, be sure to share this episode with your friends and colleagues. Thanks for joining us on Upstream, brought to you by Anchore.