Red Hat's Rodrigo Freire and the Impact of High-Profile Security Incidents

In this episode, CRob talks to Rodrigo Freire, Red Hat's chief architect. They discuss high-profile incidents and vulnerability management in the open source community. Rodrigo has a distinguished track record of success and experience in several industries, especially high-performance and mission-critical environments in financial services. 

• 01:08 - Rodrigo shares his entry into open source
• 02:42 - Diving into the specifics of a high-profile incident
• 06:22 - How security researchers coordinate a response to a high-profile incident
• 10:33 - The benefits of a vulnerability disclosure program
• 11:57 - Rodgiro answers CRob's rapid-fire questions
• 13:43 - Advice for anyone getting into the industry
• 14:26 - Rodrigo's call to action for listeners
• 15:53 - The importance of the security community working together

Episode links:

• Rodrigo Freire on LinkedIn
• Rodrigo's blog on Red Hat's response to the XZ incident discovery
• Get involved with the OpenSSF community

Transcript

Rodgrigo Freire soundbite (00:01)

Who do I ask and to grab by the arm? Man, I need you to, right now, please assess this vulnerability! It’s very important asset to have that Rolodex of contacts and to know the ones to ask for help. You don’t have to the information — have to know who knows.  

CRob (00:18)

Hello everybody. Welcome to What's in the SOSS? The OpenSSF's podcast where I and Omkhar get to talk to some amazing people in the open source community. Today, I've got a really amazing treat for you. Very special guest. My friend Rodrigo from Red Hat. I've known Rodrigo for awhile, and we're here to talk to about a really important topic, kind of, both of us have worked a lot with.

Rodrigo Freire (00:44)

Thanks Chris. Hello. Yes, I had the pleasure to work with CRob for a good number of years and I was in charge of the vulnerability management team at Red Hat. Yes, it was five definitely fun and character-molding years.

CRob (01:01)

So maybe you could share with our audience a little bit about your open source origin story. How did you get into this amazing space?

Rodrigo Freire (01:08)

It's funny. When you say that I worked with a Linux version 1 dot something, well that pretty much disclosed the age, right? It was back in the 90s. I was working on an internet service provider and there was that multi-port serial adapters for moldings, and that was pretty much the backbone of the ISP. And then sendmail, ISC buying DNS server. And back in the day there was not radios for authentication — it was Cisco stack hacks, so yeah. (Laughter)

I started on the classic ISP admin back in the 90s. That's when I got involved and then worked in the Brazilian government promoting the open source, and it was interesting time. When the government was shifting from mainframes and going to the to the low platform. And then the Linux as a security thing, and then the Linux was more focused on performance and the security. So this is where I started to wetting my toes in open source software.

CRob (02:22)

So let's dive into the meat of our conversation today, my friend. We've all seen them, and maybe you could share with the audience from your perspective — what is a high profile incident? You know, sometimes it's called celebrity vulnerability or a branded flaw. Could you maybe share like what is that?

Rodrigo Freire (10:33)

Yeah, definitely. I don't know how does that translate to English actually. So I live all the way down here in Brazil, but I like to perceive them as like creating commotion. So that's going to attract media audience and Twitter clicks and engagement and, h my God, look what I found! And in the end, that might be somewhat another Brazilian saying for you guys: trimming the pig.

A lot of cries for very few actual hair. 

So you create all that commotion, all that need and so that comes escalating from CEOs, whatsoever, security teams for something that in the end might be some moderate impact or sometimes even something that does not affect some customer systems. So it's a lot of brouhaha, I would say. However, on the other hand, there are some security events that are definitely something that you should pay close attention to. 

So for example, we had the Heartbleed and then there was Shellshock and ghosts. I've been over the course of the years, a number of GLIBC vulnerabilities that can elevate you to root, even to the extent that it was even used as a tool to get a root on the system that someone forgot the password. Yes, that happened once, to a customer that shall be renamed unnamed. 

And then finally, I think the mother of all incidents that I worked with, it would be the XZ security incident that happened a couple of months ago. More often than not this is something that just created distress with the security people with the good people managing the data center without something that's really putting the customer at risk. However, on the other hand sometimes, some less often there, will be definitely something that's really of concern and because the customer should pay close attention for that.

CRob (04:52)

So what do you think the motivation is that last year there were like 25,000 vulnerabilities? What's your perception of why some of these get a celebrity treatment and others don't that may be more severe?

Rodrigo Freire (05:08)

I have read somewhere on the internet, over the internet, something more like in the lines that over promoting something for personal gain. That resonated very well with me. On the community industry, there's a lot of effort that's put for you to render your portfolio, your reputation across the industry. And so, someone shows on the resume, hey, I was a guy who found the Heartbleed or the Ghost vulnerability. 

A lot of people are going to recognize you, oh my God, you found that vulnerability, So yeah, it might be something like that. Sometimes it might not be that intent, but in the end, Chris, I really don't think that's not something that's changed the tide on the security landscape for a good impact, I would say.

CRob (06:00)

Yeah, I would agree. Thinking about you managing some of these high-profile incidents, for our audience, maybe you could shed some light on what goes on behind the scenes when a security researcher comes to an open source project or a vendor like Red Hat. How do you get all the stakeholders together? How do you run these types of things? How do you keep the team focused?

Rodrigo Freire (06:22)

Internally at Red Hat we have some internal prioritization of the CV based on the scale. We use a four point scale. We are not attached to the CVSS score or the ranking. We focus on the product rank for the security issue. Say for example, I use HTTP server, for example, Apache HTTP my system. Alright, so there's a vulnerability affecting CVSS score in 10, a perfect 10 on CVSS for that. 

However, this functionality it's not exposed on my system or is not use it is not enabled is not supported. Why would I score that as  a 10 since it's not a valid usage on my product? So yes, I would just lump something as not a factor or even a factor, but the impact is low. Putting the customers at a heightened risk, we take that, so this is a Red Hat score as a product. I strongly believe that the the way we rank these vulnerabilities on our product is how the customers should actually be paying attention instead of taking the worst -case scenario in whatever possible use of the component. 

I'm not saying that this is not important. It is, it is key. However, we do have people, we have a human operator that's taking into account how that vulnerability is actually exposed on the product. So I think that's something very important for the vendors to do. So they take a general vulnerability and then you issue a score for your product. How is that actually exposed on our product? So that said, this is how we select how and went to fix something. 

And then, let's say for example, in the case of a high-profile event, oh man, there was a very ugly vulnerability that showed at the eve of 2022 to 2023. It was December the 21, something like that. It was on the 20s. So we had the company at a freeze and I was working. So the...sorry, this still has to be taken, right? And then there was a KSMDB, it was a kernel SMB server vulnerability. Actually, it was a stream of them that was disclosed by Zero Day Initiative.

That was an uphill battle because in the end it was not affecting RAN because we don't enable KSMDB on our kernels. So it was not affecting us. However, I needed to get all the techies, all the specialists to assure and ensure because customers questions were starting to pile up. It's not only RAN had that runs 24-7, our customers as well were surprised. So we have to provide the answers. And then finding the right resources. This is one of the key abilities for everyone managing any security program. So it's this vast network of contacts and who to ask and who to grab by the arms. Man, I need you to right now to please assess this vulnerability. 

It's a very important asset to have that disclosing the age again that Rolodex of context and to know the ones who ask for help to get information. You don't have to know the information, you have to know who knows.

CRob (09:55)

Right, and I think it's really important that some people in the supply chain, like a commercial Linux vendor, are able to contextualize that. Vulnerability may be abstract or not applicable, and I love that, that a lot of folks do that within the supply chain. Thinking about a vulnerability disclosure program, what we colloquially refer to as VDP, it's important for large projects, and it's required for a large commercial enterprise.

Could you maybe talk to some of our listeners about what the benefits to their downstreams would be to put the pieces in place to get some type of vulnerability disclosure program together?

Rodrigo Freire (10:33)

So Red Hat has a VDP in progress, so we credit for every finder that comes to us disclosing a vulnerability, we're going to acknowledge, we're going to point towards the person who finds this CVE. This is an integral part of our workflow for giving credit to the finder. Of course, we ask the finder, would you like to be credited? How would you like to see that credit get credited?

And also, that's not only for the CVEs, however, for findings on our infrastructures as well. So for example, on the customer portal or on some catalog or webpage or whatever else they find something at Red Hat, we give credit to every finder. We don't do bug bounties. However, we have this VDP, so someone is working their way to have a portfolio as a finder, as a pen tester, as a CVE finder. That's 100% fine. We will give credit.

And then, this is getting adjusted, we will negotiate with the finder how much of time would you want to have that under embargo? So we have all this negotiation with the finder to make something that can accommodate everyone's need.

CRob (11:48)

So it's some good points. Well, let's move on to the rapid-fire part of the interview. (Sound effect: Rapid fire!) Yeah!

Rodrigo Freire (11:56)

Here we go!

CRob (11:57)

 First question. Here we go! Are you ready? Spicy or mild food?

Rodrigo Freire (12:03)

Definitely spicy, man. I've been to India in November on the end of last year, man. It was the time of my life eating any spicy food to the point of sweating in my head, man! That was a trip!

CRob (12:20)

Nice! (Sound effect: Oh, that’s spicy!) What's your favorite whiskey?

Rodrigo Freire (12:26)

It's Talisker. And I tell you what, if you're having a Talisker and then you drink Blue Label, I'm sorry, Blue Label, that's going to fade. Blue Label is just going to fade away. Talisker for the win. 

CRob (12:42)

Very nice. Next question, Vi or Emacs?

Rodrigo Freire (12:46)

Vi, come on man!

CRob (12:48)

(Laughter) Nice! Rodrigo, what's your favorite type of hat?

Rodrigo Freire (12:55)

Type of hat? Man, I actually found that, well, my favorite one is actually a Red Hat, right? But after I got a decision to become a bald person, I actually liked being bald and I seldom use any kind of hat, right? So I'm a proud bald, I'd say. On the other hand, it would be just a baseball cap.

CRob (13:17)

OK, fair enough. And last question, tabs or spaces?

Rodrigo Freire (13:22)

Tabs! show some finesse! first of all it's plain ice so

CRob (13:26)

Nice, excellent, excellent. Well, now. (Sound effect: That’s saucy!) As we wind up, do you have any advice for someone that's looking to get into the field, whether it's cybersecurity incident response or open source development? What advice do you have for these newcomers?

Rodrigo Freire (13:43)

First of all, play nice. Show respect and make your due diligence. I think everyone is going to embrace you wholeheartedly because no one likes vulnerability. So if you're going to find new stuff or even help to fix this stuff, show the attitude. So be positive, make your relationship network. That's important because without it you're not going to succeed or you're going to earn some bad reputation as well. Everyone's already fighting a hard battle, so play nice.

CRob (14:15)

Nice. That's excellent, fantastic advice. And our last question, do you have a call to action that you want to inspire our listeners to go do as soon as they listen to this?

Rodrigo Freire (14:26)

Yeah, definitely. So, take into account your environment. So, no one likes emergencies. Emergencies are expensive. No one likes emergency maintenance windows. So, get to understand your environment. So, is this CVE, is this vulnerability really affecting? So, can you be that trusted advisor on your organization so you actually can be the person who sets the expectation, the needs of the company?

There's some pressure from these highprofile events from the upper floor asking hard questions. So get to understand your real need so you can actually schedule something that will not hurt your team or your availability or even the stability of your environment. And finally, I would say ask questions. So ask your vendor or your account reps or your consultants. So yeah, if you're in doubt, go ask your questions. And I think I am positive that they are going to ensure you that you have a secure and stable environment. 

CRob (15:38)

Excellent. That's, I think, some great advice from someone that's been there on the front lines helping fight the good fight for downstream and representing its customers. Rodrigo, thank you for joining us today on What's in the SOSS? Really appreciate you coming and talking to us.

Rodrigo Freire (15:53)

Thank you, Chris. And one last word I would like to stress here. So on the security discussion, there's no Red Hat. There's no Canonical. There's not Oracle. No. We all collaborate very closely when it gives regard to security issues. We are in close touch to everyone. Everyone knows each other. So there's no, Red Hat's only playing ball alone. No such a thing. I got to tell you guys, the XZ security incident was first disclosed to Debian and then Debian got in touch with us and then we started the coordination. So, yeah.

CRob (16:32)

I love that about our community, the fact that we all come together and able to put our colored hats to the side and come together and collaborate.

Rodrigo Freire (16:37)

Exactly, mister!

CRob (16:39)

Excellent. Well, thank you, Rodrigo. Have a great day.

Rodrigo Freire (16:42)

Thanks, Chris.

Announcer (16:43)

Thank you for listening to What’s in the SOSS? An OpenSSF podcast. Be sure to subscribe to our series of conversations on Spotify, Apple, Amazon or wherever you get your podcasts. And to keep up to date on the Open Source Security Foundation community, join us online at openssf.org/getinvolved. We’ll talk to you next time on What’s in the SOSS?