Secrets Management Simplified: Insights from Doppler's Brian Vallelunga

Imagine losing your most important digital keys and leaving your entire kingdom vulnerable to attacks. In this episode, we promise to equip you with the knowledge to prevent such disasters. Join us as we sit down with Brian Vallelunga, the CEO and founder of Doppler, to unravel the critical importance of secrets management in software development. Brian shares his deep expertise on what secrets are—those crucial digital keys that unlock access to sensitive data—and illustrates through a personal story the severe consequences of failing to protect them. Discover how data breaches can wreak havoc, leading to brand reputation damage, customer churn, legal battles, and even personal distress.

But it’s not all doom and gloom. Brian introduces us to Doppler, a game-changing tool that simplifies the tedious process of secrets management, making it an integral part of the modern development workflow. Learn how Doppler empowers developers to secure sensitive data efficiently, eliminating common headaches like managing environment files and manual secret updates. We also delve into practical implementation timelines, showing that effective secrets management is achievable for companies of all sizes with the right tools. Brian provides actionable advice for engineering teams on securing secrets within applications and highlights valuable resources for further learning. Tune in to safeguard your company’s digital assets and fortify your secrets management strategy.

Follow Brian on:

• doppler.com
• X (Twitter) - @vallelungabrian

What's New In Data is a data thought leadership series hosted by John Kutay who leads data and products at Striim. What's New In Data hosts industry practitioners to discuss latest trends, common patterns for real world data patterns, and analytics success stories.

Transcript

Speaker 1: 0:16

Hello everybody, Thank you for tuning in to today's episode of what's New in Data. Super excited about our guest today we have Brian Vallalunga, CEO and founder of Doppler. He previously was an engineering lead at Uber. Brian, how are you doing today? I'm doing great.

Speaker 2: 0:30

Thanks so much for having me here, very excited to be chatting about Seekers today.

Speaker 1: 0:35

Yeah, absolutely. It's a great topic. It's top of mind for everyone who's bringing great applications to market but really need to think about the security and making sure that they're bulletproof in terms of not allowing inadvertent access to sensitive data and controls. But, brian, first tell the listeners a bit about yourself.

Speaker 2: 0:55

Yeah, sure, so I'm the CEO and founder of Doppler. A little background about what Doppler is is we help developers and engineering teams manage their secrets, and I'll get into a little bit of what secrets are. We help companies as small as like one or two person teams and startups all the way up to multiple five companies that are in the Fortune 500. So we kind of range across the industry and helping them manage their secrets securely, but also in a way that it fits into the developer workflow. So we kind. So the kind of inspiration for Doppler is to make vegetables taste like candy, vegetables being security, candy being the developer productivity.

Speaker 1: 1:32

Oh, I love that, I love that. So what are secrets and why do they matter?

Speaker 2: 1:36

Yeah, so I'm on the show today as a PSA a public service announcement because, just in general, people, listeners on this show, engineers, and not all use a set of services every single day. So, if you think about it, me and you, in our lives, we use about 20 to 100 different companies Uber for travel, probably like Venmo for sharing payments across friends, like YouTube for videos and all of these companies have data on us and we trust them to store that data securely, this private data, and one of the biggest ways they protect that data is by protecting the keys to those systems. So, if you're like Airbnb, for example, airbnb has a database that has a bunch of homes on it. They have a bunch of transaction data. They have a bunch of data on every time someone's gone to an Airbnb and stayed at that place, and all that data is stored in a database somewhere or in a transaction system somewhere like Stripe.

Speaker 2: 2:33

And to unlock access to those systems, you have digital keys. Just like how humans have passwords to like Facebook and Netflix, so do companies, but they unlock the data of all of their customers, and that's what secrets are. Secrets are the keys that unlock access to your digital, to a company's digital kingdom, and the reason why I'm on this call today is because data breaches keep happening, and they keep happening more and more. They've impacted me personally and I think that there's a pretty big opportunity for us as an industry to prevent these data breaches from happening with a couple simple things to do, and that's why I'm on this call today.

Speaker 1: 3:11

Brian, thank you. I think it couldn't be more timely. It always seems like security and secrets are something that you know, software engineers and data engineers are familiar with, but they don't know. You know just how impactful they are. Maybe you can also describe that to us. You know what is the cost of a breach.

Speaker 2: 3:29

Yeah, really, really important question. So, for anyone listening, a breach is basically when data that should be private gets out in the public, and we've seen tons of these happen. Toyota had them, twitch had them. It's becoming even. Microsoft recently had a pretty big data breach as well, and there's lots of different ways that a data breach can impact both the company and the people. From a company perspective, it's like a massive brand reputation hit.

Speaker 2: 3:56

Imagine if you were using Equifax. I'm pretty sure you moved off of it To customer churn as a business. If you lose that brand trust, you're going to lose a large set of customers from it. You're going to have massive unplanned spend, from a huge spike in engineering efforts to stop the data breach from happening and recovering to legal and PR and insurance. And then you may also get sued for your customer, especially if they consider it negligent. If you have these keys to the locks that unlock all the private data and you are not protecting these keys, these secrets, I'd consider that negligence and I'd bet most of your customers would too, and so you may get sued for it, which then could also incur some regulatory scrutiny as well. But outside of all that, because that's all dealing with the companies. I think what's actually even more important is the people Like this is something I see a lot in the industry of. They're like oh, we have a terabyte of data or a gigabyte of data or some ton of data out there and they always say data is like this amorphous, ambiguous thing. But again, this data is my data, it's your data, it's everyone who's listening on the call data and that data comes from using these company services and so when this data gets out, it actually hurts real people, right?

Speaker 2: 5:09

I have a true story that happened to me, actually pretty recently, and I can kind of like illustrate this point. I live in Austin, texas, and I was taking my mom out to some barbecue and while we were there, I got this call from someone claiming to be at the Texas Customs and Borders and they were saying there's this package that's in my name that has a bunch of illegal drugs and money in it, and they're investigating me and I'm like, all of a sudden, I'm like, oh, wow, this is when life falls apart, moment right. Like you're being investigated for something like this that I clearly didn't do or I felt like I didn't do, and to validate who they said that they were they started sharing all this information they had on me, where I had been, places I had lived, people I know, a ton of data on me and I was like, okay, this is a government, who else would have that kind of data? And it wasn't until about 30 minutes later, when we had the lawyers on the phone, that we realized it was a scam. And not only that, but they were able to validate themselves through data that had been in previous data breaches, like the Equifax data breach and the Twitch data breach, and not only, and to compound it even further, they got new data out of me.

Speaker 2: 6:16

Because I was terrified in that moment and I was like I'm talking to the government, I have to be truthful and honest. And they'd say, oh, this is a recorded line, honest. And they'd say, oh, this is a recorded line, you're being investigated, and they will use that data to attack me even further. And the point that I'm trying to get across here is that when a data breach happens, it's not like just the company faces this brand reputation hit or some legal problems. It is real people's lives that now all of a sudden get impacted. Fraud could happen, they could get enough information to go in and steal all your money in your bank account and one day you wake up and you're at zero right Like these things happen and they've happened to me and they're going to happen to so many other people because data breaches keep happening and the rate of new data breaches every year is climbing dramatically, which means the cost of a data breach is going down, so it's getting easier and easier for attackers to hurt more and more people.

Speaker 1: 7:05

I love that, brian. I mean, that's like I don't love that that's happening, but I love the way that you articulated just the amount of sheer responsibility we have as an industry to really secure the data. And yeah, there's all types of scams going on and it really seems like the lifeblood of these attacks is really the data, the sensitive private data that these scammers have on us as a result of poor security practices Exactly. And why are these breaches happening?

Speaker 2: 7:36

It's coming from a lack of, I think, knowledge around secrets management. If you kind of think about it, every software application kind of has three parts to it Code, compute and secrets. And we have great tools today for code, for both keeping it secure and collaboration, great tools for compute, like AWS, and we have very little for secrets outside of Doppler and a couple other players in the space. And because of that, there's a lack of knowledge that these secrets need to be protected. I can't tell you how many calls I've had with CISOs, chief security officers or heads of security where I asked them what is their strategy for managing secrets and the first response back to me in a snap response was we don't have secrets. And I'm like I guarantee you you have secrets. If you have a database, you have secrets. If you have a payment processor, you have secrets. If you're sending emails or text messages, you have secrets. If you have a payment processor, you have secrets. If you're sending emails or text messages, you have secrets Guaranteed. And the fact that they didn't even know that they had it was a huge problem and that's part of why this PSA, why I'm doing this PSA. So a pretty simple question. I have three questions. It's like a sniff test and it can help you. Do you know where all your secrets are? Because if you don't know where all your secrets are, you can't possibly protect them. The secrets that you have, do you have access, controls around them and audit logs? Basically, do you have permissioning? Can you say this person has access and this person doesn't? And do you know when those secrets were read or not? Because you need to be able to build up that audit trail. If a data breach happens, you need to be able to say the data breach happened here and we're going to now go fix here. And then the last question is when a data breach does happen, when you are being hacked and your worst case scenario just started happening, which for most companies I would consider is a when, not an if, especially with the security posture they have today can you stop the attack, can you stop the data breach? And it's not as simple as just like swapping out the lock Because, remember, your production infrastructure is up.

Speaker 2: 9:33

I'll tell you a real story of a company that came to us after they experienced this really nasty pain point. They had a malicious actor in the company and they discovered this malicious actor, and this malicious actor, stole all their secrets and put on the black market. And I promise you this is a company where most people on this call have heard of this company, probably have used this company. This is a very well-known company. I can't share it for exactly the same reason why, but this is a true story. And it took their security team of three full-time security engineers six months to swap out all the locks to rotate all the secrets. So that's six months where those attackers still had the advantage to keep stealing data. It took six months of doing nothing else no other security improvements got made during that time but swapping out the lock, rotating those secrets, and so that's how painful it can be.

Speaker 2: 10:31

And then, if you have a service like Doppler, which is a first-class developer-centric secrets manager, we've now brought that ability to rotate all the locks on the doors or on the data systems within minutes, so they can click a button and boom, everything gets rotated.

Speaker 2: 10:46

And the power of that is you don't need to go figure out which individual secret got lost or got hacked, you can just click the all button and know that you're safe now. And that's the real power that a secrets manager can provide is it can answer the questions of where are all my secrets, who has access to them and can stop a data breach? And so that's the root problem that we have today is that there's a missing set of tools that can very easily be purchased or made. I don't care if you use Doppler or not, I just care that the problem stops. I need to stop having my data and everyone else's data in data breaches. If you don't end up liking Doppler, give us some feedback on why, but go with the competitor. I'm okay with that, but we got to stop the data breaches.

Speaker 1: 11:25

That's the big thing we got to stop the data breaches. That's the big thing. Yeah, absolutely. It's critical to our entire society that every company that has consumer data or data that should be private and secured about the general public is taking all these measures.

Speaker 1: 11:44

And you had this one point in the beginning where it was like making vegetables tastes like candy. And that's such a good point because you know, when software engineers or data engineers, analysts, when they're set out to do a task or build something, they're thinking about you know making it a a cool feature or an application, and then you do that code review with them and you're like, hey, look, passwords and plain text. Like, hey, there's you know other secrets that you're exposing. Like no engineer wants to like think about the. There's other secrets that you're exposing. No engineer wants to think about the security of the stuff that they're building. They just want to make a great feature that meets the business requirement. So, just to make this practical, you're saying that your solution does simplify it so that you can still ship new products quickly while making sure that the secrets are secured and there's good auditing around them, and unified or in a way where security engineers and the security teams in general have good control over this.

Speaker 2: 12:41

Yeah, that's exactly what I'm saying. So actually, before I jump into that, you brought up a really good point of like why is this happening? And it's not the developer's fault, because that's not at all what's happening here. It's that a lot of times, when you're going from zero to one, your job is to get to one. Your job is to build a product that people can start using, and it's only until you reach a certain level of success that you start to think, wow, I have to start protecting my success. And that's where, like usually, a secret manager should get implemented. Obviously, I could argue for having it day one, but I just don't think that's practical. But let's go into what it actually looks to have a secret manager and the benefits it can drive.

Speaker 2: 13:21

As you mentioned, the goal is to make vegetables taste like candy, and we really do see this, genuinely speaking, that every engineer saves about one to two hours a week using Doppler, and it's pretty simple how they save it. Before, in a pre-Doppler world, they would have an env file that's on their computer and this is a file that has a set of secrets in it, and whenever those secrets change because someone adds a set of code that needs new secrets, they have to then send that over Slack or email. And so imagine the days before Google Docs or Notion, when everyone had a Word document. Everyone's changing that document in real time and now we have to all figure out who has the right document. Doppler solves that entire problem. If secrets change, we will propagate it to your computer and do real time sync, just like Dropbox does. So you don't have to worry about your code all of a sudden not being able to execute because you're missing some secrets and you have no clue who added that requirement in those set of secrets. So we solve that whole problem on the productivity side and then on the security side, we directly integrate with your infrastructure.

Speaker 2: 14:22

So whenever secrets change, we will directly update your Kubernetes, cluster your AWS, gcp environments and restart your infrastructure with the right rollout scheme, meaning you will have no downtime when you're doing these new rollout. And all of this is centralized in a single place in Doppler. So you can have a project like the payments project, a backend project or whatever the website, and you can have environments like development, staging, production, and you can have a different set of secrets for each environment, and each environment will be connected to your infrastructure or to your developers' laptops, and that's the nuts and bolts of it is. Now you know where all your secrets are. You can set up permissions where developers get access to development and the DevOps team gets access to production, and you can automatically roll out these secrets into developers' laptops and your infrastructure when secrets change. And you can set up pull request flows, just like how you pull requests in code, to not just immediately push code in production but have a review cycle. You'll now be able to do the same thing with secrets too, and we usually see most companies integrate within a couple hours Small companies like Series A seed companies usually sub one day.

Speaker 2: 15:25

Medium-sized companies like a Series B company a couple of weeks. And we have some very, very large enterprises, companies with tens of thousands of engineers to hundreds of thousands of engineers integrating within two to four months with extremely complicated engineering staffs. So it is possible. I think that the coolest part about this problem is that it can be solved with just a little bit of time and capital and resources. It's not an extremely hard problem to solve. You just need the right tooling.

Speaker 1: 15:56

And that's a great point. Companies can take all the security measures in the world. Theoretically, the way it's perceived now is it slows down innovation because you have to go through security reviews and adding all these additional safeguards, your software, and this can take time and it can add maybe a year of scope to the deliverable. But what you're saying is that you simplify it in a way where it's practical and teams are able to onboard super quickly, which is obviously super valuable in the sense that it's saving time and helping companies do the work of critically curing their software and their customers' data without really slowing down their rate of delivery to the market, which is great yeah.

Speaker 2: 16:47

And it's not even not just slow. It's not that we're not slowing them down, we're actually speeding them up. That's the really. That's the vegetable tastes like candy part. Uh, we usually see once, two hours per week.

Speaker 2: 17:00

Well, if you start mapping that out, doppler automatically becomes a profitable ROI pretty quickly. Because an engineer, even one hour of an engineer's time, is far more than we're going to ever charge in a month pretty quickly because an engineer, even one hour of an engineer's time, is far more than we're going to ever charge in a month, right? So if you get two hours back in a week, times that by four, four weeks in a month, you're really profitable here. But also they're going to just ship faster because there's less times that developers get out of flow state. There's less times where they have to say why is my code broken? I have to go figure it out and it's broken not because I changed something, but because someone else changed something and I'm missing some dependency.

Speaker 2: 17:29

Now that I have to go figure out how to get that secret, we solve that whole problem for them. And there's tons and tons of businesses built around just developer productivity, and this is productivity plus security. So it's a two-for-one in my mind, and so, yeah, I really do think that this is a very addressable problem that people can solve. And again, you don't have to use Doppler Doppler is the only one that I know of that focuses on the developer productivity angle too and make sure that it does taste vegetables like candy. But if you just want your vegetables without it tasting like candy, there's tons of other competitors you can use too, and as long as you solve the problem of no more data breaches, I'll be a happy camper.

Speaker 1: 18:03

Absolutely. It's certainly something that we need to simplify as an industry if we really want to make big leaps in terms of securing customer data data warehouses, and you know you're trying to iterate fast and launch these new reports that the business needs. The finance team wants to do rev rec on this data. You know your sales teams want to know what the customers are doing, and all this requires like access to secure what should be secure customer data right, and you know this is another area where I think secrets have a big opportunity to really streamline that process of secure access.

Speaker 2: 19:02

It's any time that you need to get a key to a door and that door has behind that door is a digital system.

Speaker 1: 19:18

There should be a secrets manager there to make sure that there is the right permissioning and auditing throughout that entire story. Absolutely, absolutely so what are some of the ways that you know you help kind of centralize the secrets management?

Speaker 2: 19:27

Yeah, so the first thing we do is we'll pull all your secrets in to Doppler so we'll connect with your production infrastructure you have today.

Speaker 2: 19:36

We'll import all your ENV files that you have on developers' laptops and we'll give you one central view, so kind of like GitHub, where you can see a bunch of repos and in those repos you can see a bunch of branches Same exact thing things happening in Doppler.

Speaker 2: 19:52

We'll give you a bunch of projects and in those projects you have a bunch of environments and then you can start connecting those environments to your infrastructure.

Speaker 2: 19:56

So you could say the production environment is going to write to Kubernetes, the CIC environment is going to write to GitHub for GitHub workflows and the development environment is going to write to developers' laptops and then you can go in and you can see all the secrets and you can compare them across environments. You can set up access controls where developers get access to just development and production and CICD is accessed by the DevOps team. And you can set up policies that say, whenever a developer needs to change secrets in an environment they don't have access to like getting a secret into production they can set up a pull request where they can put it up and then someone on the DevOps team can pull that into the production environment and that's the nuts and bolts of what we offer, plus a very strong auditing story where we can push that data into Splunk, sumo, logic, datadog and a bunch of others to then run analysis on that data and potentially find vulnerabilities or threats.

Speaker 1: 20:48

Yeah and that's great. So then when the secret is actually represented in the code in plain text, it's not somewhere to get access to that code for some reason and install the whatever you want to call it secret identifier. You can't actually use that to authenticate with that secure database or whatever. That system is right.

Speaker 2: 21:13

Kind of. So when the application runs, it will eventually need that secret, the raw secret that is the key to that data system, and so we do most of the time for what customers want inject that secret in. The big thing is that the secret's not written in code, so it's not in disk, it's in memory. And so when the application boots up, the whole application is now running in memory and we will inject those secrets in as environment variables or as an encrypted file, depending on what the customer wants, into their running application, and then the application can use it into their running application and then the application can use it. So unless you have spyware that is inspecting applications and running applications, it can be very hard to steal that secret. And then the most important part is it's not running on. The secrets aren't stored on disk, so again, they're injected in at runtime or an encrypted ephemeral file that get cleaned up right after the process ends.

Speaker 1: 22:09

That's great. What's your advice to engineering teams that are either doing very little with secrets or have something in place but it's not really governed and centralized? How would you get started in terms of really hardening that infrastructure?

Speaker 2: 22:26

I would ask. The question so I'd figure out first is where are all your secrets today? So I'd probably run some exercise there and say, okay, so they're on developer laptops, are they on Slack? Are they in email? Are they in my code base? Are they in CICD, in production? Figure out all the places where they are. Once you do that, I would select a secret manager.

Speaker 2: 22:45

You can look at Doppler, dopplercom, d-o-p-p-l-e-rcom. There's other competitors we have. If you want to go, look at them as well and figure out what are the needs you want. Do you want just security? Do you want just vegetables, or do you want developer productivity too? Go to Dopplercom, sign up and just start importing everything, now that you know where all the secrets are, and then, most importantly, clean where they should be right.

Speaker 2: 23:10

So like, get them out of code, get them out of Slack, get them out of email, get them out of ENV files, for example, and have them all in Doppler and then connect that with your infrastructure and download the Doppler CLI and the VS Code extension for local development, and once you've done that, you're pretty much done. Again, it does not take that long at all to solve this problem. We're not talking about an AI problem, where you're training machine learning models for months on end and spending hundreds of thousands of dollars solving this problem. It's not an expensive problem to solve. It's just a tooling problem and there's very affordable, very developer-friendly tools available to you to use Doppler or not.

Speaker 1: 23:51

Yeah, that's great that Doppler makes it super simple and it's not super expensive to really adopt it from just a total cost of ownership perspective of actually deploying it. The other side of this is the cost of a breach could be massive, right? Yes, huge.

Speaker 2: 24:12

Going back to what we talked about before, I mean I'll stay away from the side of impacting people because I think we've talked about that in nauseam, but from a pure company perspective, I mean, if you're spending I don't know 20 bucks per user, 18 bucks per user on Doppler and you have I don't know 50 engineers, that's going to be significantly cheaper than a data breach where now you're going to hire a PR firm to help you rebuild your reputation.

Speaker 2: 24:38

You're going to hire a legal firm to either represent you in a lawsuit from a wide ranging set of customer maybe in a class action lawsuit, if you're big enough or just the legal troubles of cleaning all this mess up. You're going to have potentially a lot of customer churn as well. So you're going to have loss of revenue. Well, if you're a startup, startups equal growth, which means revenue can't go down. The number one rule revenue cannot go down and you will have that, so you'll have to face that problem too. So there are a lot of things that can happen when a data breach happens that are very, very hard to recover from. Like if you're looking at two companies one set of data breach and lost loss of respect of the industry and the other hasn't, you're going to choose the one that hasn't had the data breach, and that's not a mark that you can remove from company history.

Speaker 1: 25:25

Absolutely, absolutely, brian. Where can people continue to follow along with your expertise on this topic?

Speaker 2: 25:33

Dopplercom D-O-P-P-L-E-Rcom. We have a blog there where we write all the time about this kind of topics and you can also track the progress of the product. And then you can follow me on Twitter. I don't post that much. I actually love talking to people, but just not on social media. It's not really my thing, but the few times you see me post, you can follow me on Twitter at Vallalunga, brian is the username, so those are the places you can find me.

Speaker 1: 25:59

Excellent. Brian Vallalunga, ceo and founder of Doppler, thanks so much for joining today's episode of what's New in Data and thank you to all the listeners for tuning in. Thank you.