SolarWinds, supply chain hacks, and cybersecurity : Planet Money How a single hack pried open the networks of giant corporations and the U.S. government itself. | Subscribe to our weekly newsletter here.
NPR logo

One Hack to Fool Them All

  • Download
  • <iframe src="https://www.npr.org/player/embed/1001402799/1001428232" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript
One Hack to Fool Them All

One Hack to Fool Them All

  • Download
  • <iframe src="https://www.npr.org/player/embed/1001402799/1001428232" width="100%" height="290" frameborder="0" scrolling="no" title="NPR embedded audio player">
  • Transcript

SYLVIE DOUGLIS, BYLINE: This is PLANET MONEY from NPR.

(SOUNDBITE OF COIN SPINNING)

JACOB GOLDSTEIN, HOST:

In December of last year, somebody at a cybersecurity company, a company called FireEye, noticed something just a tiny bit out of the ordinary. Somebody was logging in to the company's system using an employee's username and login, but they were using a different phone number than the employee had used before.

DINA TEMPLE-RASTON, HOST:

So people get new phone numbers. That's not the big deal. This particular company, though - FireEye - is in the computer security business. So they take this kind of thing really seriously.

KEVIN MANDIA: So one of our staff members called the person, you know, whose account was used and said, hey, did you register a second phone?

TEMPLE-RASTON: That's Kevin Mandia, the CEO of FireEye.

(SOUNDBITE OF ARCHIVED NPR BROADCAST)

MANDIA: And the gentleman said, no, I did not register that phone. So who did?

GOLDSTEIN: Who indeed?

TEMPLE-RASTON: Yeah. Well, so Mandia and his team at FireEye, they start trying to figure out exactly that. You know, how did some random person get into their network and end up registering a new phone? And the more they learned, the more worried Mandia got.

MANDIA: It just felt like the breach that I was always worried about. We didn't know a lot at the time. It just felt like it was time to brace for impact.

(SOUNDBITE OF ARNAUD RIGNON AND SEBASTIEN LANGOLFF'S "DARK WOOD")

GOLDSTEIN: Hello, and welcome to PLANET MONEY. I'm Jacob Goldstein.

TEMPLE-RASTON: And I'm Dina Temple-Raston. Today on the show, the story of a single hack that got inside some of the biggest corporations in the world and even deep inside the United States government itself.

GOLDSTEIN: It was a particular style of hack that seems to be becoming more common. In fact, just today, Friday, May 28, as this show is about to go out, there is news of another, similar kind of hack. And I think our vulnerability to this particular kind of hack really tells us something not just about software and cybersecurity, but about the way business works today and about how it might need to change.

(SOUNDBITE OF ARNAUD RIGNON AND SEBASTIEN LANGOLFF'S "DARK WOOD")

GOLDSTEIN: Dina, you are on the breaking news investigative team here at NPR, and you have spent months working on this story, so why don't you just pick up where we left off with Kevin Mandia, the CEO of FireEye?

TEMPLE-RASTON: Right. He's realized there's somebody who's not an employee who's inside their network, and that's a problem.

MANDIA: So we had several weeks where I'm sitting here going, boy, I wonder how they broke in. And it is a terrible nag, Dina, when you're responding to a breach anywhere, whether it's your own house or someone else's house, and you don't know how they broke in.

TEMPLE-RASTON: So FireEye is in the business of trying to figure out exactly that kind of thing. And that's what other companies typically pay them to do. And what they do is they try to think back to what the earliest evidence of compromise could be, you know, like where they might have seen some sort of stranger in their network or where that stranger could've come in. And they traced this back literally for weeks. And they think it all started with some software from a company called SolarWinds.

MANDIA: So at that point, the only logical conclusion that I drew was something's wrong with the SolarWinds server.

GOLDSTEIN: So SolarWinds - we know now that's what this big hack that this whole story is about came to be called, the SolarWinds hack. And I want to be honest with you. I've been sort of following that story, but I don't think I have ever really understood, like, what is SolarWinds? What is it?

TEMPLE-RASTON: SolarWinds is a software company. And they make a bunch of different kinds of software, but the one that's at the center of this story is a software they make to manage computer networks.

GOLDSTEIN: OK, so nothing to do with either the sun or the wind. If I'm thinking alternative energy, I'm entirely in the wrong universe.

TEMPLE-RASTON: Entirely in the wrong - I have no idea what kind of - how they came up with the name.

GOLDSTEIN: OK.

TEMPLE-RASTON: What I can tell you is that it's what's called network management software. This is what IT people use basically so they can keep sort of an eye on the entire network. So, for example, you know, if you have that printer on the fifth floor that's always breaking down, they can see that on one screen. If there's a router that goes down, they can see all that on the same screen. So think of it as actually something that touches everything in a network. And the reason it's kind of genius to actually hack into something like network management software is because it touches everything. And it means if you're inside of it, then you can touch everything, too.

GOLDSTEIN: So it's like if you're inside this, you can get inside of everything at a company, at an organization. Can you give me just, like, a list of companies and government agencies that were using SolarWinds when this happened?

TEMPLE-RASTON: So one, obviously, is FireEye.

GOLDSTEIN: That's the company we talked about at the beginning of the show that was running SolarWinds software and figured out that something was wrong.

TEMPLE-RASTON: Right. But in addition to that, I mean, some really big companies were running the software - Microsoft, Intel, Cisco. Then if you look at the federal government, the Department of Homeland Security was running it. The Treasury was running it. Even parts of the Pentagon were.

GOLDSTEIN: Wow.

TEMPLE-RASTON: So this was something that was really widespread. And again, you'd never heard of it. I'd never heard of it. But the people who knew about this were the people who were in the back room of your IT department. And for those people, SolarWinds was everywhere.

GOLDSTEIN: And we know that FireEye figures out that the SolarWinds server was hacked. And then Kevin Mandia, the CEO of FireEye, he tells SolarWinds, you know, you've got a problem here.

TEMPLE-RASTON: And then SolarWinds does this incredibly surprising thing. It goes and tells the world. In fact, their CEO, Sudhakar Ramakrishna, was so focused on getting the whole story out, he even talked to us.

(SOUNDBITE OF ARCHIVED NPR BROADCAST)

SUDHAKAR RAMAKRISHNA: You forget about competition and competitors. And in that context, you - the right thing to do is to report. The right thing to do is to give them the ability to fix those issues and protect their customers.

TEMPLE-RASTON: What he doesn't say is that everyone was probably going to find out anyway.

GOLDSTEIN: Right, right. So now they have to figure out, you know, who hacked us, and how did they hack us.

TEMPLE-RASTON: And in order to answer those questions, they need to call in an expert. And the expert they called was a guy named Adam Meyers.

ADAM MEYERS: And so the first call we took, I'm sitting outside of my in-laws' house in the driver's seat of my vehicle. I'm sitting in the driver's seat, and I'm outside while everybody's inside, having this phone call with the lawyers. And we're kind of getting our arms around what was going on.

TEMPLE-RASTON: Adam Meyers is a genius at reverse engineering. And what that means is he looks at the hack, and he looks at all the code, and he just sort of teases it out to try and figure out what each piece of code does, how it works, what its job is. And then once he figures that out, he just keeps digging deeper and deeper and deeper until he can essentially figure out the whole hack.

GOLDSTEIN: So as best as he has figured it out, what is the story of this hack?

TEMPLE-RASTON: Well, the first thing they realize is that this wasn't a regular hack, that actually it started in a place they hadn't expected. And the place where it started was in what they call their development environment. What it is is it's this - think of it as a clean room in a factory where you actually write the software, you write the patch, and then you actually seal it up before you send it to someone else, before you put it out for people to use the patch.

GOLDSTEIN: And what happened in this sort of factory clean room where they're making the software patch?

TEMPLE-RASTON: It seems that bad guys appeared to have snuck in. SolarWinds didn't have a clean environment. What they had was a development environment that was connected to a network that was connected to the internet. So that meant at the very last second - and this is what Meyers figured out. At that very last second, instead of having SolarWinds send out their own patch, the bad guys swapped it with their own. Meyers explained it with this metaphor.

(SOUNDBITE OF ARCHIVED NPR BROADCAST)

MEYERS: Let's go with Halloween candy, right? Like, when I was growing up, you used to have to check your Halloween candy 'cause somebody might have put a razor blade in your Reese's Peanut Butter Cup, right?

GOLDSTEIN: So, OK, stay with this metaphor, right? In a typical hack, the hackers open the candy wrapper and stick the razor blade in. But, you know, now the wrapper is open. This is pretty easy to detect. But in this instance, in the SolarWinds hack, they did something much more clever and much more insidious.

(SOUNDBITE OF ARCHIVED NPR BROADCAST)

MEYERS: Imagine those Reese's Peanut Butter Cups going into the package, and just before the machine comes down and seals the package, some other thing comes in and slides a razor blade into your Reese's Peanut Butter Cup, right? So that is - you know, and then the package gets sealed and it goes out the door to the store.

TEMPLE-RASTON: And that's why this hack was so effective, because when the software patch from SolarWinds goes out to all these big companies and government agencies, it looks like it's sealed software. But in fact, there's a razor blade - malware code, essentially - that's hiding inside.

GOLDSTEIN: So there's this phrase, Dina, that I've seen in some of your reporting on this that as a sort of econ nerd interested in the cybersecurity stuff I got pretty excited about. And that phrase is supply chain hack, right? This has been called a supply chain hack. And so supply chain there refers to the idea - like in the same way we might think of, say, a car company having a supply chain, right? Like whatever - Ford buys parts from literally thousands of different companies. Software works kind of the same way, right? Like, the Department of Defense and Microsoft and Cisco - these companies don't just write their own software. They have a software supply chain - right? - all these little things like this software they're getting from SolarWinds. And so if you can hack into the supply chain, you can get everywhere with one hack.

TEMPLE-RASTON: It's much more efficient.

GOLDSTEIN: Yeah, it's much more efficient. It's a really good way to hack everybody all at once.

TEMPLE-RASTON: Exactly. Instead of trying to break into the Treasury or the Pentagon and Department of Homeland Security, just find a software that's ubiquitous, and break into that. And this is why people like Adam Meyers have been so worried about supply chain hacks.

MEYERS: The reason that software supply chain keeps me up at night - you know, think about all the apps on your mobile device, on your tablet, on your computer. You're only as secure as the development environment that those were built in, and you're only as secure as the weakest link in that chain.

TEMPLE-RASTON: And for a bunch of giant companies and federal agencies, the weakest link was SolarWinds.

GOLDSTEIN: After the break, what the bad guys got, who the bad guys are and what the United States is doing to try to prevent this from happening again.

(SOUNDBITE OF CESAR GIMENO LAVIN AND SIMON JOSEPH ALEXANDER JAMES' "THE DEVIL LIVES NEAR")

GOLDSTEIN: The story so far - the bad guys got their razor blades into thousands of packages of Reese's Peanut Butter Cups. Those Reese's Peanut Butter Cups got sent to the Department of Defense and Microsoft and Cisco and everybody else. I get that it's big. I get that it's big. And, you know, there is one other, like, admission I have about this hack - is I don't really know what happened or what the implications are. What was the bad thing that happened because of this hack?

TEMPLE-RASTON: Well, so there were two big things. The first is this was clearly an espionage operation. They were taking information out of networks. We don't know what it was, and nobody has really talked specifically about that. But we do know that, for example, they were reading emails from government officials, officials at DHS, officials at the Treasury. And the reason why that's important is because there's a lot of information that can be in an email. It could be an attachment or something like that.

GOLDSTEIN: Sure, sure.

TEMPLE-RASTON: The second thing that's worth...

GOLDSTEIN: So just spying. So this is like straight-up very successful spy operation.

TEMPLE-RASTON: We think so, right?

GOLDSTEIN: Yeah.

TEMPLE-RASTON: And that seems to be what the motivation was here. But the other thing that people aren't talking about quite as much - something called a backdoor.

GOLDSTEIN: OK.

TEMPLE-RASTON: And a backdoor is malicious code that you plant in a network for use later. What a backdoor allows you to do is, say, for example, steal emails later when everybody's relaxed. Or a backdoor could allow you to plant ransomware. Think about the Colonial Pipeline. That actually wasn't a hack. That was a ransomware attack. And their system was frozen until they paid a certain amount of money to some criminals. And the same thing could happen with SolarWinds. I mean, we don't know about it because this little piece of ransomware could be hidden in code that they haven't discovered yet.

GOLDSTEIN: So we know the bad guys got in. We are pretty sure they were spying. And maybe they also planted some things that will allow them to do bad things in the future, we just don't know that part yet.

TEMPLE-RASTON: Exactly. Exactly.

GOLDSTEIN: So do we know who the bad guys are?

TEMPLE-RASTON: We think we know. Russian intelligence, a group called the SVR, is thought to be behind this. And there are a couple of reasons for that. One, this was an incredibly sophisticated hack. Not only did they get into where they were actually building the software, but Adam Meyers told us they were super careful about covering their tracks so that there wouldn't be little clues that they might be able to find to tell them who was behind it. And that's the sort of thing that you see a nation-state do. And because of that...

GOLDSTEIN: Like, it's just - like, a criminal wouldn't care that much.

TEMPLE-RASTON: They don't care.

GOLDSTEIN: Yeah, right.

TEMPLE-RASTON: A criminal just wants their money, right?

GOLDSTEIN: (Laughter) Right.

TEMPLE-RASTON: But this was artful. This was artful.

GOLDSTEIN: And so this new hack that we mentioned earlier in the show, this hack that we are just learning about today as the show is going out, it seems similar in some key ways to the SolarWinds hack. For one, it appears to have been done by the Russians. For another, it looks like it was another supply chain hack. And it is also targeting, ultimately, the U.S. government. In this case, the hackers apparently hacked an email service software that is used by lots of people, including government agencies. And then they used that hack to send malicious emails out into the world that looked like they were from this U.S. government agency.

TEMPLE-RASTON: Exactly.

GOLDSTEIN: So this is, like, kind of Cold War-ish, right? It's definitely country versus country. Like, this is Russia at some level - well, attacking is too strong a word, but Russia coming at the United States.

TEMPLE-RASTON: Yeah, this is "Spy Vs. Spy" stuff.

GOLDSTEIN: So what's the U.S. going to do about it? Is the U.S. going to hack back? Is that the way this works? Like you hacked us, we'll hack you? Did we already hack them and we don't know it?

TEMPLE-RASTON: Possibly, although I suspect that, you know, everybody's sort of watching for it. But there's an entire military command, Cyber Command, and the National Security Agency, and their job is to do exactly that. And, you know, we were talking before about backdoors. Backdoors are put in in case you need them later, right?

So a lot of people believe that after the Sony hack, once they had determined that North Korea was behind it, that the U.S. retaliated by turning off the internet in North Korea for a couple of days just to let them know, hey, we're in your systems, and you should be careful. We're watching you. Of course, the U.S. has never admitted that publicly. I mean, this is one of the reasons they call cyber the perfect weapon, because it's short of war and it's hard to attribute it. So you can do a lot of the same things you would do with what they call metal on steel, you know, kinetic things. You can do that just by using computer code.

GOLDSTEIN: So one last thing in terms of, you know, what's coming next, what are we going to do about this - you, Dina, have reported on this executive order that President Biden just issued that's going to set standards. It's going to set rules, basically, for companies that sell software to the federal government. And the idea is that forcing companies to follow these rules should make supply chain hacks like SolarWinds less likely in the future. And I know you've described two of these rules that seem especially key, especially relevant here. What are they?

TEMPLE-RASTON: Well, one is something they call provenance. And provenance basically means you have to tell us where all the code you're using comes from. And this is a big deal because it's cheaper to actually have software written in other countries because coders in a lot of other countries make a lot less money than coders in, say, Silicon Valley or coders in the United States more generally.

GOLDSTEIN: OK.

TEMPLE-RASTON: For example, some of SolarWinds' code was written in Eastern Europe. And apparently, the government didn't know that.

GOLDSTEIN: OK.

TEMPLE-RASTON: Now, nobody has connected that to the hack, but it's emblematic of a larger problem, which is that people don't know where the code in their software actually comes from.

GOLDSTEIN: And so to be clear, it's OK to have your code - some of your code come from overseas or whatever, but you just have to be able to document for all of the code where each chunk came from.

TEMPLE-RASTON: Yes, and whether or not - for example, the federal government may decide to go with a different company because they like where their code was built better, right? This...

GOLDSTEIN: Right, right.

TEMPLE-RASTON: ...Would be another consideration. Before, it was all about price, or it was largely about price - maybe reputation, but...

GOLDSTEIN: Yeah.

TEMPLE-RASTON: ...Largely about price. Now it's going to be much more about whether or not you can set up a defense in terms of knowing where your code is coming from, knowing how your code is made, knowing how you developed your software. Those are all really important things.

GOLDSTEIN: So, OK, provenance - know where your code comes from. That is one of the new standards. What's the other one that's also important?

TEMPLE-RASTON: Well, the other one really goes directly to the SolarWinds hack. Remember, we think hackers somehow got into the so-called development environment - right? - that digital place where engineers at SolarWinds actually write the code, build the software or build a patch. So this new standard will require that the development environment be essentially cut off from the internet. They call it air gapped. And so that would make it a lot more like a clean room in a factory. And to go back to the earlier Reese's Peanut Butter Cup metaphor, this should make it harder for hackers to sneak inside and slip those razor blades inside the sealed wrapper.

GOLDSTEIN: So these kinds of changes - you know, requiring the place where the coders are writing software to be separated from the internet and requiring companies to know where all of the code comes from - these will make software safer and probably more expensive, right? It's making it less efficient in the name of safety. That's, like, a trade-off. The government is saying, let's make this trade-off at this point.

TEMPLE-RASTON: Right, although hacks cost a lot of money, right?

GOLDSTEIN: Yes. No, I agree. I agree. Yes.

TEMPLE-RASTON: So it's unclear where the trade-off will be.

GOLDSTEIN: So more expensive upfront but maybe cheaper in the long run.

TEMPLE-RASTON: Right.

GOLDSTEIN: And I do feel like it's interesting to think about this story in relation to the economy more generally - right? - because...

TEMPLE-RASTON: Right.

GOLDSTEIN: ...It seems like one of the big economic lessons of the pandemic is that what seemed optimally efficient in lots of industries - you know, automaking or whatever, this idea of, like, don't hold extra inventory, lean manufacturing - it turned out to be not very resilient. Once things started getting weird in the world, suddenly there are shortages of cars, shortages of everything.

TEMPLE-RASTON: Right.

GOLDSTEIN: And so this relentless pursuit of efficiency left us, left the economy vulnerable - surprisingly vulnerable. And it feels analogous to this SolarWinds story, where software is this incredibly efficient industry, and doing things like having programmers be networked and using code from all these different sources - these are very efficient practices that let people build really powerful software really cheaply. But what we're learning now with this hack is that, as you say, like, maybe that's not really most efficient in the long run even if it superficially seems so.

TEMPLE-RASTON: Yeah, I'm not sure we learned that from this hack...

GOLDSTEIN: (Laughter) Fair.

TEMPLE-RASTON: ...'Cause I think that we've known for some time that this was a vulnerability. And there was never...

GOLDSTEIN: Yeah.

TEMPLE-RASTON: ...Really the impetus to have people say, let's not do it this way. They were chasing, you know, who could do it the most cheaply and not necessarily the most safely. And I think that what has happened as we've seen these hacks grow more and more sophisticated, I think there's a realization that the way we used to do things, we can't do them that way anymore and that we have to have defense much more in mind than we did in the past.

(SOUNDBITE OF BERNARD BERNIE RUBINSTEIN AND JOHN MARK CACAVAS JR.'S "WIGGIN' ALONG")

GOLDSTEIN: What other stories about spying should we do? Let us know. You can email us at planetmoney@npr.org. You can also find us on many of the social media. We are @planetmoney. I'll note that we just hit our one-year anniversary on TikTok. If you haven't checked out PLANET MONEY TikTok yet, you should. It's strange and smart and great.

Today's show was produced by Maria Paz Gutierrez with engineering help from Gilly Moon. Bryant Urstadt edited the show. Alex Goldmark is our supervising producer.

I'm Jacob Goldstein. This is NPR. Thanks for listening.

(SOUNDBITE OF BERNARD BERNIE RUBINSTEIN AND JOHN MARK CACAVAS JR.'S "WIGGIN' ALONG")

Copyright © 2021 NPR. All rights reserved. Visit our website terms of use and permissions pages at www.npr.org for further information.

NPR transcripts are created on a rush deadline by Verb8tm, Inc., an NPR contractor, and produced using a proprietary transcription process developed with NPR. This text may not be in its final form and may be updated or revised in the future. Accuracy and availability may vary. The authoritative record of NPR’s programming is the audio record.