The Security Interviews: How SolarWinds came through its darkest hour
In his first major UK press interview, SolarWinds CEO Sudhakar Ramakrishna tells Computer Weekly how a relentless focus on transparency saw the company safely through a nightmare cyber breach scenario
When SolarWinds, a supplier of IT and network management solutions hitherto little known to the layperson, fell victim to one of the most significant cyber security breaches of recent years in December 2020, CEO Sudhakar Ramakrishna found out about it just a few hours before the news hit the IT press, but for good reason – at the time, he was not yet officially the CEO of SolarWinds.
In fact, the ink was barely dry on Ramakrishna’s new contract bringing him to SolarWinds from Pulse Secure, when the so-called Sunburst or Solorigate incident broke. The cyber attack, a long-running, orchestrated intrusion by threat actors linked to the Russian state, targeted multiple organisations and, critically, government bodies, by injecting tainted code into SolarWinds’ Orion network management platform in a classic example of a supply chain attack.
It was an altogether strange situation to face, he reflects. “You can’t really do much about it in the sense that you can’t direct the team to take action,” he says. “Many of the team members you don’t even know at that point in time. As a funny aside, they were even betting within SolarWinds whether I’d show up on day one or not.”
The record shows that Ramakrishna did show up – pandemic restrictions permitting. He actually moved his timeline up by a couple of weeks to be able to inform his response better, and on top of that threw the original 90-day plan he had drawn up into the bin and drafted a new one.
“Around 20 December is when I started getting briefings on what had happened and what the team was doing about it, because I wanted to be prepared to hit the road running as it relates to customers, partners and government authorities, as well as, most importantly, figuring out what we learned from this and what we can do about it,” he says.
“When the incident happened, within a matter of about 48 hours we at least had a remediation for it”
Sudhakar Ramakrishna, SolarWinds
“I give the team a lot of credit because when the incident happened, within a matter of about 48 hours we at least had a remediation for it. Then, post that, it was a matter of engaging with customers and dealing with them, helping them through the migration to the patched code, and then that has been an ongoing process actually, and it was an ongoing process until well in to April.”
Ramakrishna is clear that there were some SolarWinds customers who were spooked by the attack, but they were in a clear minority.
“I would say the vast majority of the customers, once they understood what had happened and what we were doing about it, were very understanding, very supportive, and many of them have actually re-upped their commitment to SolarWinds,” he says.
“One ironic school of thought right now is that we are probably one of the most secure companies as it relates to delivering code to customers because of the practices we have implemented.”
Ramakrishna is referring to the company’s new Secure By Design initiative, which is already bearing fruit, and of which more later.
But it wasn’t all plain sailing, and mistakes were made in the process, famously the somewhat rash implication – swiftly refuted – that an intern was to blame for leaving the door open for the bad guys. But now, nearly a year down the line, it is certainly fair to say that SolarWinds has become something of a byword for transparency in cyber security.
For Ramakrishna, transparency is a key element of any response to any issue – not just cyber – that materially affects a customer.
“We talk about building trust with customers, earning their trust, and so on,” he says. “But the way I like to think of it is the way you earn trust is by being transparent with them, what’s working, what’s not, what are you doing about it, etcetera.”
Putting himself in the shoes of a SolarWinds customer for a second, he adds: “If I have to trust you, then I have to believe that you’re being transparent with me about the state of affairs, so that was foundational to who we are, who I am and how I operate.”
“If we keep staying transparent and keep doing the right thing, we will actually benefit in the long run”
Sudhakar Ramakrishna, SolarWinds
Ramakrishna also speaks of the importance of maintaining both a sense of humility in the face of adversity, and a sense of calm. “That was the foundational set of principles I operated with alongside the team,” he says, “and while the first four or five months were incredibly hectic and crazy, now we are more normal, if I can call it normal, in terms of running our business.
“My belief is, if we keep staying transparent and keep doing the right thing, we will actually benefit in the long run, commercially as well. When you’re doing it, it doesn’t feel that way, because everyone’s attention is on you, but when you stay true to who you are and what you believe in. I personally believe that it’ll actually help us succeed better.”
SolarWinds is realising other benefits as well, he explains. “The common joke in security is there are two types of companies – ones that have been breached and know, and ones that have been breached but do not,” he says. “My view is that as a community, especially when dealing with external threat actors, we have to be more transparent among ourselves, so that we can share, learn and remediate faster.
“I hope this gives more confidence to more victims of cyber attacks to say it’s OK to come out and speak about our learnings, it’s OK to come out and say what we learned from it, how we improved, and what the community can do to defend itself better, so that’s my single biggest hope from this. And I want to learn from others’ experiences, too.”
Secure by Design
Transparency, of course, will only get you so far, and because it was ultimately SolarWinds’ software development process that came under attack, the company has been hard at work implementing its new Secure by Design strategy to ensure its customers can move forward with confidence.
There are three fundamental pillars to the Secure by Design strategy. First, to enhance and improve the security of SolarWinds’ on-premise and cloud-based infrastructure, implement better endpoint detection, role-based access controls, and other elements that make it harder for an attacker to break into its systems, and second, to improve the security of the company’s build systems themselves.
These first two pillars are, by and large, a continuation of things SolarWinds was already doing, but it is the third pillar that Ramakrishna regards as the most significant – changing build processes and methodologies themselves.
“Most software companies have one single build pipeline,” he says. “They go through the compilation and they spit out code. What we have done is build a model where we have three separate systems, with three sets of permissions, and they can be in different locations, and those locations can change.
“For a release to be shipped, we build in three different environments, and essentially create a hash in the three different environments, and ensure it matches across them, and only then do we sign off.”
The supposed advantage of this model is that it creates a more dynamic, ever-changing environment that makes it much harder for a threat actor to inject malicious code into the supply chain. SolarWinds’ experts are currently working up whitepapers on the subject, and Ramakrishna says this model could have applicability beyond software development into other areas of the IT stack – as he notes, such attacks are likely to keep coming.
Ready for the future
Having implemented Secure by Design, and other accompanying measures such as improvements to security training for employees, Ramakrishna says SolarWinds is now much more confident in its risk mitigation and incident preparedness posture than it was a year ago.
“We are much more confident today,” he says, “but any company that says they have no risk, or have very little risk, is probably not fully aware of the types of risks that can be imparted upon them. So that’s where the humility aspect has to come into play.
“Constant vigilance and constant learning – that has to be the mindset of every security team”
Sudhakar Ramakrishna, SolarWinds
“You’re confident that you’ve done the right things, you are confident that you’re training, you’re confident you have the right technology and the right policies, but you can never be assured that you don’t have risk. Constant vigilance and constant learning – that has to be the mindset of every security team, no matter what company you report into.”
With the impact of the cyber attack fading, Ramakrishna’s attention is also turning to future plans for SolarWinds. “We have a very broad portfolio of solutions that, thankfully, customers seem to benefit from and appreciate,” he says. “Our focus going forward is making our solutions a lot more simple and a lot more powerful for customers to consume. Training and budgets are a challenge, complexity is increasing, so in that world, what do we do to serve our customers better?
“We have now defined the notion of what we call SolarWinds Observability, which is integration on a common hybrid platform, automation and configuration at one level, observation of systems and databases and applications and users, visualising them, and taking automatic remediation steps through artificial intelligence and machine learning.”
More information on these ideas and proposals will be published before the year is out, as SolarWinds begins to articulate its 2022 plans externally. With any luck, SolarWinds’ people will have an altogether quieter holiday season before getting back to work on making them a reality.
Read more on Data breach incident management and recovery
