CrowdStrike Incident: What Happened and How Can We Learn from It?

Calvin Nobles, PhD, is dean of the School of Cybersecurity and Information Technology and portfolio vice president at UMGC. Nobles is a retired U.S. Navy officer and has more than 20 years of cybersecurity, military, and academic leadership experience, including serving as senior director of Cyber Support Operations for the U.S. Fleet Cyber Command, senior director and chief cryptologist and security officer for Expeditionary Strike Group 7 and senior director of Global Security Operations for Naval Information Operations Command.

Manish Patel is a collegiate professor with more than 20 years of experience teaching at UMGC. He has extensive experience in the cyber and IT industry and has a master’s degree from Mercer University and a bachelor’s degree from the Georgia Institute of Technology. He also has several industry certifications, including A+, Network+, Microsoft 365 Enterprise Administrator Expert, Microsoft Azure Solutions Architect, and MCSE.

MP: It is important to emphasize that this incident was not caused by a cyberattack but rather a routine update to a configuration file. To prevent such incidents, it’s crucial to have robust testing and quality assurance processes in software development in place. In this particular scenario, CrowdStrike deviated from the industry’s established best practices concerning software updates, specifically the tier-update process. The tier-update process, alternatively referred to as phased or staged rollout, is a strategic approach to software deployment. Its primary objective is to mitigate the risks associated with deploying new versions by initially releasing updates to a limited user base, prior to a broader distribution. This methodical approach to updates is designed to ensure stability and minimize potential disruptions.

To regain customer trust and prevent such incidents in the future, CrowdStrike must provide a comprehensive transparent explanation of the incident that includes root cause analysis, improve their testing and deployment processes, ensure robust disaster recovery mechanisms, and maintain transparent communication with their customers. Additionally, CrowdStrike must establish strong partnerships with industry peers and sharing information about the incident response best practices can lead to a more resilient cybersecurity ecosystem.

CN: I think the steps CrowdStrike is taking now, including the canary process, are key. The canary process means instead of releasing software to all of your customers at once, you release them to a smaller number of constituents and see if those constituents have any problems with the software update. If that group doesn’t have any problems, then you release it to another larger group until eventually you have it rolled out to all of your customers. Taking a phased approach like that is significantly safer and the right thing to do.

But, there’s something else they could have done, too. Releasing software updates to operational systems is a major flaw in the process. We should look to implement software updates in a quarantined environment. We call it “the sandbox.” It allows you to go in and play with things that aren’t going to be used in the real world. It’s not normal, but sometimes companies don’t use the sandbox because there’s a push to market in terms of providing better products and better services as well as a push to market to help organizations defend against oncoming threats and vulnerabilities within their systems.

CrowdStrike has a capability called rapid release content, and that rapid release content is a way they update the Falcon platform to get it updated rather quickly. That process did not undergo traditional quality assurance processes. Now they have to go back and implement quality assurance practices with the rapid release content, so this never happens again.

How can cyber and IT educators and professors and professionals learn from the CrowdStrike incident?

CN: On the academic side, we can learn from this by using it as a case study, teaching students the right steps to take and processes to follow to prevent the rollout of a major update without sufficient quality assurance processes. On the industry side, we can all learn from CrowdStrike and see what CrowdStrike comes up with, how they approach it. You can be more prepared for incidents like this by looking at your procedures, your processes. What would we do if this happened to us? If we had a major outage, how would we respond? What is our emergency incident response to something of this magnitude? So we can all learn from it, and we can all show some empathy and sympathy to CrowdStrike because CrowdStrike is a really good company.

Typically, when an incident happens like this, if it’s not the magnitude of the global IT outage we’ve just seen, the information is not made public. CrowdStrike has been making their information public to everybody. CrowdStrike quickly published their preliminary investigative report, and they talk about what caused it and the things they are going to do better. They also are going to do a root cause analysis and make that public as well. This is the opportunity for all of us to pay attention to what happened.

MP: The incident underscores the critical need for educators to emphasize real-world scenarios in cybersecurity curriculums. By dissecting this high-profile breach, faculty can illuminate the complex interplay of technology, human error, and organizational culture. Incorporating lessons on redundancy, system interdependencies, incident response planning, and continuous learning will help equip students with the practical skills necessary to mitigate risks and respond effectively to future threats. Additionally, fostering a culture of critical thinking and risk assessment will empower students to become more proactive cybersecurity professionals capable of preventing and addressing vulnerabilities.

Is there any concern that releasing all of this information could open organizations to cyber attacks?

CN: Absolutely. They’re listening, they’re learning, they’re reading just like we are, but a lot of these individuals are already up to speed on how all these things work in the first place.

Are these topics covered in UMGC programs and courses?

CN: We do have a class in our cybersecurity management and policy master’s curriculum called Human Factors in Cybersecurity. It’s the perfect course—but not the only course—to talk about this and really get into the details. In the course, we study human factors and analysis classification system. We don’t just look at the incident’s immediate cause. We look at it organizationally and systematically because there’s a ton of research that tells us that one incident should not lead to something catastrophic. So, what other failures were there within the organization that contributed to this becoming something as chaotic as it was?

This is a course that attracts a lot of people from all areas—not just cybersecurity students, but people in psychology, business, marketing, and biotech—who don’t just want to accept that we have weaknesses or vulnerabilities but want to know how we can design around those so that we have a defensible mechanism in place.

You can look at the executives and see if they provided adequate resources, adequate policies. Have they implemented the proper oversight? Have they set the right culture? When you look at it from that perspective, you quickly learn that something catastrophic happening at the bottom or an incident happening at the bottom doesn’t happen in isolation. Human error is the start of the investigation, not the end. At UMGC, we are preparing students to go out and be part of the solutions to these issues, to be part of the teams that are helping develop solutions. In this type of community, everybody’s on the same side, even though we’re working for different companies or organizations. At the end of the day, our job is to protect the organizations and people we work for.