Todays we are seeing major disruptions to IT infrastructure worldwide, afecting Airlines, Banks, Hospitals, Emergency services, Telecom companies, Media outlets, Payments processing, among others. [1] [2]
The root cause seems to be a faulty update released by CrowdStrike, a Cybersecurity company, for Falcon Sensor, their Endpoint Protection solution, which caused computer to lock-up and not turn-on properly, showing a Blue-screen error.
In an unrelated event, Microsoft Azure Cloud services also had major issues around the time of the “CrowdStrike problem” a Central US Azure outage (Tracking Id: 1K80-N_8) – those issues with Azure seem to be already mostly resolved. [3]
Solution for IT admins
As the affected computers and not running properly, unfortunately it seems they will need to be fixed one-by-one.
The solution seems to be:
1. Boot into Safe mode or the Windows Recovery Environment
2. Run the command:
del "C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys"
3. RebootKey takeaways
- Software and Hardware Vendors have a high responsibility and should extensively test their products – both with automated tools and manually
- Vendors should do gradual releases of software updates, with live monitoring for issues and clients reported issues
- You don’t release significant updates on a Friday!
- Vendors should provide easy options or tools to control updates: delay updates for X amount of days, completely disable updates should always be available for IT admins
- For companies at large, they should have mechanisms to delay or manually approve updates for their entire IT infrastrucutre – unfortunately this is not always easy and is largely dependant of options provided (or not provided) by Vendors
Remember the 3 basic principles of Cybersecurity are Confidentiality, Integrity and Availability, in the quest to secure the systems we should pay attention to issues like this, which end-up causing as much disruptions as a major CyberAttack.
We should also be aware that Cybersecurity is, in large part, a risk-management and a balancing act between those 3 pillars.
Advice from Microsoft to solve this issue
