Global IT outage: What it is and how to fix it
July 19, 2024 •
Internet users around the world awoke on Friday to a global IT outage, impacting Windows devices everywhere from home offices and workplaces, to airports, banks, hotels, and more.
What caused this outage and what can you do to get your systems back up and running?
What is causing the Global IT outage?
The outage appears to have been caused by a cybersecurity software operated by CrowdStrike, known as the Falcon Sensor. Falcon Sensor is part of CrowdStrike Endpoint Detection and Response platform. A flaw in a recent update is causing Windows computers to crash and display a blue error screen, commonly referred to as the ‘blue screen of death’.
The error message reads:
“It looks like Windows didn’t recover properly. If you’d like to restart and try again, choose “restart by PC” below. Otherwise, choose “See advanced repair options” for troubleshooting tools and advanced options. If you don’t know which option is right for you, contact someone you trust to help with this.”
However, even when attempting to restart, the computers only crash again, sending users into an infinite ‘boot loop’.
The company has said this is not the result of a cyber attack or breach, but rather a misconfiguration in the latest update to the product.
How is the global IT outage impacting people and businesses?
The IT outage is grounding flights around the world as dozens of international airlines try to recover from the error. Flights out of Hong Kong, Edinburgh, Toronto, Sydney, Berlin, and more are all reporting delays or outright cancellations by major airlines.
A major broadcaster in the UK, Sky News, is down owing to the outage, as well as their National Health Service’s patient records and appointments system. Banks, such as Kiwibank in New Zealand, are also reporting that some services are unavailable as IT teams work to restore internal systems affected by the outage.
Municipal departments are also reporting that 911 call centers in some locations are down, as the outage affects phone lines and cell service companies.
How do you fix the Windows error?
CrowdStrike has released a statement regarding the outage confirming that the update has been reversed, but despite this, impacted computers are still seeing issues persist.
The Director of CrowdStrike Overwatch released additional information on X, outlining steps to get individual devices out of the boot loop.
1. Boot Windows into Safe Mode or WRE.
2. Go to C:\Windows\System32\drivers\CrowdStrike
3. Locate and delete file matching “C-00000291*.sys”
4. Boot normally.
Many companies are reporting that each individual impacted computer must be fixed one at a time, which is causing prolonged effects of this outage even after the original cause of the outage has been reversed.
Update: Microsoft has released a new recovery tool to help organizations impacted by the boot loop.
What’s next?
Some are calling this the largest outage in the internet’s history, and many are raising questions about how such an outage was able to occur.
One major question is the issue of IT ownership.
As companies move more and more services into the cloud, the responsibility over management of critical IT infrastructure is further removed from individual IT managers.
This offloading of IT management responsibilities leads to lapses in testing and an over-reliance on automatic updates. It’s unclear at this time if automatic updates are a requirement of the Falcon Sensor product, or if automatic updates can be enabled or disabled by individual IT managers.
This outage serves as a reminder that while modern IT solutions move towards a hands-off approach, taking an intentional and thoughtful management approach is still necessary to ensure the security and reliability of IT systems.