Global IT issue strikes Windows machines, linked to CrowdStrike software update

Quick Summary:

In the late hours of July 18, CrowdStrike released an update which saw Windows machines BSoD (Blue Screen of Death) across the world. Initially this was reported as a Microsoft centric issue, with Azure and Office365 being impacted, but it later transpired that CrowdStrike's update of its Falcon Sensor which detects and reacts to threats to systems, was the cause. Official confirmation of CrowdStrike being the root cause has been made, and a workaround fix has been issued. Things are slowly returning to normal, but as each time zone wakes up, more cases are being reported.

If your Windows Client or Server VM is running the CrowdStrike Falcon agent, then the BSoD bug may see your VM stuck in a restarting state. Using the Azure Portal or Azure CLI / Shell you need to reboot your VMs a number of times. Microsoft states that some users have rebooted up to 15 times to get past this issue.

George Kurtz, President and CEO of Crowdstrike has been interviewed by NBC News and issued an apology for the disruption caused by the global outage triggered by CrowdStrike's update.

"We're deeply sorry for the impact that we've caused to customers, to travellers, to anyone affected by this, including our companies" 

QUICK FIX

Need to fix the issue quickly? Here are the steps that you need to take. Note that this may not work for everyone, and you do so at your own risk. This fix comes courtesy of Brody Nisbet, CrowdStrike's Director of Threat Hunting.

1. Boot Windows into Safe Mode or WRE.

2. Go to C:\Windows\System32\drivers\CrowdStrike

3. Locate and delete file matching "C-00000291*.sys"

4. Boot normally.

Microsoft now says that the "underlying cause" of the issue has now been fixed for its apps. Users should experience a "residual impact" that should decline of the next few hours.

THE STORY

It seems that a recent CrowdStrike code update is bricking Windows machines across the world. The issue which occurred late in the night of July 18 is impacting companies of all scales. In the United Kingdom, the London Stock Exchange, television companies, flight operators and train companies are impacted. The dreaded Blue Screen of Death (BSoD) is appearing on Windows machines across the world. The cause is now linked to a recent CrowdStrike update which George Kurtz, President and CEO of Crowdstrike has now confirmed.

The BSoD issue is down to a misconfigured configuration issue but it does mean that users are forced to take hands on action to potentially remedy the issue. But for now we would wait for official guidance on how to remedy the issue, but later in this story we do cover one approach which is apparently working for some users.

According to the BBC News website, Microsoft released a statement which removes doubt over issues with its own services, the focus moving to CrowdStrike's services.

We spotted the start of this issue via the creator of haveibeenpwned, Troy Hunt's post on X, formerly Twitter.

Nothing says “We’re sorry for knocking a significant portion of the world offline and causing global mayhem” like a $10 Uber Eats voucher ????‍♂️ https://t.co/8yr6Nlhamc — Troy Hunt (@troyhunt) July 25, 2024

Post mortem from CrowdStrike: “On July 19, 2024, two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data.” https://t.co/u4E0tdLCRI — Troy Hunt (@troyhunt) July 25, 2024

The world has been monitoring this issue and in the first few hours there was plenty of finger-pointing on social media, nothing official was released until 05:50 EDT, when Microsoft hinted that a "third-party" was to blame, and moments later the CrowdStrike statement was released.

he source of the issue is a content update for CrowdStrike's Falcon Sensor product, "The intelligent, lightweight CrowdStrike Falcon sensor, unlike any other, blocks attacks on your systems while capturing and recording activity as it happens to detect threats fast." according to the CrowdStrike website.

What's the impact of the CrowdStrike outage?

The impact of the issue is global and it seems that today is a bad day for Windows users. CrowdStrike has confirmed that MacOS and Linux users are unaffected but airports, banks, stock exchanges, TV networks, medical services are all impacted across the world. We've compiled a list of some key areas that have been impacted during the early hours of this story.

• Reuters are reporting that IT systems for the upcoming Olympic Games in Paris are affected, with the organizers moving to a contingency process.
• United, Delta and American Airlines have issued a "global ground stop" on all of their flights. Flights already in the air will continue, and there are no apparent safety issues.
• Australian Telstra Group, a telecommunications company is also facing disruption.
• Airports across the UK are reporting delays and flight suspensions. Barcodes used for security checks at London Gatwick are not working, with security checks conducted manually.
• India's Delhi airport has resorted to manual processing of passengers and flight times communicated via a whiteboard.
  Railway companies are reporting delays.
• Sky TV and BBC Children's channel CBBC are off the air, with Sky running old stories.

Is this a hack?

Right now, there is no evidence that this is an orchestrated attack with a malicious intent. No hacker groups have come forward to claim the hack, and at the time of writing, it is believed that there are no personal data loss or safety issues.
The issue doesn't seem linked to any cyber attacks, merely a bad update is likely to blame. A bad update which has impacted many aspects of our digital lives.

What is CrowdStrike?

CrowdStrike is an American cybersecurity company. Based in Austin, Texas, Crowdstrike provides "cloud workload protection and endpoint security." The goal of the software is to prevent hacks and outages, so it seems ironic that it could now be the cause of a global IT outage. The alleged cause of the issue is CrowdStrike's Falcon Sensor, a tool that analyzes connections to and from the wider Internet for malicious behavior.

Brody Nisbet, CrowdStrike's Director of Threat Hunting has confirmed that the issue lies with CrowdStrike, but the issues lies with a "faulty channel file" and Nisbet suggests a workaround for some of those stuck in a BSOD boot loop. The fix has to be manually applied to each affected machine. Remotely managed systems can (hopefully) do this from afar, but for others will need a System Administrator (sysadmin) or IT support team member to perform the task. Remember to say thanks to your sysadmin today!

As the global outage was unfolding, we reached out to Tom Cheesewright, Applied Futurist who has worked with NASA, Google and Meta, for comment on this global issue.

"It will be interesting to find out if the two occurrences - Azure going down and the CrowdStrike issue - are connected. If not, it's an awful coincidence and one that has really compounded the chaos for Microsoft users. This is news because it's rare and we have to remember that, in spite of today's chaos. Cloud systems have proven to be a more reliable, more efficient and largely more secure way of operating. They're big news when they fail because so many people are affected. But if you aggregated the many small failures and cost of all the hardware we used to have in data centres, and the dusty servers in the corner of basements, I'm pretty sure we'd all come to the conclusion that the occasional failure is worth it."