Last Friday, the world experienced the biggest global outage of key Windows PC infrastructure in history — this issue, caused by a botched CrowdStrike update to its kernel-level Falcon Sensor software, made modern Windows systems so fundamentally non-functional that flights around the world were delayed. Southwest Airlines managed to avoid the issue, because the company was using Windows 3.1 instead of any remotely modern version of the OS.

But as it turns out, the problem isn't just isolated to modern Windows operating systems. Linux users have been reporting kernel panics and crashes related to the same software since as early as April of this year, per a report from The Register.

So, how is this issue cross-platform? Chances are the specific issue that caused chaos over the last few days is not— after all, we would've seen it cripple Windows machines much sooner if that were the case. However, what this does demonstrate is that CrowdStrike has apparently been lax with its Falcon Sensor Security software for quite a while now.

For those unfamiliar, the "kernel" of an operating system refers to the layer outside of user interaction (typically called the "shell"), and most directly connected to the hardware beneath. The thing is, very little computer software actually needs kernel access to get its work done. And while security software can certainly be an exception because threats often may attempt to infiltrate the kernel, it's still very important to ensure that the software isn't also causing kernel instability and crashes for any target platform.

An interesting sidenote pointed out by The Register is that CrowdStrike's current CEO, George Kurtz, was also CEO of McAFee during an infamous 2010 update that caused several PCs to be stuck in an endless boot loop. This likely makes George Kurtz the first CEO in history to preside over two major global PC outages caused by bad security software updates.

Linux users who have been impacted reportedly include those using Red Hat Enterprise Linux, Debian Linux (and Debian is the basis for the more-widespread Ubuntu), and Rocky Linux. All of the issues in question are impacting the underlying Linux kernel (universal across Linux distributions), though, seemingly crashing any Linux distributions using kernel versions 5.14.0-42713.1 and newer.

Linux users do seem to have more recourse for issues like this— including switching to an eBPF "User Mode"— but it speaks to the severity of CrowdStrike's kernel software development issues if the company is managing to cripple Linux and Windows operating systems. 

It also shows that there were warning signs for this past global outage, and that systems should have been in place at CrowdStrike some time ago to test these enterprise and government-targeted updates vigorously enough to prevent these kernel-level crashes. After all, most impacted users in these strictly-controlled environments likely don't have the administrative access or knowledge required to fix these problems once they occur. In other words, much-improved QA testing would seem to be mandatory for CrowdStrike's continued long-term success.

In the late hours of July 18, CrowdStrike released an update which saw Windows machines BSoD (Blue Screen of Death) across the world. Initially this was reported as a Microsoft centric issue, with Azure and Office365 being impacted, but it later transpired that CrowdStrike's update of its Falcon Sensor which detects and reacts to threats to systems, was the cause. Official confirmation of CrowdStrike being the root cause has been made, and a workaround fix has been issued. Things are slowly returning to normal, but as each time zone wakes up, more cases are being reported.

If your Windows Client or Server VM is running the CrowdStrike Falcon agent, then the BSoD bug may see your VM stuck in a restarting state. Using the Azure Portal or Azure CLI / Shell you need to reboot your VMs a number of times. Microsoft states that some users have rebooted up to 15 times to get past this issue.

George Kurtz, President and CEO of Crowdstrike has been interviewed by NBC News and issued an apology for the disruption caused by the global outage triggered by CrowdStrike's update.

"We're deeply sorry for the impact that we've caused to customers, to travellers, to anyone affected by this, including our companies" 

QUICK FIX

Need to fix the issue quickly? Here are the steps that you need to take. Note that this may not work for everyone, and you do so at your own risk. This fix comes courtesy of Brody Nisbet, CrowdStrike's Director of Threat Hunting.

1. Boot Windows into Safe Mode or WRE.

2. Go to C:\Windows\System32\drivers\CrowdStrike

3. Locate and delete file matching "C-00000291*.sys"

4. Boot normally.

Microsoft now says that the "underlying cause" of the issue has now been fixed for its apps. Users should experience a "residual impact" that should decline of the next few hours.

THE STORY

It seems that a recent CrowdStrike code update is bricking Windows machines across the world. The issue which occurred late in the night of July 18 is impacting companies of all scales. In the United Kingdom, the London Stock Exchange, television companies, flight operators and train companies are impacted. The dreaded Blue Screen of Death (BSoD) is appearing on Windows machines across the world. The cause is now linked to a recent CrowdStrike update which George Kurtz, President and CEO of Crowdstrike has now confirmed.

The BSoD issue is down to a misconfigured configuration issue but it does mean that users are forced to take hands on action to potentially remedy the issue. But for now we would wait for official guidance on how to remedy the issue, but later in this story we do cover one approach which is apparently working for some users.

According to the BBC News website, Microsoft released a statement which removes doubt over issues with its own services, the focus moving to CrowdStrike's services.

We spotted the start of this issue via the creator of haveibeenpwned, Troy Hunt's post on X, formerly Twitter.

Nothing says “We’re sorry for knocking a significant portion of the world offline and causing global mayhem” like a $10 Uber Eats voucher ????‍♂️ https://t.co/8yr6Nlhamc

— Troy Hunt (@troyhunt) July 25, 2024

Post mortem from CrowdStrike: “On July 19, 2024, two additional IPC Template Instances were deployed. Due to a bug in the Content Validator, one of the two Template Instances passed validation despite containing problematic content data.” https://t.co/u4E0tdLCRI

— Troy Hunt (@troyhunt) July 25, 2024

The world has been monitoring this issue and in the first few hours there was plenty of finger-pointing on social media, nothing official was released until 05:50 EDT, when Microsoft hinted that a "third-party" was to blame, and moments later the CrowdStrike statement was released.

he source of the issue is a content update for CrowdStrike's Falcon Sensor product, "The intelligent, lightweight CrowdStrike Falcon sensor, unlike any other, blocks attacks on your systems while capturing and recording activity as it happens to detect threats fast." according to the CrowdStrike website.

What's the impact of the CrowdStrike outage?

The impact of the issue is global and it seems that today is a bad day for Windows users. CrowdStrike has confirmed that MacOS and Linux users are unaffected but airports, banks, stock exchanges, TV networks, medical services are all impacted across the world. We've compiled a list of some key areas that have been impacted during the early hours of this story.

• Reuters are reporting that IT systems for the upcoming Olympic Games in Paris are affected, with the organizers moving to a contingency process.
• United, Delta and American Airlines have issued a "global ground stop" on all of their flights. Flights already in the air will continue, and there are no apparent safety issues.
• Australian Telstra Group, a telecommunications company is also facing disruption.
• Airports across the UK are reporting delays and flight suspensions. Barcodes used for security checks at London Gatwick are not working, with security checks conducted manually.
• India's Delhi airport has resorted to manual processing of passengers and flight times communicated via a whiteboard.
  Railway companies are reporting delays.
• Sky TV and BBC Children's channel CBBC are off the air, with Sky running old stories.

Is this a hack?

Right now, there is no evidence that this is an orchestrated attack with a malicious intent. No hacker groups have come forward to claim the hack, and at the time of writing, it is believed that there are no personal data loss or safety issues.
The issue doesn't seem linked to any cyber attacks, merely a bad update is likely to blame. A bad update which has impacted many aspects of our digital lives.

What is CrowdStrike?

CrowdStrike is an American cybersecurity company. Based in Austin, Texas, Crowdstrike provides "cloud workload protection and endpoint security." The goal of the software is to prevent hacks and outages, so it seems ironic that it could now be the cause of a global IT outage. The alleged cause of the issue is CrowdStrike's Falcon Sensor, a tool that analyzes connections to and from the wider Internet for malicious behavior.

Brody Nisbet, CrowdStrike's Director of Threat Hunting has confirmed that the issue lies with CrowdStrike, but the issues lies with a "faulty channel file" and Nisbet suggests a workaround for some of those stuck in a BSOD boot loop. The fix has to be manually applied to each affected machine. Remotely managed systems can (hopefully) do this from afar, but for others will need a System Administrator (sysadmin) or IT support team member to perform the task. Remember to say thanks to your sysadmin today!

As the global outage was unfolding, we reached out to Tom Cheesewright, Applied Futurist who has worked with NASA, Google and Meta, for comment on this global issue.

"It will be interesting to find out if the two occurrences - Azure going down and the CrowdStrike issue - are connected. If not, it's an awful coincidence and one that has really compounded the chaos for Microsoft users. This is news because it's rare and we have to remember that, in spite of today's chaos. Cloud systems have proven to be a more reliable, more efficient and largely more secure way of operating. They're big news when they fail because so many people are affected. But if you aggregated the many small failures and cost of all the hardware we used to have in data centres, and the dusty servers in the corner of basements, I'm pretty sure we'd all come to the conclusion that the occasional failure is worth it."