Massive Windows/CrowdStrike Fail

This is the fault of...

  • Micro$oft, because they write software where you need stuff like this

    Votes: 3 13.0%
  • CrowdStrike, because their code broke the world

    Votes: 11 47.8%
  • Those who chose Windows over Linux, because that was the critical market decision

    Votes: 5 21.7%
  • Giant Death Robots, because they did it

    Votes: 4 17.4%

  • Total voters
    23
Did the outage affect only the US?

Nono, it cropped up all kinds of places. Berlin Airport closed early and caused all kinds of air-travel problems across Europe fx.

Also India, Australia, the Netherlands... everywhere.

I hope everyone has sold their crowdstrike stocks already. Depending on the legal situation and compensation for damages, I don't think they'll survive that.


My employer was affected (which means there's a good chance a big bunch of the goverment agencies and ministries were....), but I had free on Thur/Fr, didn't have my laptop turned on on Thur, so my machine is still alright :thumbsup:.
 
Regarding the Microsoft vs Linux discussion earlier, what distros are typically used in Linux servers?
Red Hat is big in the enterprise space. Even if you're running some weird Kubernetes / container-ised setup tend to fall back on the known names at the corporate level.
 
Also India, Australia, the Netherlands... everywhere.

I hope everyone has sold their crowdstrike stocks already. Depending on the legal situation and compensation for damages, I don't think they'll survive that.


My employer was affected (which means there's a good chance a big bunch of the goverment agencies and ministries were....), but I had free on Thur/Fr, didn't have my laptop turned on on Thur, so my machine is still alright :thumbsup:.

No, not everywhere. Just on those countries where the ruling elites put too much trust on american software corporations.

Crowdstrike, I recall, got big and famous thanks to political services rendered some 8 years ago. Not due to being good at anything, and in fact it lied shamelessly back then. I'm not surprised it recruited a CEO that had already overseen another catasthrophic incompetence and the now vanished McAfee. These people manage fall upwards because they have political protection. We'll see what happens to Crowdstrike's management. I bet nothing will, they know where some corpses are buried so won't be held accoutable. Even if the company crashes and burns.

Those "sanctions" are getting better and better for the targeted countries. :lol: They avoid being inflicted with crapware.
 
It is not really connected, but this shows that we can charge IT service providers if the will is there. Over £1,000 per failed call (to 999 admittedly) is a significant chunk of change. Imagine if we charged Mirco$oft that?

BT fined millions for failing to connect 999 calls

BT has been fined £17.5m for a "catastrophic failure" of its emergency call handling service which led to thousands of 999 calls not being connected.

The network fault, which lasted for more than 10 hours on 25 June last year, led to 14,000 calls to emergency services failing to connect.

Following an investigation into the company which manages the 999 phone system, the regulator Ofcom said that BT was "ill-prepared" to respond to the problem.

BT also provides text relay services for deaf and speech-impaired people but the outage meant that these users were "unable to make any calls, including to friends, family, businesses and services".

"This left deaf and speech-impaired users at increased risk of harm," said Ofcom.

The telecoms watchdog said the emergency call handling system's outage was caused by an error in a file on a server, which meant systems restarted as soon as call handlers received a call.

It led to staff being left staff logged out and calls being disconnected or being dropped as they were transferred to the emergency services.

An attempt to recover from the problem then failed, Ofcom said, due to a human error because instructions on how to solve such an issue were "poorly documented".
 
There is not much Microsoft can do about it, because if it tried to, the anti-malware industry will claim foul to the anti-trust agencies immediately.
It seems Microsoft agree

EU gave CrowdStrike the keys to the Windows kernel, claims Microsoft

Did the EU force Microsoft to let third parties like CrowdStrike run riot in the Windows kernel as a result of a 2009 undertaking? This is the implication being peddled by the Redmond-based cloud and software titan.

According to a report in the Wall Street Journal, a Microsoft spokesperson pointed to a 2009 undertaking by the IT giant with the European Commission as a reason why the Windows kernel was not as protected as that of the current Apple Mac operating system, for example.

The agreement [DOC] is about interoperability and came as Microsoft was subject to European scrutiny. The undertaking seeks a level playing field and includes the following clause:

Microsoft shall ensure on an ongoing basis and in a Timely Manner that the APIs in the Windows Client PC Operating System and the Windows Server Operating System that are called on by Microsoft Security Software Products are documented and available for use by third-party security software products that run on the Windows Client PC Operating System and/or the Windows Server Operating System.​

In other words, third-party security vendors must get the same access as Microsoft's own products. Which, on the face of it, is fair enough.

However, nothing in that undertaking would have prevented Microsoft from creating an out-of-kernel API for it and other security vendors to use. Instead, CrowdStrike and its ilk run at a low enough level in the kernel to maximize visibility for anti-malware purposes. The flip side is this can cause mayhem should something go wrong.

The Register asked Microsoft if the position reported by the Wall Street Journal was still the IT titan's stance on why a CrowdStrike update for Windows could cause the chaos it did. Redmond has yet to respond.

Windows is far from the only operating system that permits software to run at a level low enough to crash a kernel. However, failures of third-party software running at a low level in Windows can be embarrassingly public, even if Microsoft is not directly to blame [Hint: They are partially to blame].
 
Does anyone know how CrowdStrike updates itself on Linux? From what I gather, it uses a kernel module as well, so I suspect very much the same thing could have happened to Linux servers as well. It might be just by chance that this affected the Windows definitions and not the Linux definitions.
 
Does anyone know how CrowdStrike updates itself on Linux? From what I gather, it uses a kernel module as well, so I suspect very much the same thing could have happened to Linux servers as well. It might be just by chance that this affected the Windows definitions and not the Linux definitions.
I bet it would have taken down a much smaller percentage of Linux machines.

In fact it happened (sort of). I do not understand how it can be GDPR compliant to give them this much access to our data to train their AIs.

CrowdStrike's Falcon Sensor also linked to Linux kernel panics and crashes

CrowdStrike's now-infamous Falcon Sensor software, which last week led to widespread outages of Windows-powered computers, has also caused crashes of Linux machines.

Red Hat in June warned its customers of a problem it described as "Kernel panic observed after booting 5.14.0-427.13.1.el9_4.x86_64 by falcon-sensor process" that impacted some users of Red Hat Enterprise Linux 9.4 after (as the warning suggests) booting on kernel version 5.14.0-427.13.1.el9_4.x86_64.

A second issue titled "System crashed at cshook_network_ops_inet6_sockraw_release+0x171a9" advised users "for assistance with troubleshooting potential issues with the falcon_lsm_serviceable kernel module provided from the CrowdStrike Falcon Sensor/Agent security software suite."

Red Hat also advised that "disabling the CrowdStrike Falcon Sensor/Agent software suite … will mitigate the crashes and provide temporary stability to the system in question while the issue is investigated." The issue was "observed but not limited to release 6 and 7."

We've also spotted reports of CrowdStrike being suspected of causing problems in Debian and Rocky Linux.

Linux kernel panics and Windows Blue Screens of Death are broadly comparable. The occurrence of kernel panics mere weeks before CrowdStrike broke many Windows implementations therefore hints at wider issues at the security vendor.

The Register has asked CrowdStrike to comment on the issues identified by Red Hat, and will update this story if we receive substantial information.
 
Last edited:
Delta Airlines is still convulsing over this.


Delta Air Lines cancels hundreds more flights as DOT opens probe
The Atlanta-based carrier has canceled more than 4,000 flights since last Friday.

A Department of Transportation probe. Oof.
 
I bet it would have taken down a much smaller percentage of Linux machines.
Yes, but not so much because Linux is in any way more secured than Windows, but because Linux admins tend to be much more hesitant to install dodgy software on their machines.

In fact it happened (sort of). I do not understand how it can be GDPR compliant to give them this much access to our data to train their AIs.

The argument for GDPR compliance is very likely that CrowdStrike claims to not record any GDPR-relevant data. So GDPR does not apply to it. And so it becomes just one of the vendors whose software you trust to not leak data.

I suspect that corporate interpretations of GDPR might even encourage such solutions as they become another tickbox on the "we need to protect the data"-900-page-checklist.
 
Yes, but not so much because Linux is in any way more secured than Windows, but because Linux admins tend to be much more hesitant to install dodgy software on their machines.
Yeah, that is kind of my point.
The argument for GDPR compliance is very likely that CrowdStrike claims to not record any GDPR-relevant data. So GDPR does not apply to it. And so it becomes just one of the vendors whose software you trust to not leak data.
It is certainly possible. However in that case why did Cloudstrike join the UK Data Privacy Framework List to allow such transfers? If they do packet analysis in the US that would count.
 
Related to the recent discussion in this thread

That sounds great, but I am not sure it is as strong as it could be:

This new law requires all public bodies to disclose the source code of software developed by or for them unless third-party rights or security concerns prevent it.

Crowdstrike has third party rights so ...

Somewhat related, I always liked the idea of this though I do not know how that survived implementation.

Spoiler Open Forge Project :

Open Forge is Government of India's platform for open collaborative development of e-governance applications. Through this platform, the government wants to promote the use of open-source software and promote sharing and reuse of e-governance related source code.
 
Last edited:
If Linux usage was about as widespread as Windows among private citizens, corporations and authorities, hackers would be all over Linux too.

It's best 'protection' atm is that its considered a novelty and not worth investing resources into undermining.
 
If Linux usage was about as widespread as Windows among private citizens, corporations and authorities, hackers would be all over Linux too.

It's best 'protection' atm is that its considered a novelty and not worth investing resources into undermining.
It is at least 76% of web servers. I think it is enough of a target if it was as vulnerable.
 
CrowdStrike fiasco highlights growing Sino-Russian tech independence

Some of the common arguments for moving away from proprietary operating systems are about increasing personal (or corporate) freedom and decreasing expenditure, but there are bigger things at stake.

CrowdStrike's bad update took down Windows-based computer systems around the world and had wide-ranging impacts outside of IT. Microsoft software permeates so much of the connected, computer-driven world that it's easy to believe it's universal.

What's received less attention is that, due to geopolitical maneuvering, two of the world's largest countries were largely spared. As the BBC put it, China swerved the worst of the global tech meltdown. The South China Morning Post's coverage said Chinese cybersecurity firms are taking a victory lap.

Until last year, The Reg FOSS desk was based where Eastern and Western Europe meet. We took a close personal interest in Russia's invasion of Ukraine, and we've reported on how Linux adoption is spreading in Russia due to Western sanctions. One Russian Debian-derivative vendor was already planning to IPO by 2022, and other distros we never see in the West, such as ROSA Linux and the Calculate Linux family, are thriving.

As a result, there are reports that Russia was relatively unaffected and emerged unscathed.
China started getting rid of Windows years ago. Its government is instructing companies to replace non-Chinese OSes with domestic Linux distributions, such as Kylin and openKylin, based on Ubuntu. Kylin is doing well, reporting more than 800,000 users a year ago, while Debian-based sibling Deepin claims more than 3 million paying users.

As Windows users often tell us in the comments to our Linux distribution reviews, the Linux world is confusing and strange, and often the products are simply not quite as good as commercial alternatives. What the car industry calls fit and finish often are inferior ... and if that's what you're used to, the free software experience can be markedly inferior.

Even though, as we argue, you cannot in fact buy software at all. Despite this, new software keeps new hardware selling. It's constantly getting bigger and more complicated and slower, but not really objectively much better. Proprietary desktop and server OSes haven't vastly improved in 30 years.

There is a bigger picture here. FOSS frequently isn't as polished as proprietary software. The thing is, that sometimes doesn't matter. So long as an alternative does the essential parts of the job at all, that may be enough. If it's free – or at least, much cheaper – that is enough to clinch the deal.

An example in the West is ChromeOS and Google Docs. Yes, it's true, although there is a choice of rich local clients and backend servers to replace Microsoft Office and the combination of Outlook and Exchange Server – and frankly far too many desktops to launch them from – none are perfect replacements. That opened up the opportunity for Google to bypass the entire rivalry. If no alternative office suite is a perfect replacement, some companies have worked out that Google Apps in a browser is good enough to get by, and it comes effectively for free with Gmail (alongside Google Calendar and Google Contacts). If it lets your staff communicate and share what they need to get their work done, that is enough to suffice.

It may need an additional motivation, such as a ransomware attack. Long before CrowdStrike, the Conti ransomware took Nordic Choice Hotels' Windows machines out, so they used CloudReady to switch to ChromeOS Flex. If everything is web-based already, an OS that only offers a browser and nothing else will get the job done.

Don't underestimate the power of 'good enough'​

ChromeOS is of course no use in China, behind the Google-blocking Great Firewall, but that's not a problem. As well as a choice of domestic Linux distributions, both for servers and clients, China is busily working on its own processors as well. The Register was already reporting on Godson processors in 2011. More recently, although still not super fast, China's Loongson processors are getting there. GCC supports them. They are in shipping hardware in a variety of form factors, including from Lenovo. If you hadn't guessed already, China is selling them to Russia.

Trade restrictions and sanctions, including blocking Russian contributions, are actively fostering local developments. Even Western companies buying up Russian software doesn't block this, it just results in domestic forks. Once the code is out there, taking it and forking it and developing it for local use is perfectly in line with the principles of free and open source software.

China is watching developments in Ukraine closely, as is occasionally even visible in tech circles. Similarly, the West is nervously monitoring Chinese tensions with Taiwan. TSMC's chip fabs are so hugely complex that it's highly unlikely a hostile invader could take over and keep them running, but just in case, there are remote kill switches in place. If China were to invade its smaller neighbour as Russia did, the effects on world chip supplies would be absolutely devastating. The US reportedly buys 92 per cent of its leading-edge chips from Taiwan.

Someone somewhere made a serious mistake that caused the CrowdStrike outage. The update was inadequately tested, and poorly deployed, without staging. (As Reg readers know, staging or phasing updates means not pushing them out to everyone at once. Canonical does this with its LTS releases, as we noted for 22.04.1 and more recently for 24.04.) But this botched update, arguably, hasn't hurt CrowdStrike that badly. Its share price is down but remains nearly twice what it was a year ago.

The real error here is so much of the IT industry blindly trusting large corporate vendors not to mess up. The webcomic XKCD has been eloquently skewering this for years. (This comic is from August 2018, presumably rather before Boeing putting the beancounters in charge, but XKCD has been doing so for years.) The big vendors are drowning in code like everyone else, but the difference is letting the marketing department guide decision-making.

Every business and organization is free not to take the mainstream route, but most simply follow the herd. That's what is leading to the commoditization of software.

The government of China has been doing something different, and as a result it need not care about the rest of the world's computer industry. It has its own OSes, running on its own silicon. It and its big Eurasian ally might be inconvenienced – but not crippled – by the collapse of the worldwide chip industry… just as it was, apparently, not massively affected in any significant way by CrowdStrike causing many of the world's Windows computer systems to collapse.

If Russia gets away with destroying Ukraine – allegedly in order to save it, of course – then China might prove willing to destroy Taiwan in much the same way. As a side effect, it could do a more effective job of destroying the world computer industry than even CrowdStrike managed. The world might suddenly be grateful for resource-frugal FOSS if it does.
 
I just thought I'd throw in a bit of random information about this issue. This is not the first time CrowdStrike screwed up a bunch of servers. Back in april an update caused similar issues to Linux servers around the world. It just wasn't as obvious to the public because most of the public facing stuff does not run on Linux so the news didn't care.
 
I can't buy a bowl of soup for that at restaurant.
 
Top Bottom