Massive Windows/CrowdStrike Fail

[Samson]

Basically I am not at all opposed to the idea of people making money off software and do not consider the existence of software as a product to be a bad thing.

Nor am I. I am a software developer first and foremost (I think), and I have developed my fair share of closed source as well as open source code.

Sure, open source projects would still exist. But without an entire software development infrastructure driven by paid jobs there would be far fewer developers to make it and they would have far less knowledge, free time or interest to do so on account of the fact that we would be forced to find something else unrelated to the trade to put food on the table. And frankly that is far more important than any sort of "good" that might or might not come from what I do.

I am very much supporting a model where open source developers do not have to "find something else unrelated to the trade to put food on the table". This is the sort of thing I am would like to avoid.

I am familiar with the science model, where everything is open and everyone collaborates to get the most of what little we have. I think that woudl be a better model for pharmaceuticals and software. We could vote to make that happen.

And furthermore I do not see closed source development it self to be any more prone to bad development practices than open source.

It is more like bad deployment practices, in this and the Solarwinds one. They are inherent to closed source.

 

[PPQ_Purple]

I am familiar with the science model, where everything is open and everyone collaborates to get the most of what little we have. I think that woudl be a better model for pharmaceuticals and software.

Except in science nothing is open, everything is financed and controlled either by governments or private companies that than keep the patents and all the knowledge is behind a paywall that you can only get around by spending obscene amounts of money to access journals or by being sponsored by one of the aforementioned.

We could vote to make that happen.

How? By collectively as a human race refusing to pay for software? By collectively as software developers refusing to work for closed source? By starting a revolution to put the pirate party in power all over the world thus destroying the intellectual property system at its foundation?

And more importantly why? Why would we have any desire to change a system that is plainly working and doing so in our advantage? I very much enjoy working for a closed source development company and using the resulting money to put food on the table. And I see no way that some idealistic concept of some abstract "good" that may or may not come from changing the way the world works can compare to that. Especially not since realistically that would most likely lead to me being out of a job.

It is more like bad deployment practices, in this and the Solarwinds one. They are inherent to closed source.

No they are not. Any project can be mismanaged and any team or individual can screw up. Mistakes are inherent to any and all human individual and organization.

And how would an open source model even have prevented a bad patch from going into production? It's not like every time I post a patch to Github there is a million contributors all waiting to review it and check it for bugs before anyone downloads it. At most I get as many eyes on it as I would in a well run closed source shop that has adequate testing. At worst there is no review and no testing because those are boring and we developers like to each work on our own little improvements to the project instead. Which is the same as what would happen in a badly run closed source shop.

Open vs Closed source only dictates who can contribute to the project. It does not define how those contributions will be handled nor does it assure their quality.


To add to the above. I feel that while modern capitalism is far from a good system its fundamental premise is a good one. "I work in order to get money to buy things that make my life better" is an infinitely preferable model to "I work in order to produce some abstract good that is somehow abstractly supposed to make my life better by osmosis, probably. But even if not doing like I did good, abstractly. So that counts for something."

 

[Moriarte]

The open source tool chain is remarkably good, especially considering what it costs.

Which open source tool chain is good enough to hook up an airport/bank network with very unique devices within the next 24 hours?

An accident at the airport, leading to damages, financial loss. The culprit identified as the operating system. Who will be the designated responsible person for Squeaky Wheel OS failure?

The person who installed it, and not Microsoft. We kick responsibility down the chain, that’s why head engineer/technician at the airport is going to choose Windows. And so will the airlines.

It’s not perfect, but it has a respectable entity behind it.

 

[Samson]

This system that everyone chose to give kernel level access to? Set up by the guy that did this exact thing in 2011. This is public knowledge, anyone could look it up.

McAfee-caused PC meltdown and Microsoft-CrowdStrike outage have a common connection

George Kurtz, co-founder and CEO of CrowdStrike, who is at the center of this current issue, was also the CTO of McAfee during its notorious 2010 glitch with Windows XP.

Released in April 2010, the McAfee update led to a widespread malfunction of Windows XP SP3 systems.

The malfunction was identified as a false positive which marked a regular Windows binary, "svchost.exe," as "W32/Wecorl.a," a virus.

The symptoms were described as "Blue screen or DCOM error, followed by shutdown messages after updating to the 5958 DAT on April 21, 2010."

The McAfee incident raised questions about the company's operational efficiency. It also cost the company so much that they ended up selling to Intel, which was among those affected by this issue.

 

[Kaitzilla]

This system that everyone chose to give kernel level access to? Set up by the guy that did this exact thing in 2011. This is public knowledge, anyone could look it up.

McAfee-caused PC meltdown and Microsoft-CrowdStrike outage have a common connection

Adding to the scrutiny, George Kurtz, co-founder and CEO of CrowdStrike, who is at the center of this current issue, was also the CTO of McAfee during its notorious 2010 glitch with Windows XP.

Released in April 2010, the McAfee update led to a widespread malfunction of Windows XP SP3 systems.

The malfunction was identified as a false positive which marked a regular Windows binary, "svchost.exe," as "W32/Wecorl.a," a virus.

The symptoms were described as "Blue screen or DCOM error, followed by shutdown messages after updating to the 5958 DAT on April 21, 2010."

The McAfee incident raised questions about the company's operational efficiency. It also cost the company so much that they ended up selling to Intel, which was among those affected by this issue.

That was a long time ago.

Windows XP Service Pack 3 = Malware

or so thought McAfee :mischief: Barry McPherson, executive vice president of support and customer service, issued the apology on behalf of McAfee, saying the company was sorry for the headaches it caused for so many customers. At 6 a.m. PDT Wednesday, the company released a faulty update...

forums.civfanatics.com

Kind of crazy that the current Chief Executive Officer and co-founder of Crowdstrike was the Chief Technical Officer of McAfee back in 2010.

 

[Samson]

That was a long time ago.

Windows XP Service Pack 3 = Malware

or so thought McAfee :mischief: Barry McPherson, executive vice president of support and customer service, issued the apology on behalf of McAfee, saying the company was sorry for the headaches it caused for so many customers. At 6 a.m. PDT Wednesday, the company released a faulty update...

forums.civfanatics.com

Kind of crazy that the current Chief Executive Officer and co-founder of Crowdstrike was the Chief Technical Officer of McAfee back in 2010.

What did people think was going to happen?

 

[Samson]

What is the normal fix for blue screens? Turn it off and on again. What is the fix for the world blue screening, esp. if you are running Windows in a virtulised Azure thingy? Turn it of and on again more!!!

Azure VMs ruined by CrowdStrike patchpocalypse? Microsoft has recovery tips

Microsoft's top piece of advice to fix your broken Azure VMs is to turn them off and on again - repeatedly. No, even more than that.

"We have received reports of successful recovery from some customers attempting multiple Virtual Machine restart operations on affected Virtual Machines," Microsoft said on its Azure status page as of writing. "Several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage."

Microsoft says affected users can reboot their VMs in the Azure portal, or by using Azure CLI or Azure Shell.

Of course, that's not going to help everyone, and from there the steps are largely similar to what's been reported by other people, like CrowdStrike's head of threat hunting, Brody Nisbet: You gotta do it manually.

First, if you have a backup from before 1900 UTC yesterday, just restore that. If your backup habits are lax, then you're going to have to repair the OS disk offline, which will be more difficult for those with encrypted disks.

Once you've successfully attached a recovery disk, Microsoft says customers need to delete Windows/System/System32/Drivers/CrowdStrike/C00000291*.sys, the same recommendation Nisbet made for other affected users.

CrowdStrike's notice page for the outage has been updated to add more recovery options, as well as specific steps for AWS users and those whose Windows VMs are secured via Bitlocker.

 

[Bonyduck Campersang]

Did the outage affect only the US?

 

[Crezth]

Well it's CrowdStrike's fault, because everyone knows for 30 years or more that Microsoft is just Like This, but the real story is that the promised advantages of new age operating systems have only materialized heavier and fatter user interfaces and not any kind of inherent structural resistance to this sort of thing. I mean, CrowdStrike is also just performing a service, and one that people want or ostensibly even need. The fact Microsoft didn't anticipate the problem or that CrowdStrike didn't realize there would be a problem really just reflects the astonishing lack of specification these systems actually use. They're not altogether unlike magic at a certain level and it's not hard to imagine someone accidentally tripping over the cord that allows all Microsoft services to work. Thankfully they've got this new mandatory cloud connection thing so I'm sure that's headed in the right direction on this.

 

[Smellincoffee]

Did CS not test the live update before they pushed it to millions of machines?

 

[Crezth]

Maybe they did! But maybe they were running on an unnaturally sanitized testbed. Who can truly say?

 

[amadeus]

How come these companies I’ve never heard of all have stupid names? BlastCrunch! SwipeSpear! ActionBunt!

And doesn’t anyone know how to fix their own computers anymore?

I’m Andy Rooney and that’s my beef.

 

[r16]

when one knows actually fix computers , one can accidentally notice spying against one .

 

[Kaitzilla]

1 billion computers are blue screened and need to be manually fixed?

https://twitter.com/x/status/1814234785226604963

Ya, not a good day to fly in USA.

https://twitter.com/x/status/1814268813879206397

Woops.

1 billion was a gross exaggeration.

It was 8.5 million.

CrowdStrike’s faulty update crashed 8.5 million Windows devices, says Microsoft

Here are the details about what went wrong on Friday.

www.theverge.com

Unfortunately, it was all the computers who could afford Crowdstrike and ran the airports, hospitals, and half the Fortune 500 companies.


Southwest Airlines was unaffected because they run on 30 year old Windows.

Windows 3.1 saves the day during CrowdStrike outage — Southwest Airlines scrapes by with archaic OS

Windows 3.1 and Windows 95 save the day.

www.tomshardware.com

Like something out of Battlestar Galactica.

 

[Verbose]

Did the outage affect only the US?

Nono, it cropped up all kinds of places. Berlin Airport closed early and caused all kinds of air-travel problems across Europe fx.

 

[EvaDK]

CrowdStrike is potentially liable for some massive lawsuits coming their way, so anyone who holds stock in the company...

 

[Qactus]

CrowdStrike is potentially liable for some massive lawsuits coming their way, so anyone who holds stock in the company...

That would certainly be the way forward, but a lot of things would have to change first. Right now I don't think they can be held accountable...
Have you looked at their Terms and Conditions? It's full of these disclaimers, I wonder why hospitals and airports were running it in the first place

THE OFFERINGS AND CROWDSTRIKE TOOLS ARE NOT FAULT-TOLERANT AND ARE NOT DESIGNED OR INTENDED FOR USE IN ANY HAZARDOUS ENVIRONMENT REQUIRING FAIL-SAFE PERFORMANCE OR OPERATION. NEITHER THE OFFERINGS NOR CROWDSTRIKE TOOLS ARE FOR USE IN THE OPERATION OF AIRCRAFT NAVIGATION, NUCLEAR FACILITIES, COMMUNICATION SYSTEMS, WEAPONS SYSTEMS, DIRECT OR INDIRECT LIFE-SUPPORT SYSTEMS, AIR TRAFFIC CONTROL, OR ANY APPLICATION OR INSTALLATION WHERE FAILURE COULD RESULT IN DEATH, SEVERE PHYSICAL INJURY, OR PROPERTY DAMAGE. Customer agrees that it is Customer’s responsibility to ensure safe use of an Offering and the CrowdStrike Tools in such applications and installations. CROWDSTRIKE DOES NOT WARRANT ANY THIRD PARTY PRODUCTS OR SERVICES.

 

[danjuno]

Regarding the Microsoft vs Linux discussion earlier, what distros are typically used in Linux servers?

 

[EvaDK]

Interesting...

If those terms violate EU law and regulations though, they may not apply in the 27 EU nations. It'll be interesting to see what happens next.

 

[Samson]

Regarding the Microsoft vs Linux discussion earlier, what distros are typically used in Linux servers?

It depends what you want. It has been a while since I had anything to do with it, but it used to be the Ubuntu was the stable free option, Fedora the cutting edge free version and Red Hat the corporate version. The kids these days are all into their virtualisation for servers, and I think there are specialised distros for that.