Massive Windows/CrowdStrike Fail

[Kaitzilla]

I had a family member get their non-emergency surgical procedure cancelled today.
Crowdstrike

 

[PPQ_Purple]

I am not convinced by the legal argument "windows cannot make its systems secure or else the anti-malware industry would sue them". Linux vendors have survived.

That is not the argument though. Allow me to try and explain this in layman's terms.

When it comes to how operating systems allow a program to operate every operating system this side of UNIX has two modes of doing so. They are called user mode and kernel mode.

User mode is where your typical programs operate. This includes things like video games, browsers, etc. User mode can be seen as a sort of sandbox. The analogy is not perfect but it works. In user mode your program basically gets a chunk of memory and interacts with the OS for everything outside of that. So for example if it wants to draw to the screen it calls the OS which than calls the graphics card driver. If it wants to read from the hard drive it has to call the OS which calls the hard drive driver. Etc. Etc. And for most programs this is good enough. They do not need the benefits of direct hard drive access and benefit greatly from the security that comes from having the OS as a middleman who checks the calls for anything stupid or malicious.

And as far as security goes, this works remarkably well. The built in API of modern operating systems be they Windows, Unix or what ever apple is calling theirs these days is remarkably good. And the days of viruses operating in user mode and breaching the built in security of Windows to mess your system up is basically over and has been since something like XP Service Pack 3. Unless you are running an ancient operating system or are one of those people that absolutely refuse to update for years you are pretty much safe with just the base windows alone.

The issue though is that while this setup is good for most programs it does not work for all of them. Some programs like drivers simply must be able to bypass the system and have direct access to hardware in order to operate. This includes things like drivers. You can after all not have a hard drive driver that stores data to the hard drive by calling the operating system to tell the hard drive driver to store data. And that is where the vulnerabilities in modern systems come from.

So what happens if the driver, the piece of software that absolutely has to operate in kernel mode is horribly insecure? Well, you would think that does not happen but actually it happens a lot. Hardware vendors are notorious for being good at making hardware and than cheeping out on the drivers. Just look at your average printer software as an example and multiply that by 11 for all the drivers your average user can't interact with and complain about to customer support.

And that is how viruses get into your system. A good virus these days is an user mode program that uses innocuous and nominally safe operating system API commands that should with a correctly written driver do absolutely no harm to send commands to a very bad driver it knows is on your system and exploit bugs in that driver to make it do horrible things like for example running your virus code instead of its own. And once that happens you are what we in the industry call capital S Screwed. And not the nice little wood screws that go under your finger nails either...

And there in lies the problem. Once bad code is in kernel mode it can do anything it likes including and not limited to disabling or destroying any other program running on the system. This includes any and all antivirus software that might be running. At the same time, that software can literally do nothing to prevent or stop that unless it also is running in kernel mode. Which is why antivirus software does.

And OS vendors are not unaware of this problem. They each have their own ways of handling it but it all boils down to basically making it so that you can only install drivers that have been tested and certified by the vendor through some certification program that the creators have to pay for. And this too usually works remarkably well.

In this case however, the problem was that one company published a patch for its kernel mode program that was defective. And patches generally do not have to be decertified each and every time because that would make regular updates basically impossible.

So it is not a legal thing. It's just some really bad testing on the publishers part coupled with frankly really bad luck.

---

Or is that really the case?

Why do all these devices even need antivirus software to begin with? Antivirus software is only really required if you have a machine that is actually likely to be injected by malicious code which in these days basically means only one thing which is the internet. That is why basic corporate security practices which at this point I am pretty sure are even in ISO standards is that devices in large internal networks such as banks, hospitals or airports should be behind layers of security and operating within a corporate controlled network.

Therefore they should not require individual AV protection any more than someone living in a house made of bricks with no windows whose only contact with the outside world is a telephone should require a bullet proof vest.

Indeed, why do they even have access to the internet to begin with? Does your average doctor, administrator or air traffic controller really have to check facebook from his work machine? I think not. And the applications that do interact with outside customers should be hosted on secured servers which should be sufficiently isolated that even if brought down the only thing to fail would be the customer facing side.

So what really happened here is not just some poor developer pushing a bug to millions of people and crashing the world. No matter how much everyone screeches that it is.

Rather it's an open and poignant demonstration of the fact that bad security and computer infrastructure practices are so insanely prevalent in the modern landscape that there is an entire unseen industry of corporate software security built around patching the problems caused by it just barely well enough that everything keeps working. That is until it doesn't.

And when that one point of failure makes a mistake the whole house of cards comes crashing down.

Spoiler TLDR :

EDIT: Included TLDR.

 

[Samson]

So what happens if the driver, the piece of software that absolutely has to operate in kernel mode is horribly insecure?

This is the whole point, and why I cannot consume DRM'ed content over HTML5 or use Nvidia graphics cards (NSFW, language) because I will not allow closed source drivers on my machine. You should not be giving untrusted entities kernel access to your machine. If I can do it my doctors medical records company can do it.

 

[PPQ_Purple]

This is the whole point, and why I cannot consume DRM'ed content over HTML5 or use Nvidia graphics cards (NSFW, language) because I will not allow closed source drivers on my machine. You should not be giving untrusted entities kernel access to your machine. If I can do it my doctors medical records company can do it.

Thing is, the windows certification program is actually remarkably good at making sure these closed source drivers are legitimately safe to use in the vast supermajority of cases. To the point where I would actually say that the threat of someone looking through the source of an open source driver and finding (or creating) a bug to exploit is on par or greater than the threat of someone finding a bug in a closed source one these days. Say what you want about MicroSoft but they are serious about this and have put their money where their mouth is.

And situations like this one where someone pushed a bad patch can and do happen to open source drivers as well. Just remember what happened with OpenSSL back in the day. Or that time university students pushed bugs into the Linux kernel as a laugh. The reality is simply that at some point you have to trust someone. And that someone is occasionally going to fail be that through ineptitude, bad luck or malicious intent. Which is why you should strive to build your system around sound security and usage practices such that when the inevitable does happen the damage is minimized.

On an end user level this means running regular software updates and following basic security best practices such as using Addblock, not visiting shady sites and downloading fishy attachments and on a corporate level it means not building your corporate network on a house of cards and than hoping a fix all software vendor can come and make it secure. Because single point of failure solutions work great right up until they don't.

It's no different than having a spare tire in your car really. Tires will pop. And if you are stuck in the middle of the desert hundreds of kilometers from civilization in 50 C weather without a spare it's not the manufacturer that's to blame.

 

[Samson]

Thing is, the windows certification program is actually remarkably good at making sure these closed source drivers are legitimately safe to use in the vast supermajority of cases.

In what way is it "remarkably good"? One fat fingered mistake producing one invalid file and the whole world blue screens!!!

The open source tool chain is remarkably good, especially considering what it costs. About the OpenSSL thing, this is the team that was responsible (primary article has disappeared from the web?)

How it happened (can you imagine the amount of money that was spent on Falcon Platform, compared to to these two guys?):

Early in the morning of Thursday, June 15, 2006, Marquess and Henson were sent a near-impossible task. The Cryptographic Module Validation Program, a joint U.S.–Canadian validation body that fell under the auspices of each country’s government, wanted the team to make a raft of complicated code changes to meet the requirements for accreditation under its security standard. And it had to be done fast. “If we didn’t do it by Monday morning, they’d reject our validation, we’d have to start over again, and it’d take another three years,” says Marquess, bitterness rising in his voice at the memory. “Now this is a huge amount of work — days of silly, pointless work. And this pissed Steve Henson off.”

Henson was on his summer vacation in Great Yarmouth, a seaside resort near Norfolk, England. He had nothing but an HTC Hurricane cell phone, a laptop, and a frustratingly slow internet connection for company. Like Marquess, he was incensed: After several years struggling with government bureaucracy for little to no pay, he wasn’t about to give up now. As Marquess puts it, “At that point in time, completing the project became a matter of stubbornness.”

Henson sent off an email to Marquess around 4 a.m. “I was irritated by this ultimatum, couldn’t sleep, and decided to use this time to see if I could get a solution,” Henson explains.

“And he got it done,” Marquess says quietly. Henson worked through the night, and sent off a preliminary solution that could work. Big government was placated. “That’s the kind of guy you want at your back.”

Before that June weekend and since, companies and government departments have benefitted from OpenSSL’s free price and constant updates, often without giving back. Overwork and understaffing — two things that have been cited as the main causes of the Heartbleed bug, which suddenly brought OpenSSL and its gatekeepers to the world’s attention — aren’t news to Steve Henson and Steve Marquess. But thanks to Heartbleed, everyone else is beginning to understand what the duo have known for a while: Something needs to change, and goodwill and fond words alone won’t cut it. Right now significant parts of the internet’s cryptographic security rely on a tiny handful of people who are already stretched to the limits. If that fails, the modern world as we know it could cease to work as it should.

 

[PPQ_Purple]

In what way is it "remarkably good"? One fat fingered mistake producing one invalid file and the whole world blue screens!!!

Mistakes happen. They happen a lot. They happen every waking moment of every day of every software developer. Some are caught immediately, others in testing and some even make it to production. And I say this as someone who has been typing code for so long my first programs can legally start making baby programs of their own by now.

The very fact this sort of thing is even news as opposed to just another day at the office proves that the system is working.

The open source tool chain is remarkably good, especially considering what it costs.

Of course it is. But so is closed source. A banana being tasty does not make steaks taste less good. But live on one alone and you'll get malnutrition all the same.

About the OpenSSL thing, this is the team that was responsible (primary article has disappeared from the web?)

Showing that Open Source is in fact not safer than closed source regardless of what the evangelists would have you believe.

After all, who is more likely to produce good code? A team of professionals whose paid day job it is to do so or some guy who does it in his spare time as a hobby with occasional help from random people on the internet?
The one who is a better developer in general and happens to have had his morning coffee just right that day. Because to err is human.

 

[Samson]

After all, who is more likely to produce good code?

It is not a question of likely, we can look at the data. Linux runs most of the web, supply chain issues have brought down Windows globally multiple times.

 

[PPQ_Purple]

It is not a question of likely, we can look at the data. Linux runs most of the web, supply chain issues have brought down Windows globally multiple times.

But it was newer Windows that was at fault but secondary software that was closed source some times and open source at others. Which is the point.

Neither system of software development is the panacea for bad code.

 

[Samson]

But it was newer Windows that was at fault but secondary software that was closed source some times and open source at others.

I mean the closed source Micro$oft Windows / ISS tech stack has taken down the world multiple times, the open source Linux / Apache etc. tech stack (that runs more than 3/4 of the internet) has not. Yet many magnitudes more money has been spent on the MS one.

 

[PPQ_Purple]

I mean the closed source Micro$oft Windows / ISS tech stack has taken down the world multiple times, the open source Linux / Apache etc. tech stack (that runs more than 3/4 of the internet) has not. Yet many magnitudes more money has been spent on the MS one.

Did you seriously just make the claim that all software running on Windows is always closed source and all software running on Linux is always open source?

Also, while we are at it riddle me this. If Linux really is the backbone of most of the internet how come every time most of it goes down it is a problem with Windows systems? Surely all those Linux machines that run all the servers in all the companies should still be up and running? After all, if the Server Windows market share is so low even a complete catastrophic collapse of all such systems should hardly be this noticeable.

 

[Samson]

Did you seriously just make the claim that all software running on Windows is always closed source and all software running on Linux is always open source?

No, not at all (I am including Lightspeed in Linux's 76%). But presenting the stack as generally open compared to closed source gets at the heart of the problem. The idea of a a MANDATORY (presumably contractually) unattended kernel driver installation in the linux world sounds mad.

 

[PPQ_Purple]

No, not at all (I am including Lightspeed in Linux's 76%). But presenting the stack as generally open compared to closed source gets at the heart of the problem. The idea of a a MANDATORY (presumably contractually) unattended kernel driver installation in the linux world sounds mad.

The heart of the problem though is NOT that the driver was good or bad or how it was installed. It's that it existed in the first place. Or rather that it had to exist.

The heart of the problem is the fact that the world of corporate IT (and by this I mean all businesses) is built like a house of cards by people who are more concerned with pinching every penny that they can and only complying to the bare minimum letter of the law if that. Which leads to our entire world being built on systems that are full of misconfigured and misbuilt and filled with appalling security and data integrity practices that would make you loose sleep at night if you knew about them.

So much so that there is an entire industry built on providing snake oil solutions to the problem. That is why you can't spend a week without hearing of some IT disaster or data breech or similar.

 

[Samson]

The heart of the problem though is NOT that the driver was good or bad or how it was installed. It's that it existed in the first place. Or rather that it had to exist.

The heart of the problem is the fact that the world of corporate IT (and by this I mean all businesses) is built like a house of cards by people who are more concerned with pinching every penny that they can and only complying to the bare minimum letter of the law if that. Which leads to our entire world being built on systems that are full of misconfigured and misbuilt and filled with appalling security and data integrity practices that would make you loose sleep at night if you knew about them.

So much so that there is an entire industry built on providing snake oil solutions to the problem. That is why you can't spend a week without hearing of some IT disaster or data breech or similar.

I am not quite sure how this is different from "The heart of the problem is the fact that the world of corporate IT (and by this I mean all businesses) is built like a closed source house of cards by people who are more concerned with pinching every penny, rather on the open source house of cards that has been built. If they had give the OS world one hundredth of what they gave the closed source world it would be so good by now.

Are vehemently agreeing?

 

[PPQ_Purple]

I am not quite sure how this is different from "The heart of the problem is the fact that the world of corporate IT (and by this I mean all businesses) is built like a closed source house of cards by people who are more concerned with pinching every penny, rather on the open source house of cards that has been built. If they had give the OS world one hundredth of what they gave the closed source world it would be so good by now.

Are vehemently agreeing?

Because open vs closed source is irrelevant to the discussion.

The way your car was made hardly matters if you drive it off a cliff.

 

[Samson]

Because open vs closed source is irrelevant to the discussion.

The way your car was made hardly matters if you drive it off a cliff.

Fair enough. I think the economic model has been influential in the output, but that I cannot prove.

Unrelated:

Spoiler Fairly authoritative youtube on the subject :

 

[PPQ_Purple]

Fair enough. I think the economic model has been influential in the output, but that I cannot prove.

As far as sheer software quality is concerned both Open and Closed source solutions have given us some fantastic examples of working software and some real lemons .After all, one could hardly argue that such gems as Visual Studio and Paint.NET are bad products just because they are closed source or that most versions of desktop oriented linux are a good idea for the average consumer.

But just because we can pick bad or good examples from both categories does not render either of them good or bad. Indeed, they are not even in opposition. Closed and open source solutions and development models exist in a symbiotic relationship to form the wider computing landscape that we exist in. Remove either one and the other simply can not close the gap.

After all, if companies could not keep their software secrets hidden they could not sell software as a product. And if companies could not sell software as a product than all us developers could not have a day job to pay for our life of tinkering with open source projects. And consumers could not rely on having a reliable (as reliable as software gets anyway) baseline product for things like operating systems that comes with robust (well, as much as such a thing even exists these days) support for when it fails for our projects to integrate with.

A good microcosm of this is the modding community. If not for the closed source product that is the game you could not have enthusiasts making mods for it. And without mods some games (*cough* bethesda *cough*) are not playable. Both are needed to create the whole.

It's just that in general when you pay for something, and with closed source you usually are, you are more likely to remember the times it worked bad and you felt cheated where as when something is free you are more likely to remember the times you were pleasantly surprised.


But all this hardly matters if the entire stack you have is just plain wrong because some penny pincher decided that a bank does not need a backup solution for their financial data and the hard drive your money is on gives out.

 

[Samson]

After all, if companies could not keep their software secrets hidden they could not sell software as a product. And if companies could not sell software as a product than all us developers could not have a day job to pay for our life of tinkering with open source projects. And consumers could not rely on having a reliable (as reliable as software gets anyway) baseline product for things like operating systems that comes with robust (well, as much as such a thing even exists these days) support for when it fails for our projects to integrate with.

A) Some companies sell OS software as a product, though the chalenges to that model are significant.
B) We could look at it another way, more like a public good. It is up to us, but based on input compared to output I see no evidence the closed source model is best for the world.

I think Linux is a much better reliable baseline operating system that comes with robust support compared to Windows. What support does the consumer get with Windows?

 

[PPQ_Purple]

A) Some companies sell OS software as a product, though the chalenges to that model are significant.
B) We could look at it another way, more like a public good. It is up to us, but based on input compared to output I see no evidence the closed source model is best for the world.

I mean no offense with this. But frankly you are being extremely naive if you honestly think that what is best or even remotely good for the world has any place in any serious discussion of how the world actually operates.

And if you truly are committed to doing good you need to learn how to work with and within the system rather than dreaming about things that can newer be.

I think Linux is a much better reliable baseline operating system that comes with robust support compared to Windows. What support does the consumer get with Windows?

The entire windows support infrastructure set up by Microsoft. I mean, it's not fantastic but it beats trawling internet forums and reddit posts for help with your particular Linux distribution of choice. And that's before you buy into paid support like corporations do. There is a reason why the most used Linux distributions for corporate are ones like Red Hat that come with support included.

 

[Samson]

I mean no offense with this. But frankly you are being extremely naive if you honestly think that what is best or even remotely good for the world has any place in any serious discussion of how the world actually operates.

In a way I agree, but if you take this attitude what are we talking about? I am saying something like "The world would be better if more people used Open Source Software". That is a valid argument and I would say individually actionable, and so is reasonable to say here. The fact that it will have an insignificant effect at best is not actually that relevant.

And if you truly are committed to doing good you need to learn how to work with and within the system rather than dreaming about things that can newer be.

I have contributed a bit to open source code over the years, and more to open knowledge.

 

[PPQ_Purple]

Basically I am not at all opposed to the idea of people making money off software and do not consider the existence of software as a product to be a bad thing. Nor do I see a realistic world where software would be as developed or as prolific if not for a financial incentive to create it.

Sure, open source projects would still exist. But without an entire software development infrastructure driven by paid jobs there would be far fewer developers to make it and they would have far less knowledge, free time or interest to do so on account of the fact that we would be forced to find something else unrelated to the trade to put food on the table. And frankly that is far more important than any sort of "good" that might or might not come from what I do.

Not to mention the fact that without for profit interests pushing software on people the proliferation of computers would be far less leading to both fewer developers and less need for software overall.

In short, I feel that the evidence shows that without closed source for profit development computers would be a niche interest field like HAM radio like it was in the early days before someone in marketing decided we needed a mainframe in each bank and a PC on each desk top.

And furthermore I do not see closed source development it self to be any more prone to bad development practices than open source. In both cases the problem or lack there of lies in the development team and the management that runs them. I have worked with some excellent closed source teams and some really bad ones. And I have seen open source projects get both well run and mismanaged. And in all case it was not the model that is at fault but the people using it.

Finally I do not consider the very concept of capitalism and working for personal gain as opposed to the public good to be a bad thing. It is just the way reality is. And we need to accept it and go with the flow.