Crowdstrike update causes IT outage - affects flights / banks / railways / TV channels and more

That is the problem, the plethora of Windows only enterprise apps.

In my place, aside from outlook and Teams, all my tools are browser based. We use Remedy which is browser based, but Ive worked in places before which had a thick client for Remedy, and a thick client for some other database system, and various other thick clients with only Windows clients.

Ya'd have a designated person on site to restore each unit. Not much IT training needed.

Which is an additional ~40k cost per site.

Therein lies the problem.

I get that but my main point is that the restoration using boot recovery software (Acronis or similar) can be done before the bsod

Even I could perform that task on each unit and travel between sites

No waiting round for a fix or a IT expert

You could in theory have a burger flipper at McDonald's designated to restore PCs with a days training and real time guidance from central IT

Can someone point out where I'm wrong here without the wise cracks

I wasn't wise cracking, and no there is nothing wrong with your idea in theory. I was simply saying the main reason against it is budget.

I work for a multi-billion dollar company and we can't get budget to buy a pencil right now. The burger flipper is still going to cost the company 25k a year per site. All for a disaster that may not happen again for another 10 years.

Edit: And your solution will only work if the next catastrophic disaster results in the same outcome. The next dodgy update that gets pushed out could cause a completely different scenario where acronis won't help.

Acronis or any such other "recovery" software is absolute junk aimed at the domestic user. To get Acronis to load before Windows somebody has to, a) sit in front of the affected computer tapping F8 or F10 etc and b) have administrator rights for said computer to actually respond to this key tapping. The actual fix itself is easily enough done and IT training is most certainly required to go drilling down in the C drive looking for the line that needs to be deleted, also a lot of sensitive information is stored on these machines and any old Joe or Mary can't be let loose on these machines.

I thought you were being serious until:

The burger flipper will of course be flipping burgers 99% of the time, therefore limited outlay

He will have the training to do the restores if needed and take directions from a IT person upstream who centrally controls the whole operation for the various affected sites.

Play the ball

What's your counterpoint apart from bigging yourself up here with wisecracks

It isn't meaningless waffle, acronis is shockingly unreliable as are all those "recovery" software packages that are aimed fair and square at the home user, windows restore is actually more dependable. What you're proposing is to install Acronis locally on every machine used by a company, have you any idea how much that would cost both in the actual cost of the software and the time setting it up in each machine before giving it to the employee? Hackers get word that company X are using Acronis so they design a little programme to hide in the "hidden" Acronis partition, goodnight dick💥. In the XP days the first thing my brother who works in IT told me to do was to turn off window restore as it was a favourite hiding place for malware etc. Anyway recovery software wouldn't stop the crowdstrike programme automatically updating itself and going bang on the newly installed image (if it actually worked) never did for me. I just have Windows in a separate partition and all my data, pics etc in another partition.

Most of the issues in your first paragraph wouldn't prevent a boot restore ?

I was mostly playing devils advocate here and I take your point that the corporate world is more complex than amsimple home user doing a restore or booting from a clone

Anyone with IT equipment already has IT support staff.

The fix for it is vastly faster than what your suggesting.

It doesn't prevent resolving the CrowdStrike issue but it does mean each affected system needs on site frontline IT staff to fix it. It may need multiple passwords (bios/uefi, local administrator, disk encryption), running a utility to get an identifier which is used to retrieve the key to access the ecrypted boot disk and then either manually delete the problematic file or run a script to do it. If booting from a USB stick with the utility and script, it also needs making bios/uefi configuration changes to boot from USB and changing them back afterwards. Whatever way it is fixed there are manual steps that can't be done remotely.

Ours were back within 30 mins of people being in the next morning. We only had a handful of systems effected though.

Some places might have had bring servers back up following a specific process.

Most places seems to have recovered which as Lamb says is quite impressive. Only a handful places seem to be still down.

Surely they'd have to offer something. My son's company lost a day's work on Friday as they couldn't access the VPN they use, him and his coworkers will get paid I can't see his company being happy about paying staff for twiddling their thumbs, these well qualified guys are on strong bobs, we aren't talking call centre minimum wage here. If I was CEO I'd be making strong enquiries about getting money from crowdstrike, eff that for a caper.

Ask your Legal and Risk Department. It's what they are there for.

I worked for a company once where at the Christmas party in lieu of a Christmas bonus they gave everyone a 50 euro Ryanair voucher. Its not much but its something. My friend decided around the 5th of January to book flights for him and his girlfriend to go away for Valentines Day and discovered the voucher had expired on December 31st.

Ooops.

As usual the only winners will be the lawyers.