Ransomware & HSE

KildareP

Prisoner6409 wrote: »

Are their IT people living in the land of Nod, who did not see this coming so much so they would have an off site backup that is maintained off the Internet, one live system and at least 2 backups, 1 not connected, its hardly rocket science.

If the source of the ransomware is some generic back office PC, then restoring from backup and/or a hot standby system will get you absolutely nowhere, the ransomware will start making instant mincemeat of that too.

You have to find the source of the compromised PC and that's no easy feat. Imagine someone clicks a dodgy link five minutes before heading home and leaves their PC turned on - that PC could progress through terabytes of data overnight. In the meantime, the user of that PC is none the wiser, most other staff who might notice are also gone home, you finally spot something is wrong but now have tens of thousands of PCs to blindly scan through (which PCs do you prioritise first?) and by the time the problem user gets in the next day to notice something is far worse on only their PC they will be one of thousands choking the helpdesk to say they can't get onto their systems.

Any PC from the last 5 years and a good office network could easily encrypt 100GB+ of data in about half an hour, not only on the compromised PC itself but on fileshares, network drives and exploits in other PCs running an unpatched OS. That can do serious damage to a network in just a few hours.

sphinxicus

JDxtra wrote: »

Times have changed since, but I remember around 10 years ago a company I was working with had a standby disaster recovery site which was online and ready in a recovery centre. We could flip between live and standby as needed.

The folks from HSE IT were using the same recovery centre. They were testing their recovery processes one day, which involved wheeling in servers on a trolley with a load of recovery tapes. It takes an incredible amount of time to recover systems and data in this manner.

Ahh memories. Takes me back to when we used to do our annual DR test. back in the early 2000's. We had a Hot DR site like yourselves. Of course, its no help when file systems are being encrypted and this is being replicated to the DR site in near real time.


Read-only file and block snapshots a big lifesaver here.

Hurrache

ShayNanigan wrote: »

I don't understand why they are still using Windows! From what I've seen, in many European countries they use Linux for several reasons.

Because it works.

BailMeOut

JDxtra wrote: »

Times have changed since, but I remember around 10 years ago a company I was working with had a standby disaster recovery site which was online and ready in a recovery centre. We could flip between live and standby as needed.

This would not help in this situation as the DR site is a replica of the production site so if the one is encrypted the other would be as well. The only way to get your data back is to decrypt it (pay the ransom) or recover from a backup.

AndrewJRenko

Prisoner6409 wrote: »

Are their IT people living in the land of Nod, who did not see this coming so much so they would have an off site backup that is maintained off the Internet, one live system and at least 2 backups, 1 not connected, its hardly rocket science.

Why would you assume that they don't have these backups?

dubrov

JDxtra wrote:

Times have changed since, but I remember around 10 years ago a company I was working with had a standby disaster recovery site which was online and ready in a recovery centre. We could flip between live and standby as needed.

Was the database replicated across both sites in real-time as well? If so then it also would have been encrypted and inaccessible.

The only option is to restore from a backup but the data would be old and could be reencrypted if the source of the malware is not removed first

AndrewJRenko

biko wrote: »

It's not "redundant", it's outdated.

But yes, I think quite a few of their systems are outdated. Too expensive to stay updated I assume.
They should just go full Linux.

What's the cost of retraining 100k users to Linux?

PatrickSmithUS

ShayNanigan wrote: »

I don't understand why they are still using Windows! From what I've seen, in many European countries they use Linux for several reasons.

Also this bit in the Journal article sort of made me smirk: "A contingency plan has been put in place to revert back to the “old-fashioned” paper-based system". Isn't that what they normally use most of the time anyway...? Time for an upgrade in the systems in any case I'd say. There's a good chance of mistakes daily just because either some papers go missing or they give a patient the wrong medication (unless the patient is very observant) because they have no system where to check. And of course then there's these types of cyber attacks because the system they do have in place is running on Windows.

There's no issue with using Windows if you have the correct system in place. All it would take is a decent email spam filter (explainer here), proper off site back ups (which should be standard for every Govt body) and some other AI solutions and this would have been either avoided or mitigated pretty quickly.


It's nigh on impossible to stay ahead of cybercriminals and the Babuk strain that is after rearing its head recently might be to blame here. https://www.theguardian.com/technology/2021/may/11/washington-police-hack-russian-speaking-babuk-gang

Hurrache

AndrewJRenko wrote: »

What's the cost of retraining 100k users to Linux?

As well as contracting professionals to port over or create APIs to allow all the systems communicate, and then have the various vendors supply applications running across Unix and Linux.

It's a stupid idea.

LillySV

Hope the bastards who did this die roaring

BailMeOut

Hurrache wrote: »

As well as contracting professionals to port over or create APIs to allow all the systems communicate, and then have the various vendors supply applications running across Unix and Linux.

It's a stupid idea.

+ this hack used a Linux command to steal the data! (rclone)

Stewball

The first page of this thread is probably the most idiotic collection of posts I've ever read on boards.

leahyl

Deleted User wrote: »

Apologies, I meant to say "zoom, teams, etc".

I did not mean to misrepresent your post and I'll edit it now. My point still stands. Blocking the installation of non-standard applications is a good thing and indicative of a good ICT policy and infrastructure, and not a bad one (which you implied).

Fair enough, thank you for the clarification

AndrewJRenko

AndrewJRenko wrote: »

What's the cost of retraining 100k users to Linux?

To the average end user a modern desktop Linux system wouldn't be a massive leap. Then I remember my days training end users on new updates and applications and it was like dealing with big kids, the resistance to changes even for minor things like going to 365 from an older version of Office was unreal.

Supporting Linux could be an issue. Of the people that work with me only 3 of us would have any experience with Linux or the need to regularly use it. Many would have done some stuff at college and never used it again so would need some refreshing.

JP Liz V1

Cancer appointments cancelled those poor patients

magicbastarder

Deleted User wrote: »

The discussion about out of date software is something of a red herring because even if they were running Windows 10 across the board, the threat actors could still leverage a zero day exploit to get access.

this is true - but if a target is running XP say, your options for compromising the system are far more open.

circadian

I wouldn't doubt for a second that upgrades to the IT system has been recommended for a long time. We seen recently some third level institutions suffering from these attacks. It was obvious institutions in Ireland were being targeted.

Let's face it, anyone in an IT management role should be treating ransomware prevention as a regular maintenance task. It's not difficult to mitigate against although I suspect a lack of funding or willingness to fund upgrades being a problem here. More often than not IT departments are seen as a cost that needs to be kept down, I've worked in enough companies to see how non tech companies often underfund their IT services, including one that got hit with ransomware and got wrecked in 2015. I promptly handed in my notice as I was the one thrown under the bus despite raising the flag constantly.

I doubt those responsible in the civil service will ever get questioned, even worse, they'll probably move to another department and continue to make the same mistakes.

madcabbage

Most of the systems are either Windows 7 or Windows 10

magicbastarder

did the HSE purchase extended support for Win7?

JP Liz V1

JP Liz V1 wrote: »

Cancer appointments cancelled those poor patients

Appalling. We have such a post Covid backlog and now this.

As for the Windows v Linux argument. Windows is perfectly fine if you keep on top of the latest cybersecurity updates and the organisation is informed and trained on latest security risks. I wonder did someone click onto a phishing email which allowed them in? Anyone know?

Corruptedmorals

This is insane. And naturally enough it's the hospitals and departments that have gone to electronic records that are affected. My department is still on charts so we are fine but so many appointments cancelled it's a disgrace. I pity anyone going into labour or having emergency surgery with no notes available. Dangerous isn't the word.

whippet

Northernlily wrote: »

Appalling. We have such a post Covid backlog and now this.

As for the Windows v Linux argument. Windows is perfectly fine if you keep on top of the latest cybersecurity updates and the organisation is informed and trained on latest security risks. I wonder did someone click onto a phishing email which allowed them in? Anyone know?

I'd say the recovery team in the HSE don't know this yet ... they may have an idea but all efforts will be on locking down and then developing a plan to bring systems back online

twinytwo

magicbastarder wrote: »

did the HSE purchase extended support for Win7?

They did.

Head of IT Ops from the HSE on the radio now saying they believe it was a zero day exploit.

BrianD3

whippet wrote: »

I'd say the recovery team in the HSE don't know this yet ... they may have an idea but all efforts will be on locking down and then developing a plan to bring systems back online

Absurd if all of this is because somebody clicked on a link or attachment in an unsolicited email. Any consequence for them?

IME these emails are very easy to spot even if they are more clever than the "click here to win money!!" type emailed links that we used to see.

If someone uses a computer as part of their job, it's also part of their job not to make these errors. Just as if someone works in a warehouse full of valuable stock, it's part of their job not to leave the door open and alarm off.

AndrewJRenko

circadian wrote: »

I wouldn't doubt for a second that upgrades to the IT system has been recommended for a long time. We seen recently some third level institutions suffering from these attacks. It was obvious institutions in Ireland were being targeted.

Let's face it, anyone in an IT management role should be treating ransomware prevention as a regular maintenance task. It's not difficult to mitigate against although I suspect a lack of funding or willingness to fund upgrades being a problem here. More often than not IT departments are seen as a cost that needs to be kept down, I've worked in enough companies to see how non tech companies often underfund their IT services, including one that got hit with ransomware and got wrecked in 2015. I promptly handed in my notice as I was the one thrown under the bus despite raising the flag constantly.

I doubt those responsible in the civil service will ever get questioned, even worse, they'll probably move to another department and continue to make the same mistakes.

What mistakes did the HSE IT folks make?

topdecko

This is a huge issue. Healthlink down for us in GP land and we don't seem to have a back up in place. We cannot refer for COVID tests now and there will be backlog with other referrals, hope they get sorted soon however the haphazard nature of irish Health IT infrastructure is very concerning. Absolutely no plan in place to respond to this - these attacks are part of modern life, must have contingency ready to go - not merely cancelling appointments etc.

Hamsterchops

Thankfully all the Anti Virus software is up to date & the latest version, so that's good news. The HSE also acted very quickly to shut down all systems . . .

So how much is the ransom? and who's behind it?

Pandiculation

Unknown - apparently they’ve made no demands yet. It’s a human directed attack though, not a purely automated thing.

BrianD3

BrianD3 wrote: »

Absurd if all of this is because somebody clicked on a link or attachment in an unsolicited email. Any consequence for them?

IME these emails are very easy to spot even if they are more clever than the "click here to win money!!" type emailed links that we used to see.

If someone uses a computer as part of their job, it's also part of their job not to make these errors. Just as if someone works in a warehouse full of valuable stock, it's part of their job not to leave the door open and alarm off.

I was just in a meeting and one of my colleagues had been involved in phishing training. After the training, they sent a phishing mail to the attendees. Literally, the next email they received, was a phish. 20% of the attendees clicked the link. 20%! Having literally just received the training.

Some people: