What is wrong with the site? (Please read updated post #1)

seamus

glued wrote: »

Seriously what is wrong with the site? Whatever is going wrong needs to be fixed. Most of yesterday I couldn't log on. No information about the down time again. What are Boards.ie paying people for?

It's being DDOSed.

Which is something that the site itself can't really "fix" quickly without spending obscene amounts of money.

You can heavily mitigate it by moving your infrastructure onto a cloud service like AWS or Azure. But that's a project that takes in the region of several weeks if you already know what you're doing, several months if you don't.

NachoBusiness

glued wrote: »

Seriously what is wrong with the site?

The problems are quite obviously to do with the new rebuild and so I don't know why nobody has just come out and just said as much.

Anyway..

"Find all posts that quote.." is not working for me if that's of any use to anyone anywhere who's pressing buttons.

Still using the old version. Haven't pressed this Beta thing as of yet.

glued

seamus wrote: »

It's being DDOSed.

Which is something that the site itself can't really "fix" quickly without spending obscene amounts of money.

You can heavily mitigate it by moving your infrastructure onto a cloud service like AWS or Azure. But that's a project that takes in the region of several weeks if you already know what you're doing, several months if you don't.

That's fair enough but the lack of communication is annoying. Boards.ie is a business. Its revenue is generated by its users. The longer this goes on the less likely people will be using the site in the future.

This is a great site bar the constant downtime and lack of communication.

Boards.ie: Chris

seamus wrote: »

It's being DDOSed.

Which is something that the site itself can't really "fix" quickly without spending obscene amounts of money.

You can heavily mitigate it by moving your infrastructure onto a cloud service like AWS or Azure. But that's a project that takes in the region of several weeks if you already know what you're doing, several months if you don't.

This.

NachoBusiness wrote: »

The problems are quite obviously to do with the new rebuild and so I don't know why nobody has just come out and just said as much.

Not this.

seamus

glued wrote: »

That's fair enough but the lack of communication is annoying. Boards.ie is a business. Its revenue is generated by its users. The longer this goes on the less likely people will be using the site in the future.

This is a great site bar the constant downtime and lack of communication.

To be fair, the site has been having some performance difficulties over the last two weeks. I wouldn't call that "downtime", and I can't actually remember the last time before this that where the site suffered more than a few minutes of issues or unplanned downtime. They've done a great job in recent times in keeping a high-traffic site on the air on a relatively tight budget.

Especially if you were around for the 2003-2009 era

No, that's not an excuse and doesn't make it "OK", but at the same time there's no need to make a mountain out of a molehill.

The site is under DDOS, and DDOS attacks are weird and annoying. There's a lot of hesitance to even acknowledge them because then you're giving the attacker legitimacy. 99% of the time these are pathetic basement-dwellers who DDOS because of some perceived wrong or desire to get a kick out of it. They're hoping to see the staff get frustrated or beg for it to stop. Because that's the kind of sad and pathetic morons these kids are.

Is it related to the beta? Unlikely, but possible. I've worked on sites that have managed to "DDOS" themselves by writing some bad code that generated stupid amounts of traffic, or through the actions of another site that messed up (google "slashdotted").

I would trust though that if this had anything to do with the beta, it would have easily been tracked down and stopped by now.

Boards.ie: Chris

Quazzie wrote: »

Friday now, and not a single fúck was given all week by the staff of Boards.ie

Thanks for the feedback Quazzie.

Minor reflection on things.

When these things started kicking off, I spent an hour on the side of a Belgian motorway on holiday liaising with multiple people to try and figure out what was happening and what we could do try and make things better, which we managed.

Shortly after that I left the company, but out of a courtesy to people I consider friends I said I'd stick around outside of office hours while they found a suitable replacement, who'll be starting in a couple of weeks, to try and help keep things ticking over when I could (considering I have a full time job elsewhere and a family, that's not a whole lot of time to spare). So here I am on a Saturday afternoon (a nice change as the past few weeks it's been Saturday and Sunday early mornings and late evenings), now trying to mitigate the kind of attack that is often leveraged at multi-billion dollar companies. 71Gbps at times for anyone that can contextualise that. Is it likely I'll be able to do much about it? Not really. Is that frustrating? Better ****ing believe it. Am I still going to spend hours of my weekend trying? I don't know why, but sure.

KERSPLAT!

Can you not do a geographical block on IPs? Block everything from outside Ireland and UK or whatever suits

Penny Tration

Boards.ie: Chris wrote: »

Thanks for the feedback Quazzie.

Minor reflection on things.

When these things started kicking off, I spent an hour on the side of a Belgian motorway on holiday liaising with multiple people to try and figure out what was happening and what we could do try and make things better, which we managed.

Shortly after that I left the company, but out of a courtesy to people I consider friends I said I'd stick around outside of office hours while they found a suitable replacement, who'll be starting in a couple of weeks, to try and help keep things ticking over when I could (considering I have a full time job elsewhere and a family, that's not a whole lot of time to spare). So here I am on a Saturday afternoon (a nice change as the past few weeks it's been Saturday and Sunday early mornings and late evenings), now trying to mitigate the kind of attack that is often leveraged at multi-billion dollar companies. 71Gbps at times for anyone that can contextualise that. Is it likely I'll be able to do much about it? Not really. Is that frustrating? Better ****ing believe it. Am I still going to spend hours of my weekend trying? I don't know why, but sure.

sorry to hear you've left the company and thank you for your time in trying to fix the issues.


Ultimately, if the site is under a DDOS attack, they're doing pretty damn well to even keep the site live, lads.

It sucks yeah but do you really think dav, niamh, and the other staff are sitting on their holes doing nothing? Maybe there's nothing to update with yet. Doesn't mean they're not trying to fix it!

glued

The site clearly needs more resources.

glued

OSI wrote: »

Wouldn't make a difference for these kinds of attacks unless you happen to be the owner of a Tier 4 data center.

What is point in someone attacking the site in this way?

KERSPLAT!

OSI wrote: »

Wouldn't make a difference for these kinds of attacks unless you happen to be the owner of a Tier 4 data center.

Why not? Genuinely interested.

Penny Tration

glued wrote: »

What is point in someone attacking the site in this way?

It's generally basement dwelling nerds with a grudge and way too much free time.

Happened on a much bigger site I modded several years ago. Even with the site running at a very large profit, it was offline for two full weeks before it could be brought back from the DDOS

KERSPLAT!

OSI wrote: »

Imagine a motorway with lots of small roads coming off of it. The motorway is equivalent to the connection coming from the internet to the a datacenter, and the small roads off it, the lines rented to the various people that have cabinets in the data center. The cabinet owners only have control over a gate at the end of their road and no control over the motorway. Now image you flood that motorway with so much traffic that no one can even get access to the gate, so any rules you have about access at that gate never come into the equation.

That motorway would be Digiweb in this case? Or one of Digiwebs lines?

Azalea

glued wrote: »

Seriously what is wrong with the site? Whatever is going wrong needs to be fixed. Most of yesterday I couldn't log on. No information about the down time again. What are Boards.ie paying people for?

When you wrote that did you honestly think people weren't working on fixing it?

No, I don't want to be a mod but having experience of being in similar situations where something has gone wrong and we're all running around like headless chickens behind the scenes and then people pipe up that we don't care and aren't bothering and are ignoring the issue, when it's the exact opposite... it's unfair, that's why I empathise with Boards here.
I agree the issues are annoying (and I disagree you have to be a subscriber to have the right to criticise) but just because nothing seems to be happening doesn't mean there isn't work going on behind the scenes.

Zaph

glued wrote: »

The site clearly needs more resources.

If you could make a list of what's needed and where the funding for them will be coming from that would be great, thanks.

Look, everybody's frustrated as hell right now with all the problems, myself included. But as Chris has explained, this isn't something that's happening within Boards, it's external to the site and as a result there's only so much that can be done. Would we like more info from the office? Sure we would, but to be fair they're suffering as much as the rest of us and probably don't know when things will be better either. Maybe more info would be worthwhile, but it's not going to fix anything and tbh I'd rather the guys in the office spent their time trying to fix things instead of wasting time posting the same thing repeatedly on this thread.

KERSPLAT!

OSI wrote: »

Yep

Would Digiweb as a whole, all of their data centre, not be down? And would Digiweb themselves not have resources in place, a firewall, to do a geographical block on IPs causing the issue?

If its only one of the Digiweb lines would it not be down to them and their firewall to handle this?

KERSPLAT!

Zaph wrote: »

If you could make a list of what's needed and where the funding for them will be coming from that would be great, thanks.

Look, everybody's frustrated as hell right now with all the problems, myself included. But as Chris has explained, this isn't something that's happening within Boards, it's external to the site and as a result there's only so much that can be done. Would we like more info from the office? Sure we would, but to be fair they're suffering as much as the rest of us and probably don't know when things will be better either. Maybe more info would be worthwhile, but it's not going to fix anything and tbh I'd rather the guys in the office spent their time trying to fix things instead of wasting time posting the same thing repeatedly on this thread.

I agree with most of that but to post here takes 30 seconds, something along the lines of 'its a DDoS, we're working on it' wouldn't go a miss. People had no idea if it was servers, DDoS, some DNS meltdown in the US or the Beta being rolled out

Only last week Dav apologised for the lack of communication.

Anyway, fair play to you, Chris, and everyone else working on it. Hopefully things get sorted soon.

KERSPLAT!

Sorry to spam but the site seems much better now, on touch at least. Is there a let up in the DDoS or has something changed?

KERSPLAT!

OSI wrote: »

I can't really speak for Digiweb, but as a former NOC your priority is the bigger picture. If one customer is getting hammered to the point it's affecting others, you just stop accepting traffic for that customer.

That's my point, if boards is being hit with 60GB/s that would surely affect a lot of other customers, bow is boards up at all!?

glued

Zaph wrote: »

If you could make a list of what's needed and where the funding for them will be coming from that would be great, thanks.

Look, everybody's frustrated as hell right now with all the problems, myself included. But as Chris has explained, this isn't something that's happening within Boards, it's external to the site and as a result there's only so much that can be done. Would we like more info from the office? Sure we would, but to be fair they're suffering as much as the rest of us and probably don't know when things will be better either. Maybe more info would be worthwhile, but it's not going to fix anything and tbh I'd rather the guys in the office spent their time trying to fix things instead of wasting time posting the same thing repeatedly on this thread.

I get what you're saying but it takes 2 seconds to post a quick message on Twitter or whatever. The lack of communication is making people frustrated not the site itself.

There is no excuse for the lack of communication. Especially for a business like Boards.ie.

cowboyBuilder

DDOS of attacks, the purpose is to just take down the site ? they don't gain anything from it other then disrupting it for everyone else ?

Why would boards be a target ?

SMJSF

I don't know if anyone is having the same problem as me on chrome, but the past 2 nights, on laptop and phone, boards refuses to load, but when I get into incognito mode, it works fine.
And yes, I've reset my internet settings and cleared cookies and the rest on both machines, and still getting a total blank screen without "private" mode.
I don't know if it's something connected to the first problem.

Dolbert

SMJSF wrote: »

I don't know if anyone is having the same problem as me on chrome, but the past 2 nights, on laptop and phone, boards refuses to load, but when I get into incognito mode, it works fine.
And yes, I've reset my internet settings and cleared cookies and the rest on both machines, and still getting a total blank screen without "private" mode.
I don't know if it's something connected to the first problem.

Huh, you're right! Touch and mobile sites work perfectly on incognito mode too, weird.

_Tombstone_

Dolbert wrote: »

Huh, you're right! Touch and mobile sites work perfectly on incognito mode too, weird.

For now.

Second Toughest in_the Freshers

cowboyBuilder wrote: »

DDOS of attacks, the purpose is to just take down the site ? they don't gain anything from it other then disrupting it for everyone else ?

Why would boards be a target ?

With the amount of banned accounts here, I wouldn't be surprised if it was someone holding a grudge....

Does a ddos explain the 404s and scripts going a bit mad and blank PMs being randomly generated?

I can second what Chris said about the DDoS attacks being external. I'm aware of another company who have servers in the Digiweb datacentre (confirmed by <1ms pings to www.boards.ie ) and they have also experienced an increase in DDoS attacks as of late. It's a pain in the hole, might need to look at Cloudflare or something similar.

uck51js9zml2yt

Karsini wrote: »

I can second what Chris said about the DDoS attacks being external. I'm aware of another company who have servers in the Digiweb datacentre (confirmed by <1ms pings to www.boards.ie ) and they have also experienced an increase in DDoS attacks as of late. It's a pain in the hole, might need to look at Cloudflare or something similar.

A company I worked for had client sites with Digiweb and we used CloudFlare as standard.

When DDOS attacks occured(which they did frequently) they took down the whole DC so CloudFlare just didnt do anything

challengemaster

Second Toughest in_the Freshers wrote: »

With the amount of banned accounts here, I wouldn't be surprised if it was someone holding a grudge....

It would want to be a serious grudge to spam 70Gbps of data during a DDOS. You'd either need a botnet of hundreds of thousands of machines at your command, or spam a DNS resolver...

https://blog.cloudflare.com/65gbps-ddos-no-problem/

Gael23

None of these problems were happening before the "improvements" of recent months. It's technology, things don't always work as they should, but it's happened once too often at this point.