Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie
So what on earth did I do?
Options
Comments
-
Dravokivich wrote: »Did it need a mass PM? Surely a site wide announcement would have been enough?
I liked the PM. Made me feel loved
0 -
-
-
-
-
Advertisement
-
-
-
-
Maybe the delay is because the imps have to run up and down popping the PMs in everyone's pigeon holes? And maybe your imp stopped for a toilet break? And he'll be on to you in a minute?
Hope so Not tech savvy at all but it must have been a nightmare trying to get the site back up. 0 -
-
Advertisement
-
-
-
-
-
-
Boards.ie: Declan wrote: »The PM was ... just people who had posts that were deleted.
Ah, that makes sense. To be fair now, I think the site going sideways was an Act of God: some of the stuff I wrote that got deleted, was probably better never posted in the first place. I'll take it as a reprieve
0 -
-
-
Boards.ie: Declan wrote: »The PM wasn't to everyone - just people who had posts that were deleted. We want to make sure they have the opportunity to repost.
how do you know who posted in that time if the data is gone?
Well done dav for owning up, it must have been a nervous phone call to the admins saying i broke the whole site. I think i was in college with him.
0 -
-
Advertisement
-
-
-
-
-
Honesty is, as they say, the best policy.
I fked up, I have often times said here on Boards that people shouldn't be afraid to put their hand up and admit a mistake and I'd be some can of piss to not apply that same rule to myself.
When something like this happens, there is literally nothing I can do from a technical point of view to help the team, so the very least I can do is take to the Twitters and keep people informed - it's the handiest means of talking to you when something's wrong. Lots of people were asking what happened and I told them. Others were wondering if they could help and that was very much appreciated. One lady thought that she'd call us idiots for doing maintenance on "our busiest day" which is more funny than anything else given that Sunday is our quietest day by a huge margin. There were a lot of roundball people talking about Manchester Rovers and how it was a good thing there was no discussions about them (is my contempt for professional soccer showing?
). By and large though, you've all been really understanding and patient with us and me and I can't thank you all enough for that, you had every right to be annoyed but you weren't.
It is unfortunate that this seems to contrary to people's expectations and experiences from dealing with other companies when there are large scale problems but it is also fortunate that I was in a position to tell everyone what was going on and how it was being fixed - this isn't always possible.
It's worth pointing out some of the technical reasons (and I'm trying not to be very technical in my explanations mostly cause I'm not fully versed in them myself) why it takes 8 hours to put things back in place. You can't simply "un-do" a mass-move of threads in VBulletin. Even if I had direct database access and could remember how to MySQL properly (and I used to be reasonably competent with it 10+ years ago), I couldn't have rolled this error back. It's yet another on the giant list of reasons why we're breaking away from VBulletin.
So Colm and Chris (two of the Distilled Media Sys Admin team who are the true heroes of this story) downloaded that morning's backup to one of the internal machines and extracted it (that alone took about 2 hours). They then fired it up onto the Master database server and onto the slave database servers. There are numerous reasons why you simply can't just flick the switch back and say "OK, go" at that stage. Some of you may have seen we were back briefly around 7pm but had to come down again as there are some non-trivial challenges in spreading the load across all the database servers and making sure that they're all synchronised (this is why you might have seen multiple posts of the same thing by people). That and making sure the load balancers (machines that intercept all the traffic to the site and spread it out amongst all of the servers evenly to keep us running) are all behaving themselves and talking to each other correctly is no small task.
After all that is done, you need to rebuild the cache's on the database servers. This is basically a buffer of the most accessed data on the site which makes things run quickly. There's no artificial way to do this, the servers build the cache themselves as people start using the site, so whilst things seem really slow to start with, the more people using the site, the faster the cache rebuilds and the faster the site becomes.
And so that brings us back up to speed. There were a few follow on glitches like Stickies not behaving themselves which a fix was rolled out for earlier this afternoon. Declan thought it a good idea to send out the PM to anyone who would have had any posts deleted, and whilst I didn't think it necessary at first, he's absolutely spot on and it's all a part of keeping everyone informed and continuing our "honesty is the best policy" stance.
Again, I want to say thank you to all of you for the understanding and kind words during and after the outage. It was a great reminder of how lucky we are to have such an excellent membership and group of communities here on Boards.0 -
-
-
-
-
Advertisement
-
Feel so unloved right now. Good on Dav for admitting his mistake though. 
Advertisement