So what on earth did I do?

DeVore

Yep... unfortunately a double hardware failure cost us another day. However we've had several months where we werent off-line for 1 second
Swings and roundabouts.



Anyway, it was totally Dav. Totally!

woodyg

DeVore wrote: »

Yep... unfortunately a double hardware failure cost us another day. However we've had several months where we werent off-line for 1 second
Swings and roundabouts.



Anyway, it was totally Dav. Totally!

Double host server fail or DB server fail?
I take it you guys use a DB clustering setup for load balancing\redundancy

Purely Nerd interest here no way would i be bashing the Admins as i am one myself so know the pain!
We had a corrupt DB Monday in a travel agent client was not a fun day

cython

woodyg wrote: »

Double host server fail or DB server fail?
I take it you guys use a DB clustering setup for load balancing\redundancy

Purely Nerd interest here no way would i be bashing the Admins as i am one myself so know the pain!
We had a corrupt DB Monday in a travel agent client was not a fun day

I would be the same, but just as a matter of interest (primarily from dealing with servers in HA setups in my own line of work) twitter posts made reference to having to install the OS on the new disks, while other posts on here have mentioned DB data being on the failed RAID as well - Assuming these currently all reside on the same RAID, has any thought been given to putting these on separate physical arrays? I only suggest it as while it obviously wouldn't have prevented the DB outage when it arose, it could potentially have reduced the recovery/rebuild time if only either the OS or the DB had to be reinstated, as opposed to both.

Now perhaps both components already reside on separate RAIDs, and as such the suggestion is redundant (see what I did there? :pac:), or perhaps the cost of this wouldn't be feasible, but it's just an observation of a means that is employed in places to minimise the recovery time, which can be similarly important to trying to prevent the outage entirely.

DeVore

As I understand it (which is "not very well at all")... two HD's in the RAID controller failed which is kinda weird but I guess...

As for the suggestion, I'm sure the guys will look at it (we definitely suggest stuff from the community in our meetings and the techs of course read this thread themselves!).

Implementing change can be tricky though, its like saying "your car needs new tyres" and that might well be very true, but when its going 90 mph and rarely stops for pitstops and you cant be sure the new tyres will work, but there is no going back to the old ones and even 2 days is a long LONG outage well.... its easy to see how sometimes this sort of work might get long fingered (which aint optimal, but IS human )

But yeah, we need to keep improving the bulletproof-ness of the architecture.... *gulp*.

. two HD's in the RAID controller failed

OUCH!!

woodyg

cheers for the info Dev as i said earlier was purely Nerd interest.
if was a RAID crash due to 2 drives failing i will assume the storage is RAID 5?
a look into RAID 6 or what we use for DB's RAID 10 might be an idea
there's a new version of RAID 6 which doesn't have the write perf issue as before but gives a better redundancy than 10

2 drives failing at the same time is rotten luck as the odds are pretty massive for it to happen.
wouldnt have fancied being the Admin that discovered that issue!

Czarcasm

woodyg wrote: »

if was a RAID crash due to 2 drives failing i will assume the storage is RAID 5?

From another thread woody -

Dav wrote: »

One of the database slaves had a major failure with it's hard disks. Before anyone asks, yes they were in RAID (1 to be exact), but both disks failed. It's rare that your redundancy fails at the same time as the main device, but not unheard of.

Pawwed Rig

Pawwed Rig wrote: »

I would agree with you if boards was going down regularly for prolonged periods because of issues similar to DavGate. However this is the first time in a long time where I could not access the site for a few hours so I don't think a mountain should be made out of a molehill here. I am sure that they can tell advertisers that the site is available 99.99% of the time.

99.99%? So under an hour of downtime a year? Not quite sure about that.

kaimera

Q for Dav and the lads/girls:

Since the distilled media buyout of boards, has there been money for infrastructure and stuff available? Ignore if I'm prying too much of course.

I remember the subs coming online to keep boards ticking over at the time!

dr.bollocko

You could look to the community to help fund new things too when needed.
The whole subscriber idea has been overdue an overhaul for a while now.

Implement a system where people can gift people subscriber status for a nominal fee. Similar to reddit gold if anyone's on there.

It'd be like a way of super thanking a comment. Giver feels good, receiver feels good, community benefits. How bad.

Gordon

[url=https://us.v-cdn.net/6034073/uploads/attachments/3841/160830.jpg[/img][/url]

IRLConor

kaimera wrote: »

Since the distilled media buyout of boards, has there been money for infrastructure and stuff available? Ignore if I'm prying too much of course.

After the first round of investment by Distilled Media Boards went from having one person on staff (Ross) to four (Ross, me, Dav and Darragh) so yeah, Distilled Media brought some money with them.

Steve

dr.bollocko wrote: »

You could look to the community to help fund new things too when needed.
The whole subscriber idea has been overdue an overhaul for a while now.

Implement a system where people can gift people subscriber status for a nominal fee. Similar to reddit gold if anyone's on there.

It'd be like a way of super thanking a comment. Giver feels good, receiver feels good, community benefits. How bad.

Would you like a larger shovel for that hole you're digging?

only €5

seamus

dr.bollocko wrote: »

Implement a system where people can gift people subscriber status for a nominal fee. Similar to reddit gold if anyone's on there.

I just don't think it will work here. It's not in the Irish culture.

Reddit gold is effectively a form of tipping. Which is heavily ingrained in US culture, and everybody gets tipped for everything. Reddit gold being a "thanks for that great comment".

We don't really have the same thing going on, either on boards or in life. You could certainly add it, but I imagine you'll be lucky to get a handful of people per week using it. There's also a scale thing going on.

http://blog.reddit.com/2013/12/top-posts-of-2013-stats-and-snoo-years.html

Last year, between all posts and all comments, there were 440 million posts on reddit. Last year. That's 11 times the size of boards, in a single year.

So even if 0.01% of posts were gilded, that $1.76m income for reddit, last year.

Though if you look at the gold page; http://www.reddit.com/gold/about
It looks appealing. So maybe that's worth looking at for boards. Revamp the subscription package; partner with some Irish businesses, add enhancements to boards that are only available to subscribers, etc.

I don't see Irish people spending €5 on an internet stranger, but they might spend it on themselves.

DeVore

IRLConor wrote: »

After the first round of investment by Distilled Media Boards went from having one person on staff (Ross) to four (Ross, me, Dav and Darragh) so yeah, Distilled Media brought some money with them.

Not to mention the 3 community managers