The story with the slowdown and its fix.

DeVore

Lo all,

I´m writing this from Spain because I wanted to give you all an update on the slowdown and whats happening.

Our best explanation of this problem is that on occasion, particularly at peak times, our servers are getting bottlenecked and then wait-state various threads (unix threads, not boards threads) pending release of machine resources (CPU and Memory).

We cant find any rhyme nor reason for the bottlenecks but its probably a combination of things including big searchs, big threads and a few of the options we have allowed.

Ok, so we knew this would happen though I must admit I am concerned that it happened so soon after we moved to the 3 server system. All the same we have prepared for this and the system is configured for us to slide in another server fairly easily.

So thats what we´re going to do, we´re going to get another server to spread the load and hopefully stop these top outs and subsequent bottlenecks and slowdowns.

Now, this is going to take time. Cloud´s missus is expecting their first child soon (yay!) and Regi has two weeks off to study for his exams. We´re going to requisition the server in the next few days, allow 5 days for staging, another day or two for setup, a day to install and a day to debug and contingency. Thats the best case scenario. So, we can expect probably 2 weeks from Tuesday to be realistic.

Sorry about the slowdowns, yet more growing pains but we´ll get past them.

DeV

aidan_walsh

Thanks for the update.

Out of interest, are the likes of adverts.ie, boards.us etc running on the same servers, and if so do you know how much of a strain they are putting on the performance of boards?

hullaballoo

Wow, another server! You guys are really spoiling us!

I was expecting searches to go again. Thanks admins!

flogen

aye, another one... if this keeps up you'll be renting out entire apartment blocks to house them by 2010...

so is the income of boards.ie sufficient to afford a new server? Maybe that idea of biggest pledge gets to name the server could be implimented to ease the cost!

CuLT

flogen wrote:

aye, another one... if this keeps up you'll be renting out entire apartment blocks to house them by 2010...

Bucephalus!

Edit: I've just realized that this reference is incredibly vague; Bucephalus is the name of a supercomputer which took over an entire apartment building as it was expanded by its owner, "Alexander", in a short story by Isaac Asimov.

The_Minister

Does a new server cost much?

Ruu_Old

A bajillion dollars like! Id be happy to donate a few squid if needed

Beruthiel

DeVore wrote:

Cloud´s missus is expecting their first child soon

Daddy Cloud will really be a Daddy!
Congrats John!

and Regi has two weeks off to study for his exams.

good luck with them Dan

jeez ye are all busy boys, hope the new server does the trick

kaimera

flogen wrote:

so is the income of boards.ie sufficient to afford a new server?

Yea, and a trip to Spain for one lucky admin :v:

Victor

What if you put AH on one sever and spread everything else around the rest?

RuggieBear

Victor wrote:

What if you put AH on one sever and spread everything else around the rest?

I'd say banning you would lighten the server load:D

aidan_walsh

Victor wrote:

What if you put AH on one sever and spread everything else around the rest?

aidan_walsh, you do not have permission to access this page. This could be due to one of several reasons:

1. Your user account may not have sufficient privileges to access this page. Are you trying to edit someone else's post, access administrative features or some other privileged system?
2. If you are trying to post, the administrator may have disabled your account, or it may be awaiting activation.
3. afterhours.boards.ie may have crashed under the weight of white noise and bebo threads. If this is the case, the administrators have been notified by email that all is well.

mike65

I just like the way Dev made a point of saying he was writing to us from Spain.

'tis nice to know you're thinking of us as we struggle on heroically.

Mike.

Talliesin

The description of the original problem says "thread deadlock" to me. Any luck investigating that?

Mimikyu

This post has been deleted.

GoneShootin

DeVore wrote:

I´m writing this from Spain

Good to see my subscription money at work!

amp

I hear that some people do work in other countries, even Spain.

Victor

amp wrote:

I hear that some people do work in other countries, even Spain.

What? During the afternoon? On Sunday?

BuffyBot

Victor wrote:

What? During the afternoon? On Sunday?

Sangria testers?

Beruthiel

BuffyBot wrote:

Sangria testers?

where do I sign up!?

John

Beruthiel wrote:

where do I sign up!?

I think it's part of a time share weekend. You all presume Dev is having fun but he's actually trying to flog a load of apartments that will be built ~2017.

CrowdedHouse

Priority Access for Subscribers - let the rest wait

DeVore

Ruu wrote:

A bajillion dollars like! Id be happy to donate a few squid if needed

They cost about 3.5k each. We are good for cash, thats because not everyone blocks our ads. See, ad impressions = money = servers = capacity = growth = users = ad impressions = etc etc
Then people wonder why I get annoyed when people talk about blocking our ads.

We don´t need money, we are self sufficent at the moment. We plan for this sort of thing. Its just another one of those useless things I do around here. <-- token smiley.

DeV.

Victor

Beruthiel wrote:

where do I sign up!?

Realise there is no vodka in sangria.

Talliesin

There was in the sangria that was, according to the broken English of the Spanish woman I was drinking it with "made better".

This was at a party we somehow ended up in when my dad was taking me for a couple of halves to teach me how to drink responsibly.

Giblet

Have you thought about cutting back on some of the features. Like mouse over thread previews, stuff that doesn't really need to be there. Also large threads do cause problems. Maybe they can be locked after a certain point and a new thread started with the same subject.

Some of the little helpers might be causing problems too, like when you type a user name into search, it will bring up a list dynamically of all the user names. Maybe have stuff like sigs off as default and they have to be turned on by the user on each post (haha). Crap like birthday notifications can surely be dropped aswell ...


All the little things might add up :S:S

Sparky

DeVore wrote:

They cost about 3.5k each. We are good for cash, thats because not everyone blocks our ads. See, ad impressions = money = servers = capacity = growth = users = ad impressions = etc etc
Then people wonder why I get annoyed when people talk about blocking our ads.

We don´t need money, we are self sufficent at the moment. We plan for this sort of thing. Its just another one of those useless things I do around here. <-- token smiley.

DeV.

I could of swore it was your poker wins that was keeping this place running

अधिनायक

Maybe ask for some help from users in your programming forum. Database & web server optimisation is often misunderstood and you likely already have members who could help you. Could save you some cash and give a better solution.

Hobbes

अधिनाय&#2325 wrote:

Maybe ask for some help from users in your programming forum. Database & web server optimisation is often misunderstood and you likely already have members who could help you. Could save you some cash and give a better solution.

We don't answer Homework questions!

Actually if I recall there is already a forum for dev-boards related stuff.
http://www.boards.ie/vbulletin/forumdisplay.php?f=228

Sparky

अधिनाय&#2325 wrote:

Maybe ask for some help from users in your programming forum. Database & web server optimisation is often misunderstood and you likely already have members who could help you. Could save you some cash and give a better solution.

Well Regi and Cloud know their stuff well, so I'd assume they went that road aswell.

अधिनायक

No disrespect but a few times I've seen this kind of conversation:

Boss: how is the optimisation going?
Technical manager: Excellent. We have optimised our internal counters related to waits latches and locks, we are getting over 99% cache hit ratio and we haven't had a deadlock in a week.
Boss: <clicks on Buy Now button - page takes a minute to load> Doesn't look so good from my end.

I don't see much participation in the dev forum. It looks more like a place where people report bugs once in a while.