5 x nodes (and other updates)

regi · 23-09-2007 6:45pm #1

Hey guys,

Productive sunday - Asok and I have built a shiny new server image for our webnodes, and rebuilt nodes 1 & 5 - at long last, we're running five load balanced webnodes. It'll mean easy scaling too - so long as our database server, and NAS server is configured correctly, it'll only take about 30 minutes for each new webnode to be added.

ipvs output for the nerdy:

IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 89.234.66.107:http rr
-> 89.234.66.102:http Route 1 100 175
-> 89.234.66.101:http Route 1 93 175
-> 89.234.66.100:http Route 1 108 169
-> 89.234.66.99:http Route 1 107 182
-> 89.234.66.98:http Route 1 44 170

This is going to firmly place the bottleneck back to the database, so the next step is definately database focused. We're probably going to buy a new database server, hopefully this week. Preliminary specs are for an dual quad core machine with 16 GB of ram and a 15k RPM RAID 1+0 array serving data over the private GB network. This might even allow for proper searching once more.

AlmightyCushion · 23-09-2007 6:50pm

Considering you have rebuilt nodes 1 and 5, I assume that means that there were only three web nodes up until now and there will be five now. Would I be correct?

Terry · 23-09-2007 6:52pm

regi wrote:

This might even allow for proper searching once more.

I can see this coming back and biting you in the arse.

Thanks for all the work both of you have put in.

AlmightyCushion · 23-09-2007 6:57pm

Terry wrote:

Thanks for all the work both of you have put in.

Ditto although extra hamsters would help as well.

Winters · 23-09-2007 7:00pm

Long gone are the days of when muppet used to do it all by itself

Fair play on the work today lads.

23-09-2007 7:02pm

Yeah, fair play. Particularly since, I assume, you went into work on a Sunday.

amp · 23-09-2007 7:11pm

Brilliant! cheers guys!

RuggieBear · 23-09-2007 7:15pm

sweet.
getting search back would be awesome

regi · 23-09-2007 7:52pm

AlmightyCushion wrote:

Considering you have rebuilt nodes 1 and 5, I assume that means that there were only three web nodes up until now and there will be five now. Would I be correct?

Yep, 3 + 2 = 5

copacetic · 23-09-2007 8:01pm

regi wrote:

Hey guys,

Productive sunday - Asok and I have built a shiny new server image for our webnodes, and rebuilt nodes 1 & 5 - at long last, we're running five load balanced webnodes. It'll mean easy scaling too - so long as our database server, and NAS server is configured correctly, it'll only take about 30 minutes for each new webnode to be added.

ipvs output for the nerdy:

IP Virtual Server version 1.2.0 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 89.234.66.107:http rr
-> 89.234.66.102:http Route 1 100 175
-> 89.234.66.101:http Route 1 93 175
-> 89.234.66.100:http Route 1 108 169
-> 89.234.66.99:http Route 1 107 182
-> 89.234.66.98:http Route 1 44 170

This is going to firmly place the bottleneck back to the database, so the next step is definately database focused. We're probably going to buy a new database server, hopefully this week. Preliminary specs are for an dual quad core machine with 16 GB of ram and a 15k RPM RAID 1+0 array serving data over the private GB network. This might even allow for proper searching once more.

great work, and great possible news on search!

AlmightyCushion · 23-09-2007 8:02pm

regi wrote:

Yep, 3 + 2 = 5

Smart arse. Just wanted to make sure. Good stuff though. Those extra 2 web nodes should make a huge improvement. Also that database server looks expensive, I think I might have to start subscribing.

regi · 23-09-2007 8:05pm

Sorry

Actually, I don't think the nodes are going to help that much in terms of how the site will feel during weekday lunch-times. New DB upgrades will though.

What this has really helped is our ability to scale that side of the installation, and recover broken / add new webnodes easily.

Karoma · 23-09-2007 8:07pm

Thanks for the effort. Now, back on the wheels!

Ruu_Old · 23-09-2007 9:09pm

Thanks for the continued efforts chaps.

AlmightyCushion · 23-09-2007 9:13pm

regi wrote:

Sorry

Actually, I don't think the nodes are going to help that much in terms of how the site will feel during weekday lunch-times. New DB upgrades will though.

What this has really helped is our ability to scale that side of the installation, and recover broken / add new webnodes easily.

Oh right. Well cheers for all the hard work lads. I'm sure if we had that rep stuff that everyone was on about a while back, yourself and Asok would be getting loads right now.

Boston · 23-09-2007 9:31pm

Woot good stuff.

randylonghorn · 23-09-2007 9:48pm

Well done, guys ... even if I didn't understand half of that (as usual!)

seamus · 23-09-2007 10:37pm

regi wrote:

This is going to firmly place the bottleneck back to the database, so the next step is definately database focused. We're probably going to buy a new database server, hopefully this week. Preliminary specs are for an dual quad core machine with 16 GB of ram and a 15k RPM RAID 1+0 array serving data over the private GB network. This might even allow for proper searching once more.

:eek:

I'd like to book my alone time with this machine in advance. No disturbances please...