Building a 10TB backup server

colm_c

Hi All,

I'm thinking about building a 10TB backup server for work, our current one is almost full and I'd like to backup more than just our critical data (maybe user's desktops/documents as well).

Here's what we need, the only critical piece is that it needs to have 10TB+ of fault tolerant (meaning RAID) disk space. And needs to be able to run CentOS.

Looking at sourcing the case basic parts from http://www.xcase.co.uk/ - they've got a 24 bay hot swap 5u enclosure, which looks ideal. http://www.xcase.co.uk/p/293332/

The question I have is around RAID cards and hard drives - any recommendation? - I was thinking RAID 6 but I don't think a single card will support 24 drives? Also where would I get 24 500GB drives cheap? Komplett is charging less than 100 euro each but 24 of those is still 2,400.

Any thoughts? other options?

Before anyone mentions offsite backup, we're in the process of setting up an S3 backup solution which will be a replica of our internal backup machine, when this is up, it will also be a copy of this machine.

daveharnett

Well, dabs4work.ie will sell you 500Gb sata drives for €70 ex VAT. 10tb redundant storage isn't going to come cheap though. Controllers for that kind of a setup will cost a good chunk (as i understand it, you'll need multiple controllers that talk to each other directly).

Reckon that most buisnesses that need that kind of backup capacity will buy a blade for it and let hp/whoever worry about setting it up. Might be worth considering since it'll cost a bundle either way.

Regards,
Captain Buzzkill

seamus

If you're going for that kind of capacity, have a look at SANs. Not cheap, but a much more flexible solution than a server with a big RAID backend.

10-10-20

I have to ask this - what are you backing up that takes 10TB? Should you not be looking at a tape solution? Does this data have to be online at all times? Can you not archive?

I've never seen anything as odd as that Chenbro server! I reckon that the RAID card you'll need to operate 24 disks is going to cost as much as the disks themselves!

colm_c

daveharnett wrote: »

Well, dabs4work.ie will sell you 500Gb sata drives for €70 ex VAT. 10tb redundant storage isn't going to come cheap though. Controllers for that kind of a setup will cost a good chunk (as i understand it, you'll need multiple controllers that talk to each other directly).

Reckon that most buisnesses that need that kind of backup capacity will buy a blade for it and let hp/whoever worry about setting it up. Might be worth considering since it'll cost a bundle either way.

Regards,
Captain Buzzkill

Blades are more suited to high density comuting rather than storage, infact most new blades don't have any storage on them - they use a SAN instead.

seamus wrote: »

If you're going for that kind of capacity, have a look at SANs. Not cheap, but a much more flexible solution than a server with a big RAID backend.

Looked at SANs, very, very expensive for an SME.

10-10-20 wrote: »

I have to ask this - what are you backing up that takes 10TB? Should you not be looking at a tape solution? Does this data have to be online at all times? Can you not archive?

I've never seen anything as odd as that Chenbro server! I reckon that the RAID card you'll need to operate 24 disks is going to cost as much as the disks themselves!

We're currently only backing up mission critical data of current projects, which for a 3 days of backup is just under 1TB in size.

Most of the data being backed up will be for projects, development environments, users email and specified local folders, the local folders and email will take up the biggest chunk of the backup.

The last time I looked at tapes, anything that would backup 1TB (a day's worth of backup) was very pricey.

Any other ideas welcome.

seamus

Take a look at your email and local folders size before you start throwing money at the problem - why are they so big?

No company wants to force people to maintain their inboxes, but the amount of stuff that people will store is staggering. There are people who will store every single mail they ever receive and will use email for sending anything whatsoever. With the exception of legal requirements, none of your users should be storing any emails over 3 years old. They should also not be storing large attachments in email.

I implemented exchange in one place for about 200 users. Every user's old inbox got dumped to PST and everyone got a brand new fresh inbox. Within three weeks, the mail file was at 25GB - around 130MB per user.

There were two problems - some users were using their mailbox as permanent storage - transferring 5 and 6 year-old emails back from the PST into their mailbox. The second problem was the size of attachments. Sending 10MB emails to their entire team was common. A good few people attempted to send > 50MB emails to 20 or 30 people. Despite the fact that I had provided 500GB of empty network space for that kind of stuff.

This is where the political side of IT comes in and it's not a nice thing to have to do, but one thing is certain - if you provide the space, they will fill it. So upgrading your infrastructure is fire fighting, not fire prevention (urgh, I hate that marketing speak). One thing I've always maintained in terms of data security is that I.T. should only be responsible for ensuring that the network storage is there and that it's backed up. You mention backing up desktops - why? Have the users save all their data to the network and just back that up. Let them know that if their desktop goes *kaput* and they lose their data, tough ****.

Scan the user's folders for images, movies and music. I guarantee you'll find GB's of the stuff. Issue a notice, then delete all of it (except for business stuff of course :eek:).

Of course, you may very well have looked at all of these things and you still need space.

In that case, perhaps some form of NAS may be appropriate - you don't specifically need a server that does anything but takes files.

Tip - if you need a way of convincing bosses/managers about the need to implement data storage/retention policies, then just do some generous sums on the cost of backing up and storing frivilous user data like old emails, video and music. Mention cost to management and they'll swing to your side

colm_c

I take the point about email - some people have emails in their inboxes since 2001 here!

We won't be going down the exchange route, as we have a very mixed environment - Macs, PCs and some linux too, and since most of the guys are on the road and are only in the office maybe once a month - we use a hosted email service (for sending and receiving) and all mailboxes are currently local.

Besides, I'm not trusting microsoft with the company email.

Ideally I would like to limit email box sizes, but enforcing it could be difficult, I guess if I was backing it up I could run a script to check the size and tell people to reduce their mailbox.

There's a whole bunch of file type that are excluded from backups but we use video and audio files in some of our projects so it's hard to know what's personal and what's not.

I think I'll have to do some calculations based on a revised limit and see where that goes.

DublinWriter

colm_c wrote: »

The last time I looked at tapes, anything that would backup 1TB (a day's worth of backup) was very pricey.

Well, how much is your data worth?

If it's mission critical as you say, they you should go to the directors with a proposal.

I'm not sure if they would be appropriate, but have a look at the Buffalo 1Tb/2Tb NAS units. They're fairly cheap for what they do - RAID5, networkable and can run backup-jobs on their own.

You'd also need to start enforcing quotas on mail accounts.

zod

use MSA60 (http://h18006.www1.hp.com/storage/disk_storage/msa_diskarrays/drive_enclosures/msa60/qa.html ) or 70's .. you can use SAS or SATA disks, the enclosure themselves are relatively inexpensive and with a P800 controller you get great performance with the expandability of up to 73TB

irlrobins

colm_c wrote: »

I take the point about email - some people have emails in their inboxes since 2001 here!

That's one place you're going wrong, there's little chance that an email from 2001 is useful or required in 2007.

colm_c wrote: »

Besides, I'm not trusting microsoft with the company email.

A very blinkered opinion in my view

colm_c wrote: »

Ideally I would like to limit email box sizes, but enforcing it could be difficult, I guess if I was backing it up I could run a script to check the size and tell people to reduce their mailbox.

In my firm our mailbox limit (including sent items, etc) is 60MB. Low by many other firms' policies. But it forces people to go through their mails and delete useless ones and to archive or place required mails in their personal folders. We also can't send any mail until we bring the mailbox below 60Mb. A very effective method I find.

colm_c wrote: »

so it's hard to know what's personal and what's not.

I agree, sometimes it's not possible to use a blanket filter. But you should occasionally examine the files and see if someone has transferred their itunes on to their desktop. A quick delete is usually a deterrent from a repeat offence.

BostonB

Our general policy is 50mb store and anything older than 6 months is archived. Anything older than 2yrs is usually deleted. No attachments bigger than 2mb, unless you get it approved beforehand. Users require more storage than this can get special permission to do so. But still within limits. Using PSTs is discouraged.

We sweep PC's and the network for files like ACC, MP3's DivX etc that are not work related. If you find some of these files it usually is worth doing a more detailed scan of that machine in person. As only one or two users with excess files like this can consume huge amounts of network and storage resources.

zod

dell now doing 1TB Sata disks :eek:

so you could create 10TB of storage in a single disk enclosure using RAID 6 .. 2 drives could fail before total failure .. I'd imagine you'd get a good discount on the web price also

PowerVault MD1000 Qty 1
External Storage Array,SAS and SATA support

Catalog Number: 510293 MD10001
Base External Storage Array,SAS and SATA support 68866 [210-16199] 1
Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 8
Shipping Documents Multi-lingual documentation (with 2 Rack Cords included) 61686 [450-11004] 21
2nd Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 23
Standard Warranty 1Yr Next Business Day Warranty 61633 [688-10565][688-10040] 29
Enhanced Service Packs 3 Years Gold Service (Direct access to Dell experts 24x7) 61637 [680-11001][680-15069] 30
Installation Services No installation 61638 [683-11930] 32
4th Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 51
5th Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 52
6th Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 53
3rd Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 54
7th Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 71
8th Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 72
Rack Mounting Rails No Rack Rails 68786 [770-10625] 88
15th Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 95
9th Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 102
11th Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 201
12th Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 202
13th Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 203
14th Hard Drive 1TB, SATA, 3.5-inch, 7200 rpm Hard Drive 127327 [400-14054] 204
Enclosure Management Module (EMM) Two Enclosure Management Modules (EMM) 69209 [440-10966] 310
Front Bezel PV MD1000 Bezel Assembly 68585 [350-10153] 669

10-10-20

RAID-6 is to be supported in their PERC6 which is due out shortly.

zod

10-10-20 wrote: »

RAID-6 is to be supported in their PERC6 which is due out shortly.

I see .. HP have had raid 6 ( think they called it advanced data guard ADG ) for a while thought dell would have had it by now

Jackz

What is the web price for that zod?

joePC

I would go with www.encom.ie for a 10TB server - They use the 3ware RAID controllers which have excellent Linux support with full hardware RAID 0,1,5,6,10,50,Single Disk,JBOD -

Also depending on the drive sizes I would split the drives up into separate RAID sets.

Good luck...

Anti

No offence to the lads posting here. But i think the OP would be best of seeking professional advice. They can come and see what you have now, and advise you on whats the next step. Sllof the above option are viable, but i personally think you should get a company in to look. Alot of them do free evaluations these days.

colm_c

Funnily enough we've got two companies coming in next week to give us some options and quotes.

zod

Jackz wrote: »

What is the web price for that zod?

think about 14K

Cryos

Colm_c i can sympathise with you and your setup at the moment, we at the moment are modernising IT for one of our Large Irish Customers.

Ill just have to point out a few things to you and this is from a vast amount of experience with working with Large File servers (at the moment we operate upto 15 Clustered File Servers attached to a central SAN).

The main problem you are going to have with the solution you are going for is redundancy; In the case that a main component fails on your server you have no access to the data untill you repair the main component.

A Properly setup and maintained SAN storage system is invaluable, as the hardware for the san is optimised for speed and read/write ability. The major advantage of having a SAN based solution (this is in a non clustered setup with 1 stand alone server) is that should your main server go down, you can slap a fibre card into another pc/server stick in your fibre connection and hey presto your data is there rename the server to the old broke server and jobs a goodin.

Colm_c wrote:

We won't be going down the exchange route, as we have a very mixed environment - Macs, PCs and some linux too, and since most of the guys are on the road and are only in the office maybe once a month - we use a hosted email service (for sending and receiving) and all mailboxes are currently local.

Besides, I'm not trusting microsoft with the company email.

Some of your main problems are with your email and probably the storage of non essential personal music/videos. I like irlrobins would disagree with what you have said in relation to exchange, current versions of exchange (2003/2007) are incredibly reliable; You also have to remember that exchange is one of the most trusted Corporate and SME solutions around used by: HP, Dell, Siemens, Sony etc...

Your email limits can be set within 2 clicks! So can retention policies, you can filter the size and extention of each of the attachements comming in so that your file server isnt clogged with useless data.

Exchange has the OWA front end which works on anything that has a web browser (also last i checked you could connect to exchange both on a Mac and Linux).

For your remote Windows users you could do a few things:

Configure a Server for RRAS (2 minute job)
Setup Exchange over HTTPS (which alows outlook 2003+ to download email via https protocals from anywhere).


There are far worse packages out there for Groupware / email, Lotus Domino being one of them.

I would also agree with BostonB that it only takes a few users to consume many GB of free space on your server, case in point we had someone who had a 4 GB live inbox (Lotus Domino Setup), which was also Replicated to a Sync'd Personal Network share. The user didnt need all the emails, he even stated that he never deleted any of them since the day he started in that company.


I think personally what you need is a SAN and exchange setup, that will cure most of your IT Administration issues!.

But i would also agree, you do need to get a professional outfit in.

Khannie

If you're using centos, or any flavour of linux, you should look at linux kernel level softraid. It isolates you from being stuck with a single controller vendor (i.e. if one of your controllers goes tits up, just replace it with one from your local shop and you're back up and running, instead of having to hunt for an exact part).

There is a small performance hit v's hardware raid, but the benefit FAR outweights that cost IMO.

I'm using it for raid 1 and raid 5 and performance is superb (especially for reads, which is what you'll generally want). I'm not familiar with raid 6 tbh, so not sure if the kernel supports it.

edit: ok, raid 6 = raid 5 + extra parity. Grand. And a quick google shows that the kernel does support it. You should check that with the version of CentOs you're using though before deployment. If an alternate OS is possible, Debian's considered solid and does come with a kernel that supports Raid 6. Of course, you could compile your own slightly modified CentOs kernel, but long term, it would be a pain in the hole.

final edit: Good article on the benefits of kernel level raid here.