Home server build - RAID questions

Serephucus

Hi all,

Thinking about a home server build over the Summer (or maybe a bit later) and a few general questions to go along with this.

The server will mainly be used for storage - there's going to be a lot. It'll also be used to backup three or four machines, as well as host a few small Minecraft servers, and maybe a website. I'm probably going to go for a low-end Ivy/Sandy Bridge setup, with a dedicated RAID card, for RAID 5. To that end:

Are Samsung Spinpoint F4 2TB drives alright to use? I've heard conflicting reports on how reliable they are in RAID (don't know why).
Is this RAID card a good one? While I'm very familiar with computer components and the like, I know virtually nothing when it comes to servers, and my networking knowledge in general is less than I'd like.
Can new drives simply be slotted in to a RAID 5 array? I'd assume the data would have to be redistributed or whatever, but I'm assuming I wouldn't have to wipe everything before-hand.

Any general comments or suggestions on this?

BaconZombie

You will need to fully wipe the disk if you want to use RAID.
I've just gotten myself a HP N40L for ~ 170e {290e - 120e cashback offer} which has a 250Gb SATA and bay for 4 x 3.5 SATA drivers which I'm going to fill with 4 x Samsung F3 2TByte drives.
This is working out alot cheaper then a NAS or custom build unit.

Also just from a Security view point hosting a website let a loan a Minecraft Server on your "Backup" device / NAS is a REALLY bad idea if you are planning on saving anything you care about on it.

Also once again RAID is not a backup!!!
Save backups to a completely different and offsite if possible tape/HD/PrintOut/etc....

Serephucus wrote: »

Hi all,

Thinking about a home server build over the Summer (or maybe a bit later) and a few general questions to go along with this.

The server will mainly be used for storage - there's going to be a lot. It'll also be used to backup three or four machines, as well as host a few small Minecraft servers, and maybe a website. I'm probably going to go for a low-end Ivy/Sandy Bridge setup, with a dedicated RAID card, for RAID 5. To that end:

Are Samsung Spinpoint F4 2TB drives alright to use? I've heard conflicting reports on how reliable they are in RAID (don't know why).
Is this RAID card a good one? While I'm very familiar with computer components and the like, I know virtually nothing when it comes to servers, and my networking knowledge in general is less than I'd like.
Can new drives simply be slotted in to a RAID 5 array? I'd assume the data would have to be redistributed or whatever, but I'm assuming I wouldn't have to wipe everything before-hand.

Any general comments or suggestions on this?

Serephucus

Surely the fact that you can replace a drive when one fails and retrive all your data (assuming another drive doesn't die during the rebuild, which I'd be ok with risking) means that it's never going to be lost?

I'd need more than 8TB I'm afraid, otherwise I'd have got something like that. If I end up going with a separate backup solution, I might get something like that though.

duckysauce

Serephucus wrote: »

Surely the fact that you can replace a drive when one fails and retrive all your data (assuming another drive doesn't die during the rebuild, which I'd be ok with risking) means that it's never going to be lost?

lol data corruption ,bad raid striping , do not assume anything , power failure corrupt disks , etc :pac:

Sparks

Wouldn't use RAID5 myself either, not unless it was a pretty good hardware RAID controller; and honestly, you could probably spend the money you'd use on one of those on hard drives and have an equivalent sized RAID10 array using software RAID (and not have to worry about what happens to the array if your controller dies and you can't get a replacement for whatever reason). Which is what I did, if that's any help to you.

vibe666

i wouldn't use any more than 4x 2tb disks with raid5 personally, you're asking for trouble.

your chances of an unrecoverable read error on any of the remaining drives increases as the drive size and number of drives increases and at that point you are stuck unless you can get enough storage together to copy everything off the degraded array and start again which is going to be a pretty daunting prospect with 8tb or more of data.

I have 8x 2tb spinpoint f3's in my NAS, which is a HP ML110 G6 with 8gb of ram. i have freenas 8 running inside ESXi v5 with the disks mapped via RDM to the freenas VM running alongside a few other VM's in ESXi.

the disks are in a single raid6 volume (or rather the freenas equiv. with dual parity) which gives me a total of about 10.7tb of usable storage.

one of the handy things about freenas is the ability to import disks from another freenas installation very quickly, so even if my server dies completely, i could hook all the disks up to something else and get it back up and running again right away without having to worry about raid controllers or incompatible hardware.

what have i learnt from the experience?

ram ram and more ram!

freenas8 with that many disks needs a bare minimum of 8gb of usable ram before it will even get out of bed, so a 16gb upgrade is going to be coming as soon as i have the funds.

same goes for the cpu and i'll be swapping out the 2.8ghz G6950 for a nice X3480 quad core xeon the first chance i get.

ESX is awesome! but it is my job, so i'm not sure what the learning curve would be like if you weren't familiar with it.

freenas is awesome as well, altho v8 is a long way removed from the old freenas7 which you could run on any old hardware. if you want v8 and you do if you plan on using a lot of big disks, then you'll need to spec. the hardware accordingly.

if you plan on ending up with something along the lines of the size of my nas then you can save a heap of money over a pre-built prosumer/smb nas from the likes of qnap or synology, but what you have in money you will get back in headaches, so it will probably come down to what is more important to you, your free time and sanity, or your money.

good luck anyway and don't be afraid to ask questions, we're here to help.

Serephucus

I've heard similar things RE RAID 5 on other forums as well. I can almost guarantee I'd need more than 8TB in (5.5, or whatever is is usable) in a RAID 5, so that's probably out.

I've heard FreeNAS mentioned before. I'm assuming it's hardware RAID, but using your computer's hardware, vs. a dedicated card? If I use FreeNAS, that means I can only use the server as a RAID volume, basically, right? (And can't use it for web hosting, or Minecraft servers, like I'd planned) Or am I missing something obvious here?

I had thought on RAID 10 briefly, but... the amount of drives that would require. It makes sense in a lot of ways - I could have everything, inc. backup, all in one machine. It's just expensive. I figured RAID 5 might be a cheaper option. Once you factor in the dedicated card, and the fact that you're limited to 4-6 drives or so, not so good any more.

Apologies for the rant. I'm reasonably well up on computer hardware and modding, but I know virtually nothing of servers, networking, etc.

vibe666

when people talk about hardware RAID, they mean not just a cheap raid card with some ports on it, but with a dedicated hardware chip to do all the raid calculations. a lot of low end raid cards are still software raid and will use your system resources to do a lot of the calculation work.

tbh, if you're going to go down the cheap raid card route, you'd be as well to just present the disks individually to freenas rather than have the card manage them as you know that no matter what happens, you just need to build a new freenas box and plug in all the disks to get it back. with a cheap raid card, if it fails, you can't guarantee that you can find another identical one or that even if you do, that it will be able to recover your raid volume(s).

when freenas 8.2 comes out (hopefully very soon, beta3 is already out), there will be a webserver included and more of the features from freenas 7 should be working, such as a torrent client, itunes and upnp media server etc. so it will be a lot more useful.

for what you are looking to do (i.e. a heap of big consumer grade disks) i think freenas8 with RAID-Z2 (similar to raid6 with dual parity) is going to be your best low budget option.

i'll assume you're pretty much like me with a big media collection which you want easy access to and don't want to lose, but could (theoretically) be replaced by re-downloading stuff again if you had to, plus some personal data that you want to keep safe no matter what happens.

for that, i trust everything in my media collection to the dual parity on the nas, BUT i also mirror any essential files to a separate 1tb disk in another computer with docs etc. syncing from there to dropbox, family pics going to picasa and home video's to youtube and vimeo and that is enough to keep my mind at ease.

Serephucus

That's exactly what I was thinking - much easier to just download FreeNAS on a new machine than to go trawling eBay for an outdated RAID card if something happens.

And yeah, I do have a fairly massive media collection, and yeah, most of it could be re-downloaded, but obviously I want to keep it backed up locally or I wouldn't be here. :P

At present, I have my OS on a 128GB drive, and my games on a 1TB one. I could backup that lot to a 1.5TB drive or something, then trust everything else to RAID...

I'm actually quite liking this idea, except that I wouldn't be able to run Minecraft servers from the FreeNAS machine, and that's one of the big things here. I play MC a fair bit with friends, and it's a really attractive option to have all the servers we use always-on. It's not the end of the world if I can't get that - I can just run them locally as I'm doing now - but it would be nice to have.

Can I have JBOD and RAID drives in the one machine with FreeNAS? (The 1.5TB backup for my OS/games)

In the middle of exams at the minute, so I don't have the time to devote to reading up on this - hence the questions - but I'll be seriously giving this a look come next week. Thanks for the help so far!

vibe666

yep, you can have several volumes on freenas, inc. a mixture of raid and jbod.

if you wanted to, you could do what i have, run freenas on ESXi as a virtual machine and then have another VM dedicated as a minecraft server running MineOS side by side on the same physical server.

obviously you're going to have a steeper learning curve adding ESXi to the mix, but it would allow you to do what you want to do without compromising anything.

Serephucus

Looks complicated, I'll give you that, and more than what I was thinking of, but potentially very cool. I'll have a read up on a lot of this and post back.

Sparks

Serephucus wrote: »

I had thought on RAID 10 briefly, but... the amount of drives that would require. It makes sense in a lot of ways - I could have everything, inc. backup, all in one machine. It's just expensive. I figured RAID 5 might be a cheaper option. Once you factor in the dedicated card, and the fact that you're limited to 4-6 drives or so, not so good any more.

That's the killer with RAID 5. It works... but unless you have a good dedicated RAID card, the performance suffers because you have to do more work to generate the parity. With RAID 10/1+0/0+1/howeverYouWantToDoIt, you're not generating parity on the fly and you can manage RAID in software, at least for small installations (think SOHO). There's another advantage in software RAID, in that if the server motherboard dies, you just move the drives to another motherboard, fire up mdadm and rebuild the array and lose no data; but if your hardware RAID5 controller goes, you will probably have to buy another of the same model; reconstructing a RAID5 array from one controller to another can sometimes be finicky. And given how long RAID arrays can wind up being used for, that might happen after that card has gone out of production.

That's not to say that a RAID5 array with a hosed controller is always unrecoverable, you understand; most manufacturers of RAID hardware cards aren't dumb and build in forward and backward compatability for this very reason; it's just that it's not a case of plugging the disks into any old controller and going on like nothing happened.

Also, be sure you don't use fakeRAID controllers (which are usually advertised as on-motherboard RAID, but which are really just a BIOS extension to a SATA controller to let it do RAID in a software driver). They have all the disadvantages of both hardware and software RAID with none of the benefits. The driver eats CPU resources (and it's higher in the OS stack so it's more heavyweight than true software RAID); and the whole arrangement has even more weak links and per-manufacturer lock-ins than hardware RAID.


Honestly, I'm just not a fan of RAID5

vibe666

that's actually one of the caveats with freenas8 and RAID-Z2, you're effectively doing the equiv. to RAID6 parity calculations on the fly, so you need a beefy enough CPU and a generous heap of RAM to do the job without taking a performance hit.

that said, if you have both then you'll have the best of everything as far as compatibility and expandability goes.

Serephucus

That's something that I was thinking of earlier. What makes RAID-Z2 any better than RAID 6? Surely you're still dealing with the possibility of a read error in the event of a rebuild?

Edit: Forget that, Google+Wiki=Win.

vibe666

RAID6 & RAID-Z2 are functionally the same thing, using dual parity instead of the single parity used in RAID5, so you can suffer the complete or partial loss of up to two disks and still be able to recover your data, whereas RAID5 can only function with the loss of one disk, that is assuming you don't have an unrecoverable read error on any of the other disks during a rebuild.

Serephucus

Yeah, that much I knew. I more meant if you had a failure on a RAID Z array, why would it be any better than failure on a RAID 5 array (because they're the same) but Wiki helped there. Not sure I completely get it, but it writes and stores parity differently, so it rebuilds it differently. (I think)

vibe666

basically, you'll have your raid array humming along nicely until a disk fails and you need to replace it.

the array will still function before and during the rebuild of the failed disk, but to rebuild a failed disk in a raid5 array, every single bit of every other disk needs to be read in order for the raid controller to work out what should be in every single bit of the failed disk.

if however during that process the controller encounters a bit that it can't read on any of the disks, it *could* fail the rebuild entirely, leaving you stuck having to copy everything off the degraded array to some other storage and start again from scratch.

with RAID-Z2 and RAID6, you have a second set of parity, so if one of the disks has an unrecoverable read error, the controller can look at the 2nd parity bit to get the required information to continue.

in reality the way it works is a little more complicated than that, but that's pretty much the basics of it.

Serephucus

Ah, thanks. Makes sense.

RAID Z2 is the front-runner at the moment. The only negative with it is if I ever need to expand the array. Don't think I have enough spare storage to move from 10-14TB or whatever it ends up being. :P

vibe666

yeah, you really need one of the pre-built prosumer NAS's for that, or just spec your freenas box high enough that you won't need to for the foreseeable future.

my first proper nas was a thecus n7700 seven bay jobbie, but buggy firmware and terrible support put me off them for life.

if i was buying a pre-built nas now, it would either be synology or qnap (leaning more towards synology) but you're talking about serious money for the amount of storage you want to use.

that said, i could currently copy all my current 6.4tb of data to 3 or 4x 3tb disks temporarily if i was going to expand my storage needs, or plug in a 2nd 8 port sata controller and newer, bigger disks (say 3tb or 4tb) to the same box, copy everything over and then remove the old 2tb disks once it was finished, before installing the new ones in their place an closing it back up again, then sell the disks and controller on again to recoup some of the cost.

Serephucus

Yeah. The pre-built ones are nice, but once you go above 4-port ones, you really get shafted on price.

I won't be building this until the end of the Summer at the earliest. Going away for the few months anyway, and waiting for HDD prices to come back down a bit, then I'll probably just go and buy six 2TB drives to pair with my existing two or something. If I do this, I probably will go with the future-proofing mentality.

ronkmonster

vibe666 wrote: »

yeah, you really need one of the pre-built prosumer NAS's for that, or just spec your freenas box high enough that you won't need to for the foreseeable future.

my first proper nas was a thecus n7700 seven bay jobbie, but buggy firmware and terrible support put me off them for life.

if i was buying a pre-built nas now, it would either be synology or qnap (leaning more towards synology) but you're talking about serious money for the amount of storage you want to use.

that said, i could currently copy all my current 6.4tb of data to 3 or 4x 3tb disks temporarily if i was going to expand my storage needs, or plug in a 2nd 8 port sata controller and newer, bigger disks (say 3tb or 4tb) to the same box, copy everything over and then remove the old 2tb disks once it was finished, before installing the new ones in their place an closing it back up again, then sell the disks and controller on again to recoup some of the cost.

What benefits are there to raid for your setup?
Personally I just use each drive stand alone. If server crashes I can still access each drive individually. Can raid do that?

Serephucus

The benefit of RAID is that you can keep all your data. If a drive fails in JBOD, your data's gone. If it fails in RAID, everything's still there. (and yes, it can still be accessed)

@vibe, had a quick look on Newegg, and 3TB drives are actually pretty affordable now. The same $/GB ratio as Samsung's (or should I say Seagate's) 2TB F4s. Out of interest, how long have you had your server running, and have you had any failures? Just curious as to the general life-span of drives when run in a server, etc.

ronkmonster

Serephucus wrote: »

The benefit of RAID is that you can keep all your data. If a drive fails in JBOD, your data's gone. If it fails in RAID, everything's still there. (and yes, it can still be accessed)

@vibe, had a quick look on Newegg, and 3TB drives are actually pretty affordable now. The same $/GB ratio as Samsung's (or should I say Seagate's) 2TB F4s. Out of interest, how long have you had your server running, and have you had any failures? Just curious as to the general life-span of drives when run in a server, etc.

Yes but what about over standalone drives. I don't use JBOD either.
I don't see the benefit when I have multiple backups anyway.

carbsy

vibe666 wrote: »

i have freenas 8 running inside ESXi v5 with the disks mapped via RDM to the freenas VM running alongside a few other VM's in ESXi.

.

Ayup vibe! Where are you storing the actual VMs, on an SSD or the 250GB drive that came with the unit?

I ended up using my 110 G6 as a nested ESXi v5 study lab which it does quite well with 9GB ram.I currently building a new rig to run a full lab in Workstation 8 so once again my 110 G6 will be free to configure from scratch!

vibe666

ronkmonster wrote: »

What benefits are there to raid for your setup?
Personally I just use each drive stand alone. If server crashes I can still access each drive individually. Can raid do that?

well for one, my server is a vm, not a physical server so if it crashes, i could rebuild a new one and reconnect the drives in software in a few minutes and be back in business right away. in reality, i already have a clone of my freenas vm anyway, so it would be even quicker than that, just re-attaching the disks in software.

as for the RAID, i could have 2 out of my 8 drives fail completely and still have access to all my data over the network whilst i replace the failed drives with no downtime at all.

ronkmonster wrote: »

Yes but what about over standalone drives. I don't use JBOD either.
I don't see the benefit when I have multiple backups anyway.

the benefits over standalone drives are more speed and multiple redundancy.

how do you keep multiple backups? do you make a copy of each drive onto another drive or backup to tape? either way, that would get very expensive for multiple terabytes of data.

carbsy wrote: »

Ayup vibe! Where are you storing the actual VMs, on an SSD or the 250GB drive that came with the unit?

I ended up using my 110 G6 as a nested ESXi v5 study lab which it does quite well with 9GB ram.I currently building a new rig to run a full lab in Workstation 8 so once again my 110 G6 will be free to configure from scratch!

my spare ssd ended up getting re-homed in the wifes netbook in the end so the freenas vm and my iso's etc. are stored on the 250gb drive the server came with and the rest of my vm's are stored on an iSCSI LUN running inside freenas as i was starting to run out of space for vm storage on the 250gb drive so i had to move some elsewhere.

if i ever restart esxi it's a minor pain in the backside having to manually start the freenas vm first and then re-attach the LUN in esx before i can get to the other vm's, but it's been on 24/7 now since july last year when i built it aside from 2 days when i moved house in december and everything is running smoothly so far, touch wood.

ronkmonster

vibe666 wrote: »

well for one, my server is a vm, not a physical server so if it crashes, i could rebuild a new one and reconnect the drives in software in a few minutes and be back in business right away. in reality, i already have a clone of my freenas vm anyway, so it would be even quicker than that, just re-attaching the disks in software.

Cloning gives me the same benefits and I've had a server start to fail before that I just moved to another cpu+mobo and linux just booted.

vibe666 wrote: »

as for the RAID, i could have 2 out of my 8 drives fail completely and still have access to all my data over the network whilst i replace the failed drives with no downtime at all.

the benefits over standalone drives are more speed and multiple redundancy.

Speed isn't too important to me as it's mainly file storage and I'd prefer to have all my storage space.

vibe666 wrote: »

how do you keep multiple backups? do you make a copy of each drive onto another drive or backup to tape? either way, that would get very expensive for multiple terabytes of data.

Some data isn't important (tv series, game discs) so I don't protect. The stuff I do backup, I backup to external drive(with plans for another nas box just for backups), cloud backups and bluray backups done monthly depending on how important the data is. Bluray discs are stored at another house. I wouldn't use raid as a reason not to backup, a lot of the stuff I want to keep compresses a good bit.
I also replace my discs every 2/3 years regardless of health (put off for a while due to price increases). Then I use the older discs for other PCs or monthly backups/nas boxes.

Sparks

Serephucus wrote: »

The benefit of RAID is that you can keep all your data.

That's not really a benefit of RAID; it's why you have backups (yes, even when you have RAID).

RAID's purpose is to protect your uptime rather than your data per se. RAID is not a replacement for backups!

(There's also a performance bonus for RAID under certain circumstances, if your controller is smart enough, because you can pull data from several spindles simultaenously; but that's really a side benefit than a design goal)

vibe666

ronkmonster wrote: »

Speed isn't too important to me as it's mainly file storage and I'd prefer to have all my storage space.

do you mind me asking how much storage do you actually have and how much of it is used, versus free space?

my data is currently around 6.4tb and growing by around 1tb per year so far and i have 10.7tb of usable storage in my NAS so i've got around 4 years at my current level of usage until i need to upgrade it.

i still have my old 7 bay nas and i plan to use that as secondary storage (most likely 7x 3tb disks RAID6 = </> 14tb of usable storage) when disk prices drop back to normal, so i'm pretty much covered as far as storage goes for now.

ronkmonster wrote: »

Some data isn't important (tv series, game discs) so I don't protect. The stuff I do backup, I backup to external drive(with plans for another nas box just for backups), cloud backups and bluray backups done monthly depending on how important the data is. Bluray discs are stored at another house. I wouldn't use raid as a reason not to backup, a lot of the stuff I want to keep compresses a good bit.
I also replace my discs every 2/3 years regardless of health (put off for a while due to price increases). Then I use the older discs for other PCs or monthly backups/nas boxes.

i don't use RAID as a reason not to back up, i am in a similar position to yourself, anything important is backed up to a drive on another PC and to various cloud storage from there.

if i lost a bunch of movies or tv shows or mp3's it wouldn't be the end of the world, but it would be a pain in the hole, so i'm reducing the chances of losing data through the failure of a disk (well, 2 disks now).

when you have a disk fail (and you will sooner or later, it happens to us all) you will have lost everything on that disk that wasn't backed up, so potentially 2-3tb of movies or tv shows or whatever will be gone.

sure you can re-download, borrow from friends etc. but having had it happen to me more than once it's a pain in the hole i'd rather avoid if i can and if the price i pay is the cost of a couple of disks (which were £58+vat for 2tb f4's when i got them from overclockers before the thai floods) its worth it for me.

vibe666

Sparks wrote: »

That's not really a benefit of RAID; it's why you have backups (yes, even when you have RAID).

RAID's purpose is to protect your uptime rather than your data per se. RAID is not a replacement for backups!

yes very much so in an smb or enterprise environment, but the cost of backing up large amounts of data is prohibitively expensive for a consumer compared to just buying a couple of extra disks to use for parity in a RAID config to offset the chances of losing data due to a disk failure.

there are zero practical options for backing up multiple terabytes of data that don't cost more than the price of the disks i'd be backing up and it just isn't worth it, so non-critical stuff is protected from single (or double) disk failures and anything critical has a copy on the nas, a copy on another machine and a copy in the cloud wherever it fits, be it dropbox, youtube, picasa or whatever and that more than meets my needs so far.

Sparks

vibe666 wrote: »

yes very much so in an smb or enterprise environment, but the cost of backing up large amounts of data is prohibitively expensive for a consumer compared to just buying a couple of extra disks to use for parity in a RAID config to offset the chances of losing data due to a disk failure.

The problem is that consumers buy disks for their RAID arrays at the same time; so the disks are usually from the same batch, and statistically, are more likely to fail close together than disks from different batches.

In other words, Joe Average's RAID is more likely to get a double failure than Average, Inc.

It's all probabilities, but that's the same deal as with backups...

ronkmonster

vibe666 wrote: »

do you mind me asking how much storage do you actually have and how much of it is used, versus free space?

my data is currently around 6.4tb and growing by around 1tb per year so far and i have 10.7tb of usable storage in my NAS so i've got around 4 years at my current level of usage until i need to upgrade it.

File Server 5.5TB (40%to 50% but a few drives are close to full)
NAS 2 Bay 4TB used for tv+documentaries (maybe half full)
Media Box 2TB (half full)
NAS 2 Bay 500GB (not used at the moment)
External 1TB connected to netbook server 80% full

File server was fairly full until i got the 2x2tb for nas box.

vibe666 wrote: »

when you have a disk fail (and you will sooner or later, it happens to us all) you will have lost everything on that disk that wasn't backed up, so potentially 2-3tb of movies or tv shows or whatever will be gone.

sure you can re-download, borrow from friends etc. but having had it happen to me more than once it's a pain in the hole i'd rather avoid if i can and if the price i pay is the cost of a couple of disks (which were £58+vat for 2tb f4's when i got them from overclockers before the thai floods) its worth it for me.

Sure, and it's happened before so I try replace every 2 years. I had started buying my new disks until the floods happened. The old disks are still usable just not relied on as main storage. When I replace the 5x1TB in my file server, I'll buy some cheap nas boxes for them and they can store backups. Don't need to be on all the time so their live can be extended.