Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie
Building a Home Server
Options
-
20-07-2011 11:10am#1
Comments
-
-
-
-
-
-
Advertisement
-
-
-
-
-
-
Advertisement
-
i wouldn't touch any kind of tape backup system at this stage tbh. they're overkill for home use (imho) as well as slow, awkward and expensive.
a blu-ray writer is still going to use a fcuktonne of discs and is going to get really expensive over time depending on your backup schedule and how long you want to keep them for.
with hard disk prices dropping almost daily, i see no reason not to backup straight to disk.
if you're seriously talking about offsite redundancy of any kind surely disk backups fit the bill better than tapes anyway? if you're going to ship tapes offsite then why not a USB or removable caddy mounted disk(s)?
also, don't forget that with (local) disk based backups anything after the initial backup is going to be incremental unless you're going to worry about multiple file versions, but i think leaving that at the primary storage level is going to be more than adequate, even if you do implement it.
there's plenty of ways to mitigate your potential losses too using cloud based options if you group your data by type and level of importance. family pictures can all go up to picasa, important docs can go up to google docs or dropbox etc, video's bulk uploaded to youtube or vimeo (or similar, pick your poison as with most) and most of these jobs can at least be semi-automated with the right tools and again, everything after the first batch will be incremental.
lets be honest though, if (assuming you're anything like me
) you're backing up more than a couple of terabytes of data as a home user, a lot of that data isn't going to be 100% vital stuff that can't be replaced with a combination of re-downloading and borrowing from mates etc. so any offsite backup needs can *probably* be taken care of with the above cloud based options and a single or dual disk backup.
as for RAID5 or RAID6 on your storage, did you read this last year? http://storagemojo.com/2010/02/27/does-raid-6-stops-working-in-2019/
maybe i'm paranoid, but it scared the bejesus out of me.
my current set up is a Thecus N7700 NAS (which i really wouldn't recommend in hindsight, awful fecking yoke) with 5x 1.5tb samsung spinpoints in RAID5 (approx 600gb free atm) which I'm backing up to an external eSata Startech SAT3540ER2 with 4x 2tb spinpoints in RAID5.
as with the above suggestion, anything drastically important is also mirrored to a local 1tb disk in my desktop PC as well as being backed up online as well in one form or another but for the most part, looking at it in the cold hard light of day, the majority of my data isn't irreplaceable, so i'm not all that worried about it tbh and if my house burns down (knowing that the 100% vital stuff is still backed up online anyway) my multiple linux distro and old (out of copyright) movie & music collection will certainly be right down the bottom of my list of priorities.
the next step in my own data race is replacing the NAS with this bad boy which i recently put together and is a HP ML110 G6 with 8x 2tb HDD's crammed in it (and a 250gb HDD on the top left wedged in with a piece of cardboard for anyone with eagle eyes).
it's currently got ESXi v4.1 booting off a 4gb USB stick with the intention that i'll use an openfiler VM to handle all those 2tb drives to manage all my storage requirements and move it all off the NAS, but i've been procrastinating about the best way to actually present the drives to openfiler, so it's not quite completed yet.
anyway, i might well be having more trouble than you are figuring out what to do with it all and i still haven't figured out how i'm going to back it all up, although, i suspect it will end up being a modified version of what's happening now.
ho hum, it's bedtime anyway. 0 -
the next step in my own data race is replacing the NAS with this bad boy which i recently put together and is a HP ML110 G6 with 8x 2tb HDD's crammed in it (and a 250gb HDD on the top left wedged in with a piece of cardboard for anyone with eagle eyes).
nice setup you have there ... 'my eagle eyes' cannot see the cooling fans or air vents for drives !!0 -
it actually runs pretty cool as it is. the case fan the server comes with could double up as the main fan for a hovercraft at full tilt, i'm actually afraid i'm going to find a cat stuck to the front of the box one day if it walks past it too close.nice setup you have there ... 'my eagle eyes' cannot see the cooling fans or air vents for drives !!
having said that, it never runs at full speed (probably not even 30% most of the time) and you'd know it if it was, just by comparing it to the jet engine noise at POST and there is plenty of ventilation at the front for both drive bays.
also, the immediate problem is that HP have kindly fitted 6 pin fan headers on the motherboard making it a little difficult to find extra fans.
0 -
I don't really agree with that in general, but I have to agree at the moment - DAT drives are grand if you only have a few Gb to back up (anything up to 70Gb or so is managable), but the next step up seems to be LTO-5 and the drives are way too expensive right now. But tapes and disks have a habit of leapfrogging each other - one year tape's cheaper, the next disk is. And they're not slow, not if you have them on the right kind of bus, nor awkward if you have your backup procedure and software set up right.i wouldn't touch any kind of tape backup system at this stage tbh. they're overkill for home use (imho) as well as slow, awkward and expensive.
I can give you a fewwith hard disk prices dropping almost daily, i see no reason not to backup straight to disk.- Disks are not as reliable as tapes.
- Disks are larger and heavier than tapes and not as physically robust as tapes.
- Disks are therefore a pain in the tuckus to take off-site to rotate backups. And the harder a backup is to do, the more likely it is you'll skip it...
- Disk failure modes aren't the same as tape failure modes. Leave a tape sit in its case in a cool dry place for a few years, it's ready to go the moment you take it out of the drawer. With disks, it might be ready to go; or you might have a component failure on startup. Tapes have no active parts in the cassette, so you don't have that problem.
Because this is a home system. Sticking a dat tape in the drive on a sunday night and putting the tape that was in there into your case and bringing it to work the next day and locking it in the drawer at work is a perfectly acceptable offsite backup solution for home computers. Combine that with Amanda and you have a full home network backup solution.if you're seriously talking about offsite redundancy of any kind surely disk backups fit the bill better than tapes anyway? if you're going to ship tapes offsite then why not a USB or removable caddy mounted disk(s)?
Doing that with a USB/caddy-ied hard drive is not as easy, because it's more expensive, more fragile, heavier and bulkier. It can be done; but the easier it is, the more likely it is you'll do it.
Any kind of backup (unless you're doing reverse deltas) will be incremental; but if you're not doing a regular image (say, once a month at least) then your backups will be easy to do and strictly speaking complete; but your recoveries will be like trying to reassemble the cow from a hamburger. If you have one image from two years ago, and incrementals every night since then and you want a file created last week....also, don't forget that with (local) disk based backups anything after the initial backup is going to be incremental unless you're going to worry about multiple file versions, but i think leaving that at the primary storage level is going to be more than adequate, even if you do implement it.
Er, no.there's plenty of ways to mitigate your potential losses too using cloud based options if you group your data by type and level of importance. family pictures can all go up to picasa, important docs can go up to google docs or dropbox etc, video's bulk uploaded to youtube or vimeo (or similar, pick your poison as with most) and most of these jobs can at least be semi-automated with the right tools and again, everything after the first batch will be incremental.
If you have to sign EULAs that give away content rights, you aren't signing up to a backup system. Picasa is not a backup system for your family photos, for example, because all your files could be lost in an instant if your account was suspended (and there are more than enough tales of that happening in error that I wouldn't trust it with any file I didn't have a backup of).
That's not to say that there aren't cloud-based backup options, like S3 or other online backup services (and Amanda now works with some of these natively, which makes things simpler). It's just that an online backup is not the same thing as a social media site.
No, that's paranoiaas for RAID5 or RAID6 on your storage, did you read this last year? http://storagemojo.com/2010/02/27/does-raid-6-stops-working-in-2019/
maybe i'm paranoid, but it scared the bejesus out of me.
However, there are real concerns with RAID 5/6:- RAID 5/6 requires generation of parity on the fly, which requires more computation than RAID1/0/10 do; not a problem if you're using hardware RAID, but that carries its own warning stickers. For software RAID, RAID5/6 means trading storage space for performance.
- RAID 5 carries a risk in that hard drives have a tendency to fail in batches - so when one of your raid drives goes, the others may be very near their end of life as well. If one of them goes bang while you're rebuilding the drive that went bang first, you lose the entire array. And with 2Tb+ drives, rebuilding that first drive takes a while, increasing the size of the window where another failure is a critical one. RAID 6 alleviates this somewhat, but it's only reducing the probability, not eliminating the problem. With RAID 1/10 systems, you don't need to rebuild a drive, just mirror one of the other drives in the array with a straight /bin/dd copy; far faster, far safer. You do still have the risk of failure, but it's far lower because you're not thrashing the drive with seeks and reads, just doing one long continuous read.
What happens if you have a house fire? You lose your primary and your backup in one go. Same if you have a burglary, or any of the other disaster scenarios you have to plan for with a home network setup.my current set up is a Thecus N7700 NAS (which i really wouldn't recommend in hindsight, awful fecking yoke) with 5x 1.5tb samsung spinpoints in RAID5 (approx 600gb free atm) which I'm backing up to an external eSata Startech SAT3540ER2 with 4x 2tb spinpoints in RAID5.
If you can live with that, then yeah, go for it.if my house burns down (knowing that the 100% vital stuff is still backed up online anyway) my multiple linux distro and old (out of copyright) movie & music collection will certainly be right down the bottom of my list of priorities.
Personally, I prefer my way - it means I don't have to explain to herself indoors that her MSc thesis is lost forever along with the wedding photos and video and my Phd work and so on.
Just from interest, my current setup is a home server that acts as a NAS and a media server and seedbox, and will also act as the main Amanda backup server, a print server, and anything else I fire at it. 4x2Tb drives in a RAID10 array for the main storage array, 2x500Gb drives in a RAID1 array for the OS and local user files, all in a standard tower case with an overspecc'd PSU and room to expand as needed, with firewire and other handy interfaces because it's a normal (if older) desktop motherboard/CPU. Definitely a bit of overkill in some parts of the build, but for a good reason - some things will come down in price, like memory and hard drives, but others like the case and PSU won't; and my whole goal with the box was to build something I wouldn't have to spend much time on after it was set up. So far, it's running like a charm, and once it's fully installed, I'll write a bit more about it and do some benchmarking.0 -
-
damn you linux geeks and your well thought out logic and reason!I don't really agree with that in general, but I have to agree at the moment - DAT drives are grand if you only have a few Gb to back up (anything up to 70Gb or so is managable), but the next step up seems to be LTO-5 and the drives are way too expensive right now. But tapes and disks have a habit of leapfrogging each other - one year tape's cheaper, the next disk is. And they're not slow, not if you have them on the right kind of bus, nor awkward if you have your backup procedure and software set up right.
but you do have the initial set up costs to think about and a large number of tapes to store and manage which is still going to be either very expensive or take up a large amount of space somewhere, depending on what capacity tapes/drive you go for.I can give you a few- Disks are not as reliable as tapes.
- Disks are larger and heavier than tapes and not as physically robust as tapes.
- Disks are therefore a pain in the tuckus to take off-site to rotate backups. And the harder a backup is to do, the more likely it is you'll skip it...
- Disk failure modes aren't the same as tape failure modes. Leave a tape sit in its case in a cool dry place for a few years, it's ready to go the moment you take it out of the drawer. With disks, it might be ready to go; or you might have a component failure on startup. Tapes have no active parts in the cassette, so you don't have that problem.
are you sure this is a home system??? seriously, most people think i'm batty for having as much storage as i do, but you're adding a whole new level to it.Because this is a home system. Sticking a dat tape in the drive on a sunday night and putting the tape that was in there into your case and bringing it to work the next day and locking it in the drawer at work is a perfectly acceptable offsite backup solution for home computers. Combine that with Amanda and you have a full home network backup solution.
Doing that with a USB/caddy-ied hard drive is not as easy, because it's more expensive, more fragile, heavier and bulkier. It can be done; but the easier it is, the more likely it is you'll do it.
what's wrong with doing one full backup, then a weeks worth of incrementals, then consolidating those into one full backup (one of the joys of disk over tape backups) and then offsite that, then continue with the incremental backups from there for the next week, lather, rinse & repeat for a month and then wipe & re-use the oldest after a month?Any kind of backup (unless you're doing reverse deltas) will be incremental; but if you're not doing a regular image (say, once a month at least) then your backups will be easy to do and strictly speaking complete; but your recoveries will be like trying to reassemble the cow from a hamburger. If you have one image from two years ago, and incrementals every night since then and you want a file created last week....
that's the thing, you DO have a backup of them, unless whichever service you choose decides to cancel your account on the exact same day your house burns down and the chances of that happening are pretty slim i think.Er, no.
If you have to sign EULAs that give away content rights, you aren't signing up to a backup system. Picasa is not a backup system for your family photos, for example, because all your files could be lost in an instant if your account was suspended (and there are more than enough tales of that happening in error that I wouldn't trust it with any file I didn't have a backup of).
i was just using a couple of examples off the top of my head, there's plenty of options.That's not to say that there aren't cloud-based backup options, like S3 or other online backup services (and Amanda now works with some of these natively, which makes things simpler). It's just that an online backup is not the same thing as a social media site.
again though, we're talking about home systems so you have to decide at what point is it all just overkill?No, that's paranoia
However, there are real concerns with RAID 5/6:- RAID 5/6 requires generation of parity on the fly, which requires more computation than RAID1/0/10 do; not a problem if you're using hardware RAID, but that carries its own warning stickers. For software RAID, RAID5/6 means trading storage space for performance.
- RAID 5 carries a risk in that hard drives have a tendency to fail in batches - so when one of your raid drives goes, the others may be very near their end of life as well. If one of them goes bang while you're rebuilding the drive that went bang first, you lose the entire array. And with 2Tb+ drives, rebuilding that first drive takes a while, increasing the size of the window where another failure is a critical one. RAID 6 alleviates this somewhat, but it's only reducing the probability, not eliminating the problem. With RAID 1/10 systems, you don't need to rebuild a drive, just mirror one of the other drives in the array with a straight /bin/dd copy; far faster, far safer. You do still have the risk of failure, but it's far lower because you're not thrashing the drive with seeks and reads, just doing one long continuous read.
but do you? i'd be willing to bet my socks that most home users idea of keeping backups is at most a single USB drive (if anything at all).What happens if you have a house fire? You lose your primary and your backup in one go. Same if you have a burglary, or any of the other disaster scenarios you have to plan for with a home network setup.
aside from your data, what else in your home do you have backed up at all? if your house did burn down and you lost all your worldly possessions and everything you've collected over the years, if you were standing in front of the smouldering ruins and you turned to your wife and said "it's okay love, i have an offsite backup of the wedding pics and your MSc" would she really be all that bothered?If you can live with that, then yeah, go for it.
Personally, I prefer my way - it means I don't have to explain to herself indoors that her MSc thesis is lost forever along with the wedding photos and video and my Phd work and so on.
i think we might have a race on!Just from interest, my current setup is a home server that acts as a NAS and a media server and seedbox, and will also act as the main Amanda backup server, a print server, and anything else I fire at it. 4x2Tb drives in a RAID10 array for the main storage array, 2x500Gb drives in a RAID1 array for the OS and local user files, all in a standard tower case with an overspecc'd PSU and room to expand as needed, with firewire and other handy interfaces because it's a normal (if older) desktop motherboard/CPU. Definitely a bit of overkill in some parts of the build, but for a good reason - some things will come down in price, like memory and hard drives, but others like the case and PSU won't; and my whole goal with the box was to build something I wouldn't have to spend much time on after it was set up. So far, it's running like a charm, and once it's fully installed, I'll write a bit more about it and do some benchmarking. 0 -
Well, if a DAT tape will suffice, you're not looking at a huge outlay really - around the €300 level or so and tapes are about €30 each and you'd want to have at least three of those. So you're paying less for the tape backup system than you are for the RAID drive. Now LTO-5, yeah, there it's a wallop - somewhere north of €3000 right now. But that will come down - DAT used to cost that much...but you do have the initial set up costs to think about and a large number of tapes to store and manage which is still going to be either very expensive or take up a large amount of space somewhere, depending on what capacity tapes/drive you go for.
I might be an odd case today, but give it a few years and this won't be enough for your average user...are you sure this is a home system??? seriously, most people think i'm batty for having as much storage as i do, but you're adding a whole new level to it.
Nothing...what's wrong with doing one full backup, then a weeks worth of incrementals, then consolidating those into one full backup (one of the joys of disk over tape backups) and then offsite that, then continue with the incremental backups from there for the next week, lather, rinse & repeat for a month and then wipe & re-use the oldest after a month?
...if you do it.
But the image/incremental work should all be automated and then it doesn't matter if it's tape or disk (you do have amanda handling all this for you, right?); so the only considerations are things like cost, bulk, robustness and so on.
To quote a google sysadmin, "Hope is not an acceptable plan"that's the thing, you DO have a backup of them, unless whichever service you choose decides to cancel your account on the exact same day your house burns down and the chances of that happening are pretty slim i think.
Besides which, you have copyright and privacy issues. Say you "backup" your photos of your kids to flickr or some similar media site and miss the bit in the EULA where it says you grant them copyright on the image; and then someone else swipes a copy of it and posts it somewhere on the net as some sort of I can haz cheezburger meme (and that's the least stomach-churning abuse of such an image I can come up with in polite company).
And yes, it sounds like paranoia...
...but when you have to sign an EULA, it ceases to be paranoia and becomes an understanding of your legal rights and protections.
Yup.again though, we're talking about home systems so you have to decide at what point is it all just overkill?
Try doing it after explaining to your wife that the hard drive which held the only copy of the wedding video and the wedding photos is the one making that odd grinding noise and producing all the smoke.
You think explaining a failed backup to your boss is tough?
Well, it's definitely true that if you don't do the backups, you won't have the backups....but do you? i'd be willing to bet my socks that most home users idea of keeping backups is at most a single USB drive (if anything at all).
For physical possessions, you have house insurance.aside from your data, what else in your home do you have backed up at all? if your house did burn down and you lost all your worldly possessions and everything you've collected over the years, if you were standing in front of the smouldering ruins and you turned to your wife and said "it's okay love, i have an offsite backup of the wedding pics and your MSc" would she really be all that bothered?
For data, you have backups.
To a house fire?i think we might have a race on! 0 -
-
Yeah...Online backups ftw
...if you have Magnet FTTH or UPC 100Mb/s lines.
They're a bit more painful if you're on the lower speed ADSL lines, and nearly unusable if you have only dialup.
I do have some files backed up to a server in germany (not with amanda, just with an rsync cronjob... so technically not really a backup so much as a second copy) but the first rsync run took almost a week to backup one directory (granted, a 19Gb directory with all our photos, but still).
Still, I may need to bite the bullet on this one and use Amanda's S3 or other backup driver; LTO tapes were just outside the budget and even DAT-320 wasn't up to the job 0 -
-
Advertisement
-
You're basicly doing backup-by-NAS then. Which is fine, if you don't mind paying for the extra space you can't use and you don't mind your backup being in someone elses house with no control over it.
In other words, yeah, it'd do the job, but I'd prefer something else personally
(Example: After a year, to save space, you scan all your bank statements, credit card statements, mortgage stuff, etc, etc, and shred the originals and archive the scans. Would you really be happy with someone else having that data on a server they control physically? If so, cool.)0 -
You're basicly doing backup-by-NAS then. Which is fine, if you don't mind paying for the extra space you can't use and you don't mind your backup being in someone elses house with no control over it.
In other words, yeah, it'd do the job, but I'd prefer something else personally
(Example: After a year, to save space, you scan all your bank statements, credit card statements, mortgage stuff, etc, etc, and shred the originals and archive the scans. Would you really be happy with someone else having that data on a server they control physically? If so, cool.)
Not just that, even though I agree with what you said, you also have no control over the availability of your data that is stored outside your network. Companies die on a daily basis and several ones that initially looked very good went down and took their customers' data with them. In a lot of cases that data was gone while in some, mostly after going to court or threatening with legal action, data was made available again. Your data (even the backups) not being availabe can break your back as well.0 -
-
-
-
-
Advertisement