Advertisement
If you have a new account but are having problems posting or verifying your account, please email us on hello@boards.ie for help. Thanks :)
Hello all! Please ensure that you are posting a new thread or question in the appropriate forum. The Feedback forum is overwhelmed with questions that are having to be moved elsewhere. If you need help to verify your account contact hello@boards.ie
Hi all! We have been experiencing an issue on site where threads have been missing the latest postings. The platform host Vanilla are working on this issue. A workaround that has been used by some is to navigate back from 1 to 10+ pages to re-sync the thread and this will then show the latest posts. Thanks, Mike.
Hi there,
There is an issue with role permissions that is being worked on at the moment.
If you are having trouble with access or permissions on regional forums please post here to get access: https://www.boards.ie/discussion/2058365403/you-do-not-have-permission-for-that#latest

RAID backup

  • 01-04-2010 8:58pm
    #1
    Registered Users, Registered Users 2 Posts: 1,435 ✭✭✭


    hi all,

    I'm not feeling overly confident in my current multidrive software driven backup process and am looking for something more reliable. I'll probably end up with some type of RAID system, only thing is I know next to nothing about them.

    What I want is something relatively fast, quiet, small footprint (needs to fit on my desk)...stylish if possible. Probably only need 2 bays...but maybe 4. WIFI would be cool. Needs to be Mac compatible. Not looking to spend the earth on this..was thinking around eur500 but would up that if needed.

    Any recommendations or tips for things I should look out for?

    BTW - what is "NAS"?


Comments

  • Closed Accounts Posts: 4,930 ✭✭✭duckysauce


    network area storage


  • Registered Users, Registered Users 2 Posts: 2,262 ✭✭✭stcstc


    RAID of any kind is NOT a backup, its redundancy. ie if a disc fails you dont loose anything, in theory


    If you want to do it properly, so it actually will recover, avoid some of the consumer brands and buy something with 4 drives in it

    another option might be to buy a cheap PC and stick some big HD's in it and put it somewhere else in the house as a server? and connect over the network to it

    then over time you could add a proper backup system of say a tape drive of some kind


  • Closed Accounts Posts: 4,930 ✭✭✭duckysauce


    why 4 drives? for raid you only need two or three depending on the fault tolerance you are going for?


  • Registered Users, Registered Users 2 Posts: 1,435 ✭✭✭eas


    thanks Steve - can redundancy become back up?

    As in, I have two bays with 500gb each - if I mirror the drives theoretically they fill up together at the same rate, once full I remove drive one from bay 1, replace it with formatted drive from bay 2 then replace that with a new drive. Full drive that was in bay 1 is stashed away for safe keeping? If a drive fails during the process I replace & mirror the other drive and continue on?


  • Registered Users, Registered Users 2 Posts: 607 ✭✭✭brianwalshcork


    duckysauce wrote: »
    why 4 drives? for raid you only need two or three depending on the fault tolerance you are going for?

    He might want to do 2x RAID 1, or a RAID 5 array with a hot spare.

    OP, even if you have a RAID system, you still need a decent back up system. A RAID array will protect your from disk failures, but if you delete a file or folder of files, then they are still gone. This is a real problem if you don't realise that you deleted them. You also have to remember that if you get a NAS, files you delete won't go into your recycle bin - they're simply deleted.

    If you have a decent internet connection, you should look some of the online back up services. Depending on your volumes of data, they may work out cheaper than backing up to tape, and definitely less hassle than backing up to DVD.
    Google Amazon S3, and it's competitors and see which one suits you best.

    If you don't think it's worth the money to back up to tape, or to an online storage service, then you just haven't lost enough data yet :eek:

    Brian


  • Advertisement
  • Registered Users, Registered Users 2 Posts: 6,196 ✭✭✭PaulieC


    eas wrote: »
    BTW - what is "NAS"?

    Network Attached Storage

    It's basically a hard disk that connects to your router that you can connect to from anywhere


  • Registered Users, Registered Users 2 Posts: 607 ✭✭✭brianwalshcork


    eas wrote: »
    thanks Steve - can redundancy become back up?

    As in, I have two bays with 500gb each - if I mirror the drives theoretically they fill up together at the same rate, once full I remove drive one from bay 1, replace it with formatted drive from bay 2 then replace that with a new drive. Full drive that was in bay 1 is stashed away for safe keeping? If a drive fails during the process I replace & mirror the other drive and continue on?

    Won't work I'm afraid - for a number of reasons.... firstly, since you'll have 500 GB drives, when they fill up, you'll have two 500 GB drives with the same data. If you remove one... the RAID 1 (mirrored) array becomes degraded... and you're system will tell you that a disk has failed.

    So now, if you stick in another 500 GB drive, the array will get rebuilt... you'll have 3 drives with 500 GB of data, and you're still out of disk space. You could stick in a 1TB drive, and the array would either get rebuilt onto 500 GB of it and waste teh other space, or it will insist on the two drives being identical.

    Additonally, you won't be able to do much with the drive that you removed! It's not a normal disk that you can stick into another system and access the files. In theory, if you had the same raid controller, and drivers versions etc. in a spare system, (or even the original system) you could put the drive you removed into that, and it might be recognised as part of a RAID array, the system might boot & complain, and may be acecssible - but I would exactly describe this as a backup, or at least not a backup that would let me sleep easily at night!


  • Registered Users, Registered Users 2 Posts: 2,262 ✭✭✭stcstc


    why 4 drives - well ok 3 for raid 5, BUT i aint seen a 3 drive system

    what i was trying to suggest dont get caught with one of these 2 drive chassis, cos generally they dont really offer much protection

    eas

    the issue with your way is that when you put in the new drive it will just get filled back up with whats on the system. and then when you take one drive and put it in the safe, you still have a single point of failure: ie that one drive, if it fails you aint got it anywhere else

    the way to do it is on cyclic tape or online backups


  • Registered Users, Registered Users 2 Posts: 202 ✭✭ozymandius


    So you just want more safer storage? Some kind of RAID device would suit all right. But it's not backup. If you delete a file it's gone.

    But first some background and then off to Wikipedia with you for more.

    RAID = Redundant Array of Inexpensive Drives. The idea (historically) was that you bung a load of drives in a box and treat it as one big drive. With the advantage that if one (or more) drives fail the whole thing keeps chuging along. Replacement disks would then get rebuilt in the background. This can get very resiliant and very expensive.

    RAID 1 - two drives mirrored, if one fails, the data is safe. But only half the capacity of the two drives is available.

    RAID5 - 3 or more drives. Data is spread over all of them. Can survive the failure of one drive. But now 66% available space, 5 drives 80%, etc. But more drives means more failure points.

    Lots of other configurations: RAID 6 10 50 60 ...

    Typically cheaper units have three or four drives.

    Then how do you connect to the RAID device? DAS, NAS or SAN?

    DAS = Direct attached storage. Just like your USB external drive. Or eSATA or SCSI.

    NAS = Network attached storage. Now the RAID drive is mounted as a network device. In Windows etc. it will appear as a network resoource, could also be connected using other network protocols - Apple's, NFS (linux), even ftp, etc.

    SAN - Storage area network. Getting into the heavy metal. The RAID drive is connected by its own separate network, but appears as a local device to the OS. This used to be the preserve of big budget corporations, but now with iSCSI it's a real alternative to a NAS.

    Many of these allow you to take snapshots, say daily or weekly. That could give you some semblance of backup. If you delete something you could recover it easily. Big boxes allow you to add storage as you need it too.

    Have you looked at the Drobo range?

    Another idea is to roll your own. Try FreeNAS - it's really good if you have the hardware skills.

    Check out pixmania or dabs and see what they have in your price range. I've used Thecus devices at the budget end, Transtec at the middle price range.


  • Registered Users, Registered Users 2 Posts: 2,262 ✭✭✭stcstc


    i can tell you from personal experience that raid 5 nas is a cool thing BUT dont depend on it though

    i now have a windows server with an lto tape drive

    but used to only run a raid 5 nas, the actual nas failed and had to have the data recovered from it, it cost me 2.5K for a data recovery specialist to recover it

    thats why you still need some kind of proper backup


  • Advertisement
  • Registered Users, Registered Users 2 Posts: 202 ✭✭ozymandius


    Absolutely true. A mini-SAN I put together recently has two identical big RAID5 rack units synched with each other -with auto-failover. But takling care of each potential weakness costs muchly


  • Registered Users Posts: 168 ✭✭php-fox


    may be this http://www.drobo.com/ ?


  • Closed Accounts Posts: 222 ✭✭stunt_penguin


    Drobos as cool as hell, and come highly recommended from a lot of professionals . They're RAID that you don't have to worry about, as you can hot swap disks and even mix and match disk brands etc, so you could take your existing drives and use those.


  • Registered Users, Registered Users 2 Posts: 1,435 ✭✭✭eas


    thanks for all the input folks, it all makes sense.

    Drobo - yes, I've considered this option. It gets mixed reviews, negatives mainly down to speed. I decided not to go this route as it didn't do what I wanted - but as it turns out what I wanted won't work with RAID either. So maybe a drobo + tape/online backup is the best solution.


  • Banned (with Prison Access) Posts: 1,822 ✭✭✭Ballyman


    Sorry for sidetracking here but I'd like to get a little advice as well on what I currently do.

    I have two 500GB external drives. One is where all photos (Original & Post Processed) are stored (A: Drive) and the other is a synchronoised back up of it. I don't use the HD on my laptop for storage at all. Both drives are USB connected to my laptop whenever I am at my desk and I only upload photos to A: and synch the drives every night.

    This works fine for me at the moment. However once they are full I had intended to store one of them at home and the other offsite. I will then buy two 1TB drives and do the same again.

    Now I know it's not ideal as both A and snych'd drives are both in the same location at the same time but I can't justify the expense of tapes or online storage etc. Also my broadband speed/quality isn't good enough for online storage.
    Is what I am doing stupid? Wrong? Can you recommend anything better?


  • Registered Users, Registered Users 2 Posts: 1,302 ✭✭✭Heebie


    As some have said RAID is not a "backup" solution.. it is a redundancy method.

    There seems to be some debate about RAID type.

    RAID = (R)edundant (A)rray of (I)nexpensive (D)isks

    Raid type 0 actually has no redundancy, it just "stripes" information across multiple drives. If you lose one drive in the array. you lose all your data.

    Raid type 1 is "mirror" Where it keeps one disk as an identical copy of the other, and will run off the secondary if the primary dies.

    Raid type 3 - I have no idea what it does.. I've never used it.

    Raid type 4 - this keeps a disk of "parity" information that can be used to compensate for individual disk failures. With "double parity" (using two parity drives instead of one.) you get increased security. It is generally easy to resize a RAID4 array by adding drives. With double-parity you can lose up to one data drive, and one parity drive, before you begin to lose data. "ideal" layout are 8 data drives and 2 parity drives in a set, as this optimises how much data you get into a set with best resilience. Most true SAN solutions use this, or a custom version of this.

    Raid type 5 - this interweaves the parity onto the data disks instead of using dedicated parity disks. Ideally this uses 9 drives. (which will effectively make 1 drive parity.) In theory you can lose up to 3 drives of a 9-drive RAID 5 array and still replace them and get the data to rebuild. (I would not put it to the test in production.. I have seen data remain intact with the loss of 2 drives out of 6.. but was very happy that client was keeping daily tape backups of his data.)

    If you can afford a server case, halfway decent motherboard, and an absolute minimum of 3 hard disks.. and preferably a halfway decent hardware RAID controller, you should be able to make a halfway decent fileserver that can act similarly to a SAN.

    You can get off-the-shelf solutions for this from Dell, NetApp, and many others. Most off-the-shelf solutions that are "consumer" targeted use either RAID 0 or RAID 1, and you can usually CHOOSE which to use.. and most people will opt for the RAID 0 option because you "get to use all the space" but..that's defeating the purpose.. if you get a 1 or 2 terabyte drive that attaches via USB2 or FireWire.. make sure you configure it to mirror (RAID 1) so it only gives you half it's maximum capacity if you're looking for data resilience.

    Using RAID of any level, however, is **NOT** really a solution for backup. If backup is what you want, get yourself a tape drive and some tapes, preferably with tapes that will hold as much as you want to back up. (LTO = 100GB/200GB, LTO2=200GB/400GB, LTO3=400GB/800GB, LTO4=800GB/1.6TB [where first number is uncompressed storage and the second is the maximum compressed storage.. with compression on your actual usage will be somewhere in between the two numbers.. not the higher number.) If your needs are greater than the maximum tape capacity you can get.. hire a consultant to design, implement and manage your backups. [this is also not a bad idea if you aren't a technical person who understands backups well.])

    Depending on how much you want to rely on multiple tapes, you can decide how to do your backups.

    You should keep the written-to backup tapes in a separate location from where you have the "original" data, and it should be in a somewhat environmentally controlled location. (not too humid, not too hot, not too cold.)

    Failing keeping your own tapes.. using one or two online backup services that are in different locations can work, provided that you have enough bandwidth into your office to use it. (most photography labs do not tend to have nearly enough bandwidth to do backups this way, since it's outbound traffic, and most tend to use asymmetric links that are fast coming into the office, but slow going out, such as aDSL.)

    If you come up with a backup regime.. stick to it regularly, and regularly do test restorations of random files and compare them against the originals to make sure they are not modified/damaged. The day you skip doing your backups will be the day you need them.. and the day you need them is NOT the time to find out they're not working right, or that you can't restore them.

    Sorry if there's just way too much information here to easily absorb... GOOD backups are just not simple, but they can be extremely important if this is your business, or a hobby that you don't want to lose your work on!


  • Registered Users, Registered Users 2 Posts: 1,302 ✭✭✭Heebie


    Sounds like you've already been through just about everything I've mentioned.. the question is, however, how much money do you LOSE if you lose that data? How long can you keep working if it's gone?

    If you're ACTUALLY remembering to get one drive off site, you're not doing too badly.. but if you're going to use a hard-drive.. get another one.
    Have your A drive that stays at the office, a B and C drive that you rotate, so that on say Monday.. you have the A and B drives at the office, and B gets sync'ed with A before you leave.. the next day do the same with A and C, then back to B the next day. This way even when you're at the office.. you have a copy no more than 1 day old at home.
    Ballyman wrote: »
    Now I know it's not ideal as both A and snych'd drives are both in the same location at the same time but I can't justify the expense of tapes or online storage etc. Also my broadband speed/quality isn't good enough for online storage.
    Is what I am doing stupid? Wrong? Can you recommend anything better?


  • Closed Accounts Posts: 2,131 ✭✭✭oshead


    Hi Ballyman

    What you're doing sounds fine to me. I would make just one change. In any backup system you've got think about any single points of failure. With your current system the failure point is the fact that you have both drives in the same location for such long periods. This increases the risk of losing the lot due to fire, theft, power surge or dare I say it earth quake, tsunami or volcanic eruption. :)

    So the change i'd make is this. Add a third drive to your system. One stays off site at all times on a scheduled rotation system. Two drives synced on location every night. Then on a weekly basis swap drive A or B with drive C.

    To protect against accidental deletions and or changes the preferred method is to use backup software that tracks these types of changes.


  • Banned (with Prison Access) Posts: 1,822 ✭✭✭Ballyman


    Thanks for that. Sorry eas for sidetracking. Back to your problem now!


  • Registered Users, Registered Users 2 Posts: 1,435 ✭✭✭eas


    Ballyman - I'm using more or less the same method as you are at the moment. I've got software that monitors one drive and mirrors it to another whenever I make changes. Thing is it's failed me twice now. I noticed last night that it hasn't updated the redundant drive for three weeks. :eek:

    I'm in the process of rebuilding the backup drive now - Just think in my case I should be getting something more fit for the purpose.

    I'm just about to order a Drobo - which I think will solve half my problem. The other half that people rightly pointed out is proper backup. Thinking about it, online won't be an option. Moving just the files I have on site at the moment would take me until NAMA made a profit.


  • Advertisement
  • Registered Users, Registered Users 2 Posts: 2,262 ✭✭✭stcstc


    have a look around, i think you will find other raid boxes which will cost you less and will perform quicker than the drobo


  • Registered Users, Registered Users 2 Posts: 1,435 ✭✭✭eas


    stcstc wrote: »
    have a look around, i think you will find other raid boxes which will cost you less and will perform quicker than the drobo

    Thanks Steve - maybe I should have a quick look, but would a standard RAID box offer the same benfits as the Drobo? Mainly being able to swap drives in and out as required for disc space?


  • Registered Users, Registered Users 2 Posts: 437 ✭✭Tech Overhaul


    For those of us who are looking for a cheaper backup option. I think this is a good one. It's the Netgear Stora Home Network Storage. It's a 2 drive NAS. Comes with a single 1tb drive to start you off. Obviously you'll need to add a second drive for the raid function. It's available on komplett and Amazon.



  • Registered Users, Registered Users 2 Posts: 479 ✭✭Cameraman


    I've just bought one of these :

    http://www.elara.ie/products/detailsfull.asp?productcode=MMEV930725

    Plan to use it as my photo working drive initially, via a single eSATA port, and back-up to the former external drives I used to use for this (combination of eSATA and USB drives).

    Starting with 1 2TB drive, and can add another later, as needed. It has various RAID and JBOD options. Seems like the best/cheapest option for what I need.
    When I outgrow its 4TB capacity - I will use it for backup and replace with whatever is the latest and greatest then.

    I'm avoiding NAS devices - too slow and Drobo - too proprietary and overpriced.


  • Registered Users, Registered Users 2 Posts: 2,262 ✭✭✭stcstc


    eric i would look at something like

    http://www.g-technology.com/products/g-speed-es.cfm


  • Registered Users Posts: 179 ✭✭bob the bob


    RAID is not the same as backup. Sure, it protects you if one hard drive in a RAID array dies.

    But, crucially, it doesn't protect you against accidental file deletion\corruption etc.

    Once a file is deleted\corrupted, you will want to go to last night's backup to get the last working version of the file. RAID can't help you there.

    RAID+backups is nearly bulletproof. Ideally offsite backups.


  • Registered Users, Registered Users 2 Posts: 1,302 ✭✭✭Heebie


    Now you're also showing the need for a rigid workflow that includes:

    1.> Preserving the original file in unedited form. (easy with camera raw, but a lot of people will work directly on original scans off their scanner.) Marking the original as a read-only file is a good idea towards achieving this. I've always used -RAW.<extension[usually .tif]> to name scans. These files should be done within the highest available bit-depth of the original device. (most can do 42-bit nowadays, which is I think 14-bit per channel, so save it as a 16-bit per channel file.) This should also be in the largest available color space you have. (I use Kodak ProPhoto for storing raw scans, unless I have a color profile for the particular scanner that I made the scan on.. in that case I use the color profile for the scanner.)

    2.> Versioning As you make revisions to the file.. preferably using non-destructive methods like layers and such, save new versions of the file when you get to "milestones".. such as.. if you get the color *JUST* right, or you *JUST* became happy with a cloning job that you did to get rid of some gunk etc.. and choose a set number of versions to keep, such as 2 or 3 versions that you rotate through... with name ending in something like -WORKING-A.<extension>, -WORKING-B.<extension> etc.. Working files should still be in the greatest bit-depth available from the source (from a camera RAW file, this is likely to still be 16-bit per channel.) This should be in the largest color space available to you again. I prefer Kodak ProPhoto. Many cameras now can shoot in Adobe 1998 RGB, although most encode in sRGB. sRGB is a very small color space (used often because it reproduces reasonably accurately on printers and screens, unlike something like Kodak ProPhoto which generally looks fantastic on screen, but won't print at all well.))

    3.> Creating entirely separate output files optimised for whatever output device you wish to print or view on. Here you would generally resample your image down to 8-bits, for the print resolution of the printer/screen, and to a color-space like sRGB, or better, one specific for the output device to be used.

    I suppose this includes color workflow at least as much as a workflow designed to minimise or eliminate the "loss" of previous versions of a file as much as possible.. I used to do a lot of photo restoration & custom large-format printing.. so it's well ingrained in my head to do the whole thing at the same time.

    Some might ask why bother keeping the larger color-depths at all.. it's just to give yourselve the most leeway in editing etc.. the more different colors you have that aren't being "thrown away" by color management, the more fine detail you can coax out of the photo, and when you remap to the right color space later, you're creating something truly optimised for the media.. but if you get a different media later that has a better color gamut.. you're not limited to the color gamut of the original output device nor sRGB.
    Your screen scan actually show you a LOT more colors than anything to printed paper (with no backlighting can. (a color slide or a really well-made transparency can actually come REALLY close.) Of course.. your LCD flat panel doesn't come close to what a really good old-fashioned CRT can either. (A plasma screen can come close.)

    Again sorry if I wrote too much.. I tend to babble. :)
    But, crucially, it doesn't protect you against accidental file deletion\corruption etc.

    Once a file is deleted\corrupted, you will want to go to last night's backup to get the last working version of the file. RAID can't help you there.

    RAID+backups is nearly bulletproof. Ideally offsite backups.


  • Registered Users, Registered Users 2 Posts: 593 ✭✭✭davmigil


    Really interesting discussion. I have been reviewing my own back up needs recently, so interesting to to read something similar.

    Here are my conclusions, for what they are worth, and apologies in advance for long post - we are looking at a few different layers of protection:

    1.Protecting against local hard drive failure - RAID 1. Chances are if your PC is anyway recent it will have RAID abilities built into the motherboard. Buy a second identically sized hard drive (and identical drive speed too, e.g. 7200rpm, no harm in matching cache size too, e.g. 16mb). Best to buy a different manufacturer's hard drive or at least one from a different batch. Physically it's not difficult to install. Installing operating system on RAID disk can be a bit trickier (but can be done), I took the lazy way out:

    On my PC I have a single 80Gb hard drive which acts as my C: drive and contains applications etc. I have two 500Gb in RAID 1 as my D: drive, to which I have moved all the MyDocuments stuff including Photos. I.e. all the stuff I can't re-install.

    2.Back-up and protecting against small scale disaster
    As said already RAID 1 won't protect you from accidentally deleting a file etc. Also if you had a power surge it could take out both RAID disks etc.

    For this an external hard drive (ideally a good bit bigger than your internal drive) comes in handy. Beside backing up your MyDocuments, you can make a clone of your C: drive for easy re-install. For the former I use SyncBackSE (also have free version, but this paid for one is way quicker). For the latter I have been using Macrium Reflect (free edition). Some versions of Windows 7 include a reasonable backup program too.

    Am interested to have a better 'versioning' back up option - ie if have different revisions of same file, would be nice to have back up of a few of them.

    Would make sense to unplug the external drive (both from power and usb), to avoid power surges, but take away convenience of setting up scheduled back ups to run automatically.

    3.Protecting against major local disaster
    So fire/flood etc.

    This is where off-site archiving comes in. So another external hard drive or two and rotate between home and work or relatives house etc.

    On-line storage (maybe only for most precious files). Free options include flikr (but limited file types). Another interesting one is 25Gb off free storage on microsoft's skydrive service .

    I am looking into some method of automating backup of 'precious' files to skydrive.

    foot notes:

    -NAS is especially useful if sharing files between multiple PCs and/or macs. You can use the NAS as your central repository and access your files from any device attached to your local network (also can open it up for access over the internet). Disadvantage is speed of access to files (usually much slower than USB direct attached drive, depending on your network and spec of the NAS). Many NAS have RAID options, but will need to back up NAS to external hard drive (or other PC on network or both) also.

    -Backing up to DVDs/CDs(/BlueRays too I would guess) - have had bad experiences with this option, would only recommend in conjunction with external hard drive etc.

    -Tape Drives - tend to be expensive as geared at business market, but depending on needs might be an option. Not as fragile as hard drives. Iomega Rev drive interesting, but cost/capacity not favourable compared to external hard drives.

    -Mac Leopard and above - Time Machine a good option, along with other form of back up.


Advertisement