Storage & Backups

DK32

Hi,

There have probably been a few threads over the years about this kind of thing, but nothing I have found so far as offered any sound advice.
I'm running out of ideas for storing my photo's.

Workflow:

• Copy all files direct from CF cards to local hard drive.
• Process whatever images I feel are the best from that batch.
• Move both the original files and the processed images onto external hard drives.
• Sorted by month & year they were taken.

I have a few large external drives and I'm rapidly running out of space.
I also keep all files backed up onto Bluray discs.

Questions:

• Am I mad to hold onto all of the original raw files?
• How do you store & catalogue your work?
• Anyone have any best practise tips n tricks?

Thanks,

oshead

Sounds like you are on track. What programs do you use in your workflow? What size archive are you talking about. Maybe get yourself a larger hdd and dump the lot on it. Komplett sell 1.5 TB drives for a little over €100.

pixbyjohn

I use this in Raid 1 config giving me 1 tb of storage mirrored. You can get them with larger capacity drives.
2TB USB / eSATA 2-Disk RAID Desktop Hard Drive
The Verbatim 2-Disk RAID storage solution offers several RAID modes including BIG, RAID-0 and RAID-1 providing high speed and reliability for both professional and home use. Blazing through the most demanding applications allowing you to work more and wait less. Available in a USB/eSATA combination, the drive can store your music, photos, video, and data files on your PC or MAC.

3 RAID mode settings
BIG – Default setting that provides maximum storage capacity
RAID-0 – Provides the fastest performance
RAID-1 – Writes data to disk-1 and automatically copies to disk-2 for redundant backup
USB / eSATA combination drive
eSATA II connection provides data transfer rate of 3.0 Gb/s (max)
Easy-to-use backup software included *
Store thousands of photos, videos, and MP3s
3-Year Limited Warranty

Link http://www.verbatim.com/products/detail.cfm?product_id=1AE5276C-1143-3415-5FC45A62B608067A&cat_id=1AEE7041-1143-3415-5FC4B78A413DDF2E

DK32

Thanks guys.

I have 3 1tb external drives and 2 small 500gb external drives that I take with me if I'm travelling. The 500gb drives are great as they are USB powered, so no need for an extra power pack.

My main pc I use for processing images has Raid5 with a hot spare.
I use CS4 for processing images, mostly cropping, curves, levels etc...

The biggest issue I come across aside from storage space is how to actually catalogue them in some way that it's easier to go back through my catalogue and find stuff that I have done.
With rainy days more & more frequent it's nice to go over the back catalogue and find something interesting.

oshead

You could use Extensis Portfolio Server for cataloging your images. Here's a link to more info. I've seen this in action and like the look of it.

DK32

Thanks oshead. I'm not really looking to spend any money on yet another software package if i don't have to.
I'd be very interested to hear how others manage their images and how they go about sorting them for easy future reference.

democrates

I've just recently changed to a new folder arrangement:
Year at the top level
Month and Day (MMDD) sub-folders with the title of the shoot
CR2 are canon raws, I convert to DNG for archiving and delete the CR2's

2008
2009
 0123-Blackrock Park
 1120-Pub Crawl
 1120-Randomers gaff
  CR2
  DNG
   proc
   print
   email
   web

proc is processing, these are full size, usually in native gimp format with layers, or tiffs where they're HDR's out of qtpfsgui

Then there are various output folders with flattened images sized appropriately, that may seem like overkill but it means you're more likely to post/email the correct images.

Borderfox

Shoot for the day back up all files to two seperate HDD and onto DVD, process all files and then store the jpegs only. The dvd contains all the original files and stored on the two HDD are the jpegs, the only files I keep the RAW files are for stock shots.

bbbbb

In terms of IT disaster recovery, you need to think about off site backups. Your bluray disks are a backup to you HD, if that fails, but if both of these are at the same location what happens in the event say of a fire?
If you have good broadband, it might be good to look at online storage as an offsite backup.

Note that RAID 0 is designed for performance, there is no redundancy. You lose data if any disk in the array fails.

conolan

Would I be correct in assuming that online data backup might be good except for the cost? If you have terrabytes it could seem pricey. What storage does flickr pro offer?

Cameraman

conolan wrote: »

Would I be correct in assuming that online data backup might be good except for the cost? If you have terrabytes it could seem pricey. What storage does flickr pro offer?

Online data backup is not too expensive - depending on your needs. In general, costs are coming down.

However, the big problem is transfer speed. Most broadband upload speeds are a small fraction of the download speeds - so in practice it can take a long time to backup (restore will be faster - but still slower than copying from a DVD or external drive). I use online backup for selected files and all other (non-photo) files I create.

Regarding backup - the problem with RAID is that the RAID unit itself needs to be backed-up offsite. I find it easier to manage external hard drives. I have a fixed 2TB external SATA drive (about 80% free). Then I back this up to separate 1TB drives (my file structure makes this easy). I use a USB gadget for this which allows you to plug in a drive - something like this :

http://cgi.ebay.co.uk/eSATA-USB-SATA-2-5-3-5-HDD-Hard-Disk-Drive-Docking-Dock_W0QQitemZ260529320439QQcmdZViewItemQQptZUK_Computing_HardDrives_RL?hash=item3ca8c1f1f7

However, I may invest in some sort of multiple drive unit in the future - probably not RAID though.

Finally, I use Lightroom to do the cataloguing.

magicbastarder

DK32 wrote: »

[*]Am I mad to hold onto all of the original raw files?

yes. if it's creating that much work for you holding on to that many photos.

qwertz

Online backup is all well and good until you lose access to the data. There are numerous reasons why this could happen (bancruptcy of the provider, technical issue, change of service rules...). Make sure you do a risk analysis before relying on it.

JackKelly

think about buying a NAS (wiki). It's essentially a big Harddrive that is connected to your network (as opposed to a normal external harddrive that is connected to your computer). It is always on, and can be put into a cupboard/stashed away, getting rid of the inconvenience of having to connect it to the computer etc. (especially handy if you have a laptop that moves around). The one I have has space for two hard drives but you can get ones for four. Furthermore, they are reasonably cheap; a ReadyNas (like in the link above) plus 2 hard drives comes to around €350

Usually they have RAID 1 which means that all your data is duplicated across 2 hard drives, so if one fails you can replace it without losing your information. As you data is being duplicated across multiple drives, if you have 2x1Tb hard drives, you still only have 1Tb of storage space.

If you have a Mac (and I'm sure there are alternatives for Windows), you can use Time Machine to backup all of your computers contents (including photos) to this NAS with minimal effort. Every hour, any changes that have been made to you computer get copied to the NAS. If you ever accidentally delete a photo, you can easily restore it from the NAS in a few clicks. More importantly though you can restore the entire computers harddrive from the NAS if worst comes to worst. I've restored individual files, and the entire hard drives a few times using Time Machine and is so simple.

Once you have sorted your backup in your house you need to think about offsite. Like others have mentioned, you can use services like Mozy or Backblaze, and backup the entire contents of your HD to their servers over the internet. This is a good idea and quite cheap ($5 a month with BackBlaze) but it takes AGES (i.e. 3/4 weeks) to backup an entire Hard drive the first time. Some of the NASs I was talking about have a feature that allows you to connect an external harddrive to them, and automatically dump all the contents on the NAS to the external drive. This means you can backup your computer like I mentioned above, then every week or two weeks, connect an external HD to the NAS, dump all it's contents, and put the external drive in a safe offsite location.

I Rambled on there for bit longer than I meant to lol. There are 3 main steps you should try follow for proper backup.

1. Redundancy - Protection in case of hardware failure (i.e. RAID 1: having two hardrives both with a copy of the same data in case one breaks)
2. Incremental Backup - Use software that doesn't delete and older backup when it makes a new one, i.e. you can rollback and search through old backups (Time Machine does this on Mac, not sure about Windows)
3. Backup Offsite/Over Internet

EDIT: Also, away from backup, I would suggest using Lightroom as it removes the need to save edited versions of the RAW files. If you edit a RAW file in Lightroom (crop/levels etc.), instead of copying the file to an "edited" folder, it just saves the settings of the edit in a catalog. If you want to specifically export the photo or make a copy you can, but everything is managed from within Lightroom, which saves a hell of a lot of time when processing (you don't have to move files around yourself). i was using Photoshop for years and manually managing the folders, and I wouldn't dream of going back to it after Lightroom.

RoryW

This was recommended to me a few months back ( of course I didnt do anything at the time and am now rapidly running out of space)

http://go.iomega.com/section?SID=146d32184ba2895e443ed24060265260a89:4725&secid=40380#overviewItem_tab

RoryW

http://nikonusa.com/Learn-And-Explore/Photography-Techniques/g35f3hn6/1/Preserve-and-Protect-Got-a-Backup-Plan-for-Your-Photos.html

mervifwdc

A big factor is how often do you go back to search for any group or individual image? Say for example, every image of weddings, or family, or cars, or whatever. to search like that (across all dates/years) requries you to catalog the images properly using a cataloging system. Lightroom does a reasonable job of this, with speed being an issue when you attempt to handle volumes. It's seems to max out when handling 50k images per catalog.

I use lightroom because it's not too hard to recall what year I took a specific image, but there will come a time I will not reall that, and that's a problem. So, I force lightroomto write the keywords back to the IPTC data fields on the images, which means that any other search program that can handle the large volumes can search on keywords in the future. If you only use Lightrooms collections feature, it's only within lightroom and cannot be used outside of it (i.e. not future proof).

and yes, I'd hang onto the CR2's. Maybe convert them to DNG's

Merv.

Cabaal

conolan wrote: »

Would I be correct in assuming that online data backup might be good except for the cost? If you have terrabytes it could seem pricey. What storage does flickr pro offer?

Cost isn't a problem really depending on what type of online backup you use, mozy is as little as 2e a month.

I'm currently crazy enough to backup 300GB's to mozy, 10 days in and I'm upto 30GB done,
as well as online backup I have a external drive backup on site also

Borderfox wrote: »

Shoot for the day back up all files to two seperate HDD and onto DVD, process all files and then store the jpegs only. The dvd contains all the original files and stored on the two HDD are the jpegs, the only files I keep the RAW files are for stock shots.

Are you not converned about not being able to re-cover from dvd in a few years time?

Its often the case that DVDR's degrade badly over a number of years, in addition they are also not very cost affective when compared to harddrives

ThOnda

Import all RAWs into LR on local HDD, rate, reject (x key) and delete rejected files from HDD.
Weekly backup of LR catalogue with included RAW files onto the external HDD (WD 1TB).

Borderfox

Cabaal wrote: »

Cost isn't a problem really depending on what type of online backup you use, mozy is as little as 2e a month.

I'm currently crazy enough to backup 300GB's to mozy, 10 days in and I'm upto 30GB done,
as well as online backup I have a external drive backup on site also



Are you not converned about not being able to re-cover from dvd in a few years time?

Its often the case that DVDR's degrade badly over a number of years, in addition they are also not very cost affective when compared to harddrives

Not really, I have used archival quality dvdr's and will move to blu ray when it is more cost effective. They are cost effective from the point of view of that hard drives fail.

Cabaal

Borderfox wrote: »

Not really, I have used archival quality dvdr's and will move to blu ray when it is more cost effective. They are cost effective from the point of view of that hard drives fail.

Fair point,

Though I've had bad ex[periences with some good quality CDR and DVDR media in the past where even though it was good quality and stored in a good way it still experienced errors when reading.

This along with the time required with burning to DVD meant I found it more cost affective to backup long term on HD which to be honest is pretty safe once powered down and stored correctly.

qwertz

Don't make the mistake in trusting optical media or hard disk drives for your sole long-term backups. Currently there is little or no alternative to premium-grade tapes for this purpose. SSD might be a valid alternative but the technology still is too new to be able to give a definate answer.

Don't rely on a single media for critical or important data.

bigpoppa

Interesting thread as I am uber paranoid about data backup.

The film and tv industry has had this same issue as more and more hard disk based cameras are used (such as Red One and Sony PMW-EX1). Where people used to archive to tape (digibeta etc) now they use G-Tech RAID drives for short term storage (on set transfer) and then eventually archive to LTO tape at the end of the project.

I use a company in Dun Laoghaire that opened in November of last year who charge 150 E per drive to archive to LTO tape. They primarily do video backup but have done archives for professional soundmen and photographers also.

Bottom line is when you use enough hard drives you realise they fail a lot. I use a combination of mozy / timemachine and archive important stuff to LTO every few months.

If anyone wants more information drop me a pm.

Cabaal

bigpoppa wrote: »

Bottom line is when you use enough hard drives you realise they fail a lot. I use a combination of mozy / timemachine and archive important stuff to LTO every few months.

If anyone wants more information drop me a pm.

Having used Timemachine on my MacBook I'm hugely impressed by it as its so easy to use, problem is my main desktop isn't a mac

stcstc

i have an lto in my server in the office it runs twice a day

then once a fortnight i do a single once off, then also archive to it too

njburke

I picked up a bit of a bargain earlier in the week at PC world.
I bought a Netgear readyNAS duo 2150 network storage device for 167 euros, it came with a single 500GB drive. Snazzy specs here

What it do? I hear you ask.

Data:
Pretty much everything you'ld want if you frequently turn photons into data for a hobby or indeed professionally.
Firstly, its supports RAID X, so you fit it with two drives and one is a mirror copy of the other. If you need more storage buy two bigger drives, put one of the larger ones in and it will automatically copy the original smaller drive onto the larger one, when finished, put the other large one in and, bingo. You've got data redundancy again.
Dual drives will protect against failure of one drive,but not theft,fire or small children. if you've a third drive, slip that one in every now and then and give it to someone for safekeeping, lets say the Mammy not a flatmate.

Connectivity:
Plug it into your broadband router, now you can see the storage from your laptop, Xbox, media renderer etc. This particular box also allows you to access it over tinternet. It also has USB connectivity, so you can plug USB keys, hard drives and your CAMERA directly into it. You can set it up so that it automatically copies the files from say your D70 into one share, your D80 into another yada yada.

Usability.
It comes with several embedded applications and services, such as the ability to share photos stored on it accross the web, ReadyNAS photos. Theres the usual stuff so that you can share Video and music to PS3s, Iphones, Xboxs linux/Win/Mac Pcs etc. Its even got a print spooler (you can plug in a USB printer and print to it from anywhere (tinternet included, I believe, picture this , do a bit of PP at lunchtime in the office, arrive home and the photo is sitting in the printer,dry).

All in all a very useful device for 167 euros, a second 500 GB drive costs £50.
Considering my first hard drive was IR£350 and a whopping 40 MB, this is a bit of a steal. Some of the sysadmin types may come along and poo poo this box, but do bear in mind the price.

ReadyNAS duo 2150 (4/5)
NJ

al12apl

Here's my 2c worth,for the OP, basically my setup.

I have an iMac and a MacBook Pro. both have 320's in them and are perfectly in sync, however no photo files are on these drives.
I have a 1TB my book studio hooked up by firewire 800 which is my main media drive. I keep every single raw file, I have all 26534 of them I have ever taken, although some are jpeg as I used to shoot jpeg. I also have a folder that stores every set of photographs that have been for clients.

I use Lightroom for cataloguing. The photo's stay on the My Book and lightroom just references them. I choose which ones are to be exported for the client, or personal use, and these go to a collection in Lightroom, for easy reference in the future.

The My Book backs up to 2 500GB drives which are Stripped with RAID 0. It also backs up with Time Machine to another 1TB and the My Book is cloned to a 4th drive which is off site.

Some say this is to much, but you can never have enough backups. DVD's fail more times than I have ever known.

That answers the first two of the OP's questions.

As for tips and tricks, keep on top of what you do, don't ever have to "get backup to speed with what you've taken".
keep everything you take
don't delete the original files and keep exports only.
Backup
then backup again
and again.
then put another offsite.

All this is from 10 years experience working with digital images, having lost thousands that have never been recovered.

Holding onto photos shouldn't be work, if you keep on top of it all.
I don't believe that you need to spend extra money on specific RAID volumes, ordinary HD's will suffice if they're well managed and organised. That's not to say they're pointless, if you can afford them, they can certainly make things easier, but i have never used one for more than 6 months. I prefer to spend the extra money on an extra HD or two, for added space and backups.

I appreciate this is a log winded post, but it is just me rambling from my experience, no script.

A parting word; If you ever have to buy a HD as you need more space, you MUST buy backup HD's for that new drive, at the very least 1, ideally two or three. A minimum in my opinion is one onsite backup, and one offsite, but that is exactly that, minimum.

Regards

Sniipe

TimAy wrote: »

2. Incremental Backup - Use software that doesn't delete and older backup when it makes a new one, i.e. you can rollback and search through old backups (Time Machine does this on Mac, not sure about Windows)

Whats good for windows like time machine? I'm looking for a fast backup program that does incremental backup onto my NAS specifically for my photo folder.