File and Folder Management on a File Share Server

ROS123

I have a particular file share of say 300gb and one particular user is using approx 50% of that. This person works with a lot of photos and some videos. He has stuff going back 7-8 years. The thing is he is not very good at organising the stuff and has probably has a large percentage of duplicates in there and probably some very large size files which he probably doesn't need as much as he thinks. He also doesn't want to delete anything for fear he may need it again. Most of this has grown organically over this time period. I think if I could get him organised, he would begin to manage the data better. It's just that it has got so disorganised that he is afraid to delete anything.

Is there a program for interrogating these folders and identifying duplicates files and folders and highlighting the very large files ?

seamus

If it's a company working with design/photos, then one thing I would probably look at is an archiving process whereby files over a certain age (say five years) are moved off the primary file share to a less immediate file storage like NAS. You could get a 2TB (mirrored) NAS for less than €500, which should be well capable of handling his excess data for a while.
The original folder on the file share can then contain a link to the old file if he ever needs to access it again.

150GB isn't a massive amount of photos and video to accumulate over 8 years IMO, if that's what he works with on a day-to-day basis. While you may need to look at organising the data, this may also be an indicator that you need to look at expanding your available disk space to account for future needs.

To address your original question, you could do an initial sweep using TreeSize. This will tell you which folders are holding the most data, so you know where to start.
Finding duplicate files is quite time-consuming as it involves indexing the tree, and at the end of it you may only find a small number of duplicates.
I've scripted this before to clean up a music library, but it takes a while.

Finding large files is actually a very easy one to do. You can use powershell to report on files over a certain size:
http://www.christiano.ch/wordpress/2009/08/11/powershell-how-to-find-large-files/

You can also use dir from a command prompt to print out the contents of the entire share and the sizes, but that can be a little fiddly getting the exact directory.

uch

Download WinDirStat, it will go through everything and show you offending files

ROS123

Thanks Seamus and uch. Its not a design company, its a person who works with people and needs to keep photographic records of outings and events. They never realised when they started that it would grow so much, so it is badly organised. He may have duplicated folders into other folders for what seemed like a logical reason to him i.e. filing the same folder of photographs under Folder A and again in Folder X. If it was just that, I could get a handle on it.

I think I will use a combination of the suggestions, I have identified the large folders and types. I even found a product called Duplicate Cleaner, which gives me more information on the duplicates, it suggests approx 40gb are duplicates which is a high percentage. I think I will provide a cheaper NAS and ask him to at least start moving folders on to it in order to get them into some order. He can then move the important, properly organised folders back onto the Share.

Thank you for taking the time to help....

rolion

BEFORE moving stuff to NAS...i'll buy 2(two) removable hard drives,500GB each and copy full folder.
Keep one drive in a safe place in office,the other keep it offsite,possible in to his home or MD's home or yours !!!

As soon as you start moving file,he sooner than you we'll discover that files and folder ,very important, are ... missing !

To avoid this "finger up",create a backup ,a snapshot as per today on the two drives and take it from there with the clean-up !
Personal,i use TreeSize Pro when i move or archive...


Good luck...

Standard Toaster

+1 Treesize


Maybe another candidate is FreeNAS. It is based on the FreeBSD platforms and supports sharing across Windows, Apple and UNIX systems. The latest version FreeNAS 8 includes ZFS, which supports deduplication among other things.
Would be an idea if there are multiple copies of the same files.

Capt'n Midnight

there are plenty of apps that look for duplicates
ignore any that don't use checksums and preferrably full file scan too


in the linux world you can use utils like fslint to replace duplicates with links
this means that one physical file has two directory entries

the bit I'm never sure of in the windows world is creating links with MKLINK and being comfortable with it splitting the link when one of the files is edited or deleted. Yeah I still have to stop and think of the difference between hard and soft links because I'd only use them for archiving or redirecting folders

but on a lighter note you could move the old stuff to an external drive and symlink to it so it's just another folder to the user