finding duplicates across a network

[DarkTXKnight]

I posted a question similar to this like months ago and couldnt find it to see if I got an answer so I'll present the issue again. I have 5 computers at home that have mp3's and movie files on them in different directories. what would the best method of looking for and deleting duplicates and then copying all of them to a big hard drive on one machine?? any freeware or cheapeware that would help would also be appreciated!!!
Thanks

 

[Thyme]

At the risk of not really answering your question:
There might be software out there for this, but I don't know of any offhand and I assume you've searched. If the files are actually identical (as in not just deleting like matches), you could easily accomplish this with a basic scripting language like Perl.

 

[DarkTXKnight]

I don't know any perl though

 

[wkinney]

can also achieve this with linux commands find, grep, sort, diff, cp and perhaps awk

 

[Thyme]

Originally posted by: DarkTXKnight
I don't know any perl though

If you've programmed before, you can learn it.

 

[Czar]

there are programs out there that find duplicates on one computer

get one of those
map the drives of the networked computers on one computer
find duplicates

 

[DarkTXKnight]

good point czar.... can you recommend any?

 

[DarkTXKnight]

bump

 

[Nothinman]

Whatever you find, I'm sure it'll take a very long time. I wrote a script to look for and upate a single, very small, file (sometimes 2 or 3 copies were on 1 machine) and running it on a few hundred machines took hours. You have a lot less machines but you'll be comparing a lot more data and unless you want to rely solely on filenames you'll need to compare md5sums or something so it'll be an extremely slow process.

 

[Heretik]

I use NoClone on my personal PC. They also offer an enterprise edition which will work across a LAN.

http://noclone.net/