Compare many Gigs of files

User avatar
Timelord
4StarLounger
Posts: 504
Joined: 23 Jul 2010, 18:36
Location: Westlake, OHIO

Compare many Gigs of files

Post by Timelord »

I have a situation where a coworker copied several very large volumes of data from 1 location to another. The smallest copy is about 500 Gig. I need to compare the copy with the original. We are in the process of moving the files from 1 piece of hardware to another and need to validate we got everything. Does anyone know of free software that can handle this without crashing or bringing the servers to a near stop?
Who will you Inspire today?

JoeP
SilverLounger
Posts: 2079
Joined: 25 Jan 2010, 02:12

Re: Compare many Gigs of files

Post by JoeP »

Do you need to compare individual file content or just the tables of content?
Joe

User avatar
PaulB
SilverLounger
Posts: 1600
Joined: 26 Jan 2010, 20:28
Location: Ottawa ON

Re: Compare many Gigs of files

Post by PaulB »

You could try using the File Checksum Integrity Verifier (FCIV) utility to compute the MD5 or SHA-1 cryptographic hash values of your source file and of your destination file. Hashes are used to ensure the data integrity of files. Because the hash algorithm always produces the same output for the same given input, you can compare a hash of the source file with a newly created hash of the destination file to check that it is intact and unmodified.

Information about the File Checksum Integrity Verifier (FCIV) utility can be found in the Microsoft Knowledge Base:

841290 Availability and description of the File Checksum Integrity Verifier utility.

I have no idea if this utility will handle files the size you are dealing with. You may want to look into other hash generators.
Regards,
Paul

The pessimist complains about the wind. The optimist expects it to change. The realist adjusts his sails.

User avatar
Timelord
4StarLounger
Posts: 504
Joined: 23 Jul 2010, 18:36
Location: Westlake, OHIO

Re: Compare many Gigs of files

Post by Timelord »

JoeP, I need to compare the files but not their content at this time. Most files will be no larger than 1 or 2 Meg and no folder over 4Gig in size so we are talking a very large number of files and folders. I am working a simple space compare (does this top level folder hold as much as this other top level folder) but I am not confident that will actually tell me if there is an issue.
Who will you Inspire today?

PJ_in_FL
5StarLounger
Posts: 1113
Joined: 21 Jan 2011, 16:51
Location: Florida

Re: Compare many Gigs of files

Post by PJ_in_FL »

Can you get the "Properties" pop-up for the directory structure of each backup and compare the the "Properties" of the original directory structure?

In other words, do you get the same numbers for "Size:" and number of files and folders in "Contains:" in the Properties dialog pop-up windows when you right click on the top level folder of each group of files and click "Properties"?

The "Size on disk" may be different if cluster sizes differ between storage media so don't compare those numbers.
PJ in (usually sunny) FL

JoeP
SilverLounger
Posts: 2079
Joined: 25 Jan 2010, 02:12

Re: Compare many Gigs of files

Post by JoeP »

Joe

User avatar
Timelord
4StarLounger
Posts: 504
Joined: 23 Jul 2010, 18:36
Location: Westlake, OHIO

Re: Compare many Gigs of files

Post by Timelord »

PJ_in_FL. That is what I was trying but some of this is so large it takes a very long time to get the information, and as always, this is due yesterday.
JoeP. Thanks, I will look into that.
Who will you Inspire today?