ZIPping a massive set of files (10 GB)

[ChrisGreaves]

(Moderator: I'm not sure of the correct forum, so please move if required)

Question: What do YOU do when you want to make a monthly ZIP snapshot of your essential folders?

I had a bit of a scare this morning, and decided to ZIP my T:\GREAVES\ folder which contains all my client & production stuff. 10 GigaBytes of stuff.
( I do a daily RoboCopy backup but that is a massive accumulated archive, so I figured to do a monthly "snapshot" of the important files)


My first effort failed "PKZip (E11) Disk Full", although there are 56GB spare on drive T:
(It got as far as 3.4GB as a temp/part file last time I looked before it crashed)

My second effort with Win XP "SendTo compressed folder" failed (image below) and I don't feel like working through a zillion folders renaming a few of them.

1.JPG

[jonwallace]

According to certain sites whose denizens regularly compress 10GB worth of data , KGB archiver is a possible candidate. Given the nature of some of these sites (Websense here at work blocks some...), I would get it from the developer's site and scan it before running...

[Leif]

Personally, I would invest in some backup software that will do the compression for you - e.g. ShadowProtect - or just copy onto 2-3 DVDs or whatever. Life is too short to spend time faffing around with multiple procedures, and doing a mail-merge to 264 contacts...

[John Gray]

I [...] decided to ZIP my [...] folder [...]. 10 GigaBytes of stuff.

When you get it working, do tell us how long it takes!

[viking33]

Personally, I would invest in some backup software that will do the compression for you - e.g. ShadowProtect - or just copy onto 2-3 DVDs or whatever. Life is too short to spend time faffing around with multiple procedures, and doing a mail-merge to 264 contacts...

Similar to Leif, I use Acronis True Image to image the entire 26 some odd gig C: drive using normal compression and it only takes a few minutes and is reduced to about 8 gigs or so. Why pick and choose when you can do the whole thing and if you need to restore only certain files or folders it will allow this also, from the same image.

[ChrisGreaves]

KGB archiver is a possible candidate.

Thanks Jon. I have d/l it and will play with it this weekend. It seems to use proprietary (i.e. non PKZIP) standard compression, but that doesn't really matter because for my purposes the compressed file is not for export outside this office.

[ChrisGreaves]

ShadowProtect

Thanks Leif. i have d/l this too and will play on the weekend.

and doing a mail-merge to 264 contacts...

Oooh! mean!!

[ChrisGreaves]

I [...] decided to ZIP my [...] folder [...]. 10 GigaBytes of stuff.

When you get it working, do tell us how long it takes!

I can do that right now, sort of.

To do the full PKZip2.5 or WindowsXP zip of the 10GB T:\Greaves\*.* takes forever, because it doesn't complete!

I was keeping track while the PKZip2.5 ran:

Code: Select all

pkzip25 -add 20100914 t:\greaves\*.*  -dir 

8:23 - 8:30 Preparation (probably reading folders and file information) Occupied 98% of the CPU for 7 minutes and delivered a 65K zip file, presumably holding all the folder data.
8:30 - 8:53 amassed a 2.7 GB part file
8:30 - 8:59 amassed a 3.4 GB part file.
I noticed the PKZIP(E11)Disk full message at 9:11, but my guess is that by 9:00 the file was close to perfection. I have very few media files in the T:\Greaves (media are on the Big Beige Box) so I'd anticipate a typical reduction of 66%, yielding a 3GB file from 10GB of material.

A further guess is that PKZip2.5 has a 65K programming limit on the folder data, and that that got blown in my case.

(Thinks: Maybe I should break the job down into about three small chunks to see if that works ... notwithstanding the excellent suggestions to use software that DOES work!)

[ChrisGreaves]

I use Acronis True Image

Thanks Viking, i have d/l this too, and will devote even more of my weekend to playing with it.

[JoeP]

PKZip 2.5 is very old. Have you ever created a ZIP file that was 4GB with it before? That was a file size limit in the FAT32 days. Could be PKZip just does not handle that much data and does not know how to split a file.

Joe

[ChrisGreaves]

PKZip 2.5 is very old. Have you ever created a ZIP file that was 4GB with it before?

Hi, Joe.
Yes, PKZip25 is very old, but, like me (grin) seems to work consistently and is well-known.

I've zipped into a 9GB file before now, but it has been a small collection of files.
I suspect the limitation is a 65K ceiling to store the folder/file information. Tonight while playing around (next post!) I received a warning "PKZIP (Z155) Too many Files".

[ChrisGreaves]

My first effort failed "PKZip (E11) Disk Full", although there are 56GB spare on drive T:

Notwithstanding my weekend assignment, I decided to persevere with PKZip25 while I got supper ready, did two loads of laundry, fed the cats, folded the laundry, prepared for tomorrow's trip to Buffalo, ...

The attached batch file uses the -EXCLude parameter to zip all folders excluding my four primary folders.
The 4 primary folders are zipped to 4 separate zip files (a, c, p, t) and the remaining files are zipped to complete the set of 5 ZIP files which contain a snapshot of my T: drive.
(the command SetDate.bat sets the environment variables for YeaR, MonTh and DaY)

Code: Select all

Wed Sep 15 20:51:09 2010 : STARTED GREAVES_20100915 
Wed Sep 15 21:50:08 2010 : ENDED

As predicted by the attached Excel workbook, the run took an hour.
I might load the figures for the three suggested archiving packages as a comparison.
The workbook shows quantities in GigaBytes, number of Folders and number of Files for each of the four primary folders.

If I continue with the batch file I'll probably make it look for a monthly file (Greaves_%YR%%MT%.zip) and create it if it does NOT exist.

[Teunis]

Chris,

I Zip my project files by project (each project has its own folder - the folder can contain zipped and rarred files) once it's finished. The largest one is 36 GB which reduces to about 10 GB, and takes about 30 mins to do. I use the internal zipper of Total Commander (my standard file manager, V. 7.55a) in mode 6 (standard compression). I do this for already 15 years and only once had a problem with a single file. The entire HD of my laptop 160 GB used of the 320 GB, including the zipped project folders is backed-up once a week with Shadow Protect. This Full Backup takes about 1.5 hours onto an external 500GB USB 2.5" Transcend StoreJet drive. I don't use incremental backups
Total Commander lets me open a zipped folder directly, to access any file in it. Similarly Shadow Protects lets me mount a backup as a virtual drive so I can open it.

Hope this helps.
Regards, Teunis

[jonwallace]

I know that I'm reviving an old thread, and I know that this isn't about zipping, but maybe this register article will put 10 GB of files into perspective...