Solved

7zip zip file size growth

Posted on 2011-03-10
5
431 Views
Last Modified: 2012-05-11
Ok i have 140,000+ exchange log files that i need to get rid of, but they are not to be deleted. so instead of keeping 140GB on my server, i thought i would zip them. since it is all text and should zip quite nicely. so i wrote a script to seperate all 140,000+ logs into groups of 500 to be zipped. This is my result of the first 7 groups.

1) 1.754MB     <----What i expected
2) 129.455MB <----A little large, and as you can see it gets bigger
3) 181.573MB
4) 236.767MB
5) 249.138MB
6) 73.349MB   <----Now small again?...
7) 209.941MB

So my question to you all is, am i doing something wrong? nothing should be running on the pc in the background but i thought that it would just take longer to zip the files, not lower the quality.

This is what i am zipping with and the file specs
7za.exe                  <----7zip command line exe
Ultra                       <----7zip highest compression
EachFile is 1024KB <----All are the same size.

Any Ideas? i would like them to be 1MB if possible like the 1st group. :)

Thanks!
0
Comment
Question by:FnGizzardBrain
  • 3
  • 2
5 Comments
 
LVL 23

Accepted Solution

by:
Dr. Klahn earned 500 total points
Comment Utility
The fact that the log files are the same size is misleading in this case.  They won't all compress to a standard size, even though the source files are the same size.

A log file that was used very little will compress very well, because most of the contents are unused.  A log file filled with dissimilar transactions is not going to compress well, because there is little redundancy, while a log file filled with similar transactions and similar timestamps will compress well due to considerable redundancy.  There may be considerable variation in compression efficiency.

I would, however, look very carefully at that first one that compressed 500 1 MB filed down to 1.7 MB.  The only way I can see that happening is if the files were filled with zeroes.
0
 

Author Comment

by:FnGizzardBrain
Comment Utility
I see, i had a feeling it would be the contents not just the size. ok thank you for your input i will explore this and comment again tomorrow. Is there a way to compress that type of file more so than what i am getting? Any other program or type of compression i should look into?

Thank you
0
 
LVL 23

Assisted Solution

by:Dr. Klahn
Dr. Klahn earned 500 total points
Comment Utility
You could look into bzip2, but I'm not sure whethere it's going to provide significantly better results.  Still, a byte saved is a byte earned.
0
 

Author Comment

by:FnGizzardBrain
Comment Utility
I actually had some free time, i opened up several files from the first group and the second group to compare. And you are right, the first group was not blank however it is filled up with the same characters over and over again. Unlike the second that is a random assortment of characters and symbols. So that answered my question now i have a new one.

Is there a compression program that will do what i want?
0
 

Author Comment

by:FnGizzardBrain
Comment Utility
Ok sounds good, Thank you for your help Drklahn!
0

Featured Post

Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

Join & Write a Comment

Suggested Solutions

Lets start to have a small explanation what is VAAI(vStorage API for Array Integration ) and what are the benefits using it. VAAI is an API framework in VMware that enable some Storage tasks. It first presented in ESXi 4.1, but only after 5.x sup…
How to update Firmware and Bios in Dell Equalogic PS6000 Arrays and Hard Disks firmware update.
This video teaches viewers how to encrypt an external drive that requires a password to read and edit the drive. All tasks are done in Disk Utility. Plug in the external drive you wish to encrypt: Make sure all previous data on the drive has been …
This Micro Tutorial will teach you how to reformat your flash drive. Sometimes your flash drive may have issues carrying files so this will completely restore it to manufacturing settings. Make sure to backup all files before reformatting. This w…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now