Compress Large Files in a Fast Manner

Hi,

How can we compress a large file quicker. Right now the file size is like 15GB and will probably grow. Winrar takes hours. Is there any tools out there doing a better job?

thanks for your help
ebi168Asked:
Who is Participating?
 
mchkorgConnect With a Mentor Commented:
You can't get both :(

As it's a DB export, you should for example extract, let say, a 100 MB file from it and do some tests with several compression profile (from low to best).
Maybe using "best" is useless (two times the time for 2% better)
With some values from these real examples, you'll find the right solution for your needs.

If it's a DB export, you might also be able to generate it as multiple files (1 GB files ?) and see if some multi-threading compression (aka, start several winrar) can help. It depends on : hard disk interface / multiple disks / number of CPU/cores / memory etc.
Example : on a "common" computer, don't think about it, it's useless. On a real server (several SCSI disks using RAID), you might get 4 times faster by encoding 4 files. You "might".
0
 
ebi168Author Commented:
Add: compress fast but also ensures that it has good compression rate
0
 
mchkorgCommented:
Look, you can use:
winzip
winrar
winace
7zip
bzip2 (if unix/linux)
gzip (if unix/linux)
some others...

But no matter the one you choose, it's almost the same algorithm at the end.
You have to choose a compression/speed ratio and decide what you want : real smaller file or not ?
Example with winrar, in the options : "create default profile" and you choose something different from "best", it should be faster, bug a bigger file.
It also depends on your file content: text, raw data, something specific ? Some content will require more time(/techniques to be correctly compressed.

You can't just ask these tools to "do it faster".

By the way, maybe the idea would be to have something to cut your file into smaller pieces (1 GB parts). Thus, you might be able to multi-thread your process (depending on the disk speed/interface and so on.

If you want some help to decide: tell us what's this file, how it is generated and so on...

Regards
0
Cloud Class® Course: Microsoft Exchange Server

The MCTS: Microsoft Exchange Server 2010 certification validates your skills in supporting the maintenance and administration of the Exchange servers in an enterprise environment. Learn everything you need to know with this course.

 
ebi168Author Commented:
Hi this is just txt files with data rows in it, think about the export from a database table...
0
 
ebi168Author Commented:
We want a high size compression (as the compressed file needs to be sent to outside network) while the faster the better. thanks
0
 
ebi168Author Commented:
mchkorg: thanks for the comment. let me do some research into it.
0
 
ebi168Author Commented:
how about 8 duo core AMD processor, with SCSI and RAID controllers. So multithread appears the solution then.
0
 
mchkorgConnect With a Mentor Commented:
Clearly,

1) Try to generate smaller files
2) Do some tests to find the best time/compression ratio with your compression tool
3) script it all :)


0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.