Compress Large Files in a Fast Manner

Posted on 2008-10-01
Medium Priority
Last Modified: 2013-12-01

How can we compress a large file quicker. Right now the file size is like 15GB and will probably grow. Winrar takes hours. Is there any tools out there doing a better job?

thanks for your help
Question by:ebi168
  • 5
  • 3

Author Comment

ID: 22620537
Add: compress fast but also ensures that it has good compression rate

Expert Comment

ID: 22623392
Look, you can use:
bzip2 (if unix/linux)
gzip (if unix/linux)
some others...

But no matter the one you choose, it's almost the same algorithm at the end.
You have to choose a compression/speed ratio and decide what you want : real smaller file or not ?
Example with winrar, in the options : "create default profile" and you choose something different from "best", it should be faster, bug a bigger file.
It also depends on your file content: text, raw data, something specific ? Some content will require more time(/techniques to be correctly compressed.

You can't just ask these tools to "do it faster".

By the way, maybe the idea would be to have something to cut your file into smaller pieces (1 GB parts). Thus, you might be able to multi-thread your process (depending on the disk speed/interface and so on.

If you want some help to decide: tell us what's this file, how it is generated and so on...


Author Comment

ID: 22626382
Hi this is just txt files with data rows in it, think about the export from a database table...
Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.


Author Comment

ID: 22626409
We want a high size compression (as the compressed file needs to be sent to outside network) while the faster the better. thanks

Accepted Solution

mchkorg earned 1000 total points
ID: 22632328
You can't get both :(

As it's a DB export, you should for example extract, let say, a 100 MB file from it and do some tests with several compression profile (from low to best).
Maybe using "best" is useless (two times the time for 2% better)
With some values from these real examples, you'll find the right solution for your needs.

If it's a DB export, you might also be able to generate it as multiple files (1 GB files ?) and see if some multi-threading compression (aka, start several winrar) can help. It depends on : hard disk interface / multiple disks / number of CPU/cores / memory etc.
Example : on a "common" computer, don't think about it, it's useless. On a real server (several SCSI disks using RAID), you might get 4 times faster by encoding 4 files. You "might".

Author Comment

ID: 22638351
mchkorg: thanks for the comment. let me do some research into it.

Author Comment

ID: 22638387
how about 8 duo core AMD processor, with SCSI and RAID controllers. So multithread appears the solution then.

Assisted Solution

mchkorg earned 1000 total points
ID: 22650643

1) Try to generate smaller files
2) Do some tests to find the best time/compression ratio with your compression tool
3) script it all :)


Featured Post

The 14th Annual Expert Award Winners

The results are in! Meet the top members of our 2017 Expert Awards. Congratulations to all who qualified!

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

1. Start | Settings | Printers and Faxes. 2. Go to the File Menu. 3. Select Server Properties. 4. Select the Drivers tab. 5. Select the printer you need information about. 6. Once again select Properties. You should get a list of the files requ…
I originally wrote this article to compare SARDU and YUMI, but have now added Easy2Boot, since that is the one I currently use and find the easiest to create and alter.
Finding and deleting duplicate (picture) files can be a time consuming task. My wife and I, our three kids and their families all share one dilemma: Managing our pictures. Between desktops, laptops, phones, tablets, and cameras; over the last decade…
In the video, one can understand the process of resizing images in single or bulk. Kernel Bulk Image Resizer is an easy to use tool for resizing large number of images. One can add and resize multiple images with this tool in single go. The video sh…

627 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question