What is the best compression library file transfer

We have a small application we developed in house that transfers files from source to receiving server. Some of the files are large in size so the source server takes time to send them. these files come in different formats, so not just one type.

We are thinking of using a compression library as the compression lib with .net is not doing a good job.

What is the best compression library that has the best compression ratio and still acceptable in terms of cpu and time to compress/decompress?
If compression would add 2 percent but gives high compression ratio then it is probably ideal.  The receiving server is on the other side of the globe, so we have latency issue.

I noticed a file transfer agent like crashplan can transfer files in high speeds because it uses a great compression algorithm.
LVL 5
onlinerackAsked:
Who is Participating?
 
jkrCommented:
Since you apparently can use native C++, 'bzip2' might be an option to check out, see http://gnuwin32.sourceforge.net/packages/bzip2.htm
0
 
onlinerackAuthor Commented:
Thank you JKR,

Do you know which of these would provide the best compression ratio yet goo enough to use in file transfer application for large files (50+MB+)?
0
 
jkrCommented:
The binary package that comes with 'bzip2.dll' would be what you need, get it from http://gnuwin32.sourceforge.net/downlinks/bzip2-bin-zip.php

There is even a ready-made .NET wrapper for that, see https://github.com/haf/DotNetZip.Semverd and scroll down to 'The BZip2 Class Library'
0
Get expert help—faster!

Need expert help—fast? Use the Help Bell for personalized assistance getting answers to your important questions.

 
onlinerackAuthor Commented:
do you think this would be better than lzham?
0
 
jkrCommented:
Good question. All they say is 'LZHAM's compression ratio is a bit less than LZMA' (https://code.google.com/p/lzham/), but don't give any numbers. So far bzip2 is known for being quite one of the best in that area, so it's up to you to choose what's the more important part here - decompression speed or size reduction. I guess there is a point why the don't mention compression speed here.
0
 
onlinerackAuthor Commented:
Do you have any experience with 'bzip2.dll' ?
Our goal is to have the sender agent to send data to the receiver agent across high latency internet link.

So compression of files that could be large in size would be split in pecices and compressed then sent.  We will need best performance overall. Obviously smaller chunks (highly compressed) would benefit here, but agree decompression speed is good.

from what you know, what is your advice. How is bzip2.dll compare to standard compression offered by .net??

Thank you JKR for your help.
0
 
jkrCommented:
I haven't used the DLL directly in code (simply because I never had the requirement to do so), but the tools that use it are the de-facto standard compression is the OSS world. Given it's compression ratio of 15% down to 10% (for noncompressed data, that is) you might even be able to skip the step of splitting large files before compressing them, which - given the latency - would probably be the better choice. IIRC the standard compression on .NET is GZipStream, which is quite a basic one, so the bzip2 will definitely beat it regarding the compression. But, generally speaking, it's hard to give any advice, why nut just giving it a try - i.e. play around with bzip2.exe and check if the results suit your needs.
0
 
onlinerackAuthor Commented:
JKR you are a goldmine. I will go ahead and check it out and come back update you with any questions. If no questions, I will go ahead and gladly accept your valuable input.

Thank you so much. Such a great adviser with solid expertise. :)
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.