What is the best compression library file transfer

We have a small application we developed in house that transfers files from source to receiving server. Some of the files are large in size so the source server takes time to send them. these files come in different formats, so not just one type.

We are thinking of using a compression library as the compression lib with .net is not doing a good job.

What is the best compression library that has the best compression ratio and still acceptable in terms of cpu and time to compress/decompress?
If compression would add 2 percent but gives high compression ratio then it is probably ideal.  The receiving server is on the other side of the globe, so we have latency issue.

I noticed a file transfer agent like crashplan can transfer files in high speeds because it uses a great compression algorithm.
LVL 5
onlinerackAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

jkrCommented:
Since you apparently can use native C++, 'bzip2' might be an option to check out, see http://gnuwin32.sourceforge.net/packages/bzip2.htm
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
onlinerackAuthor Commented:
Thank you JKR,

Do you know which of these would provide the best compression ratio yet goo enough to use in file transfer application for large files (50+MB+)?
0
jkrCommented:
The binary package that comes with 'bzip2.dll' would be what you need, get it from http://gnuwin32.sourceforge.net/downlinks/bzip2-bin-zip.php

There is even a ready-made .NET wrapper for that, see https://github.com/haf/DotNetZip.Semverd and scroll down to 'The BZip2 Class Library'
0
OWASP Proactive Controls

Learn the most important control and control categories that every architect and developer should include in their projects.

onlinerackAuthor Commented:
do you think this would be better than lzham?
0
jkrCommented:
Good question. All they say is 'LZHAM's compression ratio is a bit less than LZMA' (https://code.google.com/p/lzham/), but don't give any numbers. So far bzip2 is known for being quite one of the best in that area, so it's up to you to choose what's the more important part here - decompression speed or size reduction. I guess there is a point why the don't mention compression speed here.
0
onlinerackAuthor Commented:
Do you have any experience with 'bzip2.dll' ?
Our goal is to have the sender agent to send data to the receiver agent across high latency internet link.

So compression of files that could be large in size would be split in pecices and compressed then sent.  We will need best performance overall. Obviously smaller chunks (highly compressed) would benefit here, but agree decompression speed is good.

from what you know, what is your advice. How is bzip2.dll compare to standard compression offered by .net??

Thank you JKR for your help.
0
jkrCommented:
I haven't used the DLL directly in code (simply because I never had the requirement to do so), but the tools that use it are the de-facto standard compression is the OSS world. Given it's compression ratio of 15% down to 10% (for noncompressed data, that is) you might even be able to skip the step of splitting large files before compressing them, which - given the latency - would probably be the better choice. IIRC the standard compression on .NET is GZipStream, which is quite a basic one, so the bzip2 will definitely beat it regarding the compression. But, generally speaking, it's hard to give any advice, why nut just giving it a try - i.e. play around with bzip2.exe and check if the results suit your needs.
0
onlinerackAuthor Commented:
JKR you are a goldmine. I will go ahead and check it out and come back update you with any questions. If no questions, I will go ahead and gladly accept your valuable input.

Thank you so much. Such a great adviser with solid expertise. :)
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
.NET Programming

From novice to tech pro — start learning today.