General Question on Compression.

Posted on 2005-04-21
Last Modified: 2010-04-17
I am 70% Computer Geek and 30% still human....a very dangerous combination. My question is more for the community to repond to for general learning....we  can all still learn....

I would like expert opinion on ........With the availability of Memory now, at low compression of any type worth the risks involved in recovery........especially when the compressed data is often (foolishly) the single valuable backup of someones irreplaceable data. We will soon have ,basically, unlimited dynamic memory. We are closing in on  Terebyte hard drives......that's 10^+12 .  If you stack pennies up and use just 1 Gig worth you can go to the sun...try comprehending multples of 1000 Gigs and more!!!! I am sure we will get into petabyte,exa,zetta, etc. etc.

And if the risk of compression is worth it what is the best method???

Just food for thought.....

Question by:rspalding
    LVL 10

    Accepted Solution

    Well it is cerainly worth it. Disk space is cheap but that is no reason to unnecessarily clog them up, disk space is still not free. As well I think the idea that that decompression is risky is a non-starter if we're using proper algorithms and programs. Sure I would not trust my client's corporate data to a compression program my nephew wrote as an excercise in Huffman Coding, but if you use a proper compression program/method then it is fine. I don't think it is any riskier than trusting one's data to say a filesystem such as FAT32 or NTFS. As well, bandwidth is still not even cheap, so we certainly must turn to compression for any sort of transfer or large data files. Well, these are just my thoughs/
    LVL 32

    Assisted Solution

    As a general rule, I would definitely recommend against using compression unless you are faced with a special circumstance, i.e. having to store large amounts of data onto limited disk space. Long-term viability is definitely put at risk when you compress your data. Not worth it in most cases.
    LVL 25

    Assisted Solution

    I have no problem with compressing data, as NetworkArchitek has said, as long as it's with the use of a well known/trustworthy application.
    For the most important data of mine however, I always back them up to some sort of storage media (usually just a DVD-R/W), uncompressed.

    As long as errors don't occur while compressing the data, then it makes sense that it's even less likely going to get corrupted, than it would if it wasn't compressed; reason for this would be that it requires less space (and thus, less sectors on your HDD, for example), therefore, any sector-errors are less probable to effect the compressed file.. do you see what I mean?  ;-)
    LVL 10

    Expert Comment

    InteractiveMind makes an excellant point. The compressed file has far less chance of being corrupted. We have to remember that compression is not a "willy nilly" way of doing things, in it is rooted in structured mathematical and tree based algorithms. There is a difference in file compression and lossy compression as that found in media files such as mp3's and mpg's.
    LVL 16

    Assisted Solution

    Its the old speed against size issue. If you want maximum speed access to your data, dont compress. If you are prepared to trade one for the other, its just where does the balance point appear for you? If the process you are running would generate a petabyte of uncompressed data per day then compress it, if this would take a decade, dont!


    Author Comment

    Yep I agree you all have good points......thanks for your opinions....Rick S
    LVL 25

    Expert Comment


    Featured Post

    Threat Intelligence Starter Resources

    Integrating threat intelligence can be challenging, and not all companies are ready. These resources can help you build awareness and prepare for defense.

    Join & Write a Comment

    Suggested Solutions

    Title # Comments Views Activity
    isEverywhere  challenge 19 49
    scoreUp challenge 14 40
    count8 challlenge 13 69
    Microsoft C++ code failing in executable that worked 9 22
    This article will show, step by step, how to integrate R code into a R Sweave document
    Whether you've completed a degree in computer sciences or you're a self-taught programmer, writing your first lines of code in the real world is always a challenge. Here are some of the most common pitfalls for new programmers.
    In this fourth video of the Xpdf series, we discuss and demonstrate the PDFinfo utility, which retrieves the contents of a PDF's Info Dictionary, as well as some other information, including the page count. We show how to isolate the page count in a…
    In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…

    745 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    14 Experts available now in Live!

    Get 1:1 Help Now