In-Memory Compression and Decompression Using ZLIB

zlib is a free compression library (a DLL) on which the popular gzip utility is built.  In this article, we'll see how to use the zlib functions to compress and decompress data in memory; that is, without needing to use a temporary file.  We'll be coding in C++ in Visual Studio.

This seems like it would be a trivial problem, and indeed, it turns out to be relatively easy.  The difficulty is in wading through the zlib documentation and figuring out what not to do!  As with most GNU projects, the documentation is written in some sort of unix-geek-speak, and for some reason, they included lots of special options and rarely-needed functionality, but failed to provide a simple function call to handle this most basic of needs.

To get started, you will need to download the pre-built zlib DLL.  The zlib main page is here:
       http://www.winimage.com/zLibDll/index.html
and the direct link ot the ZIP file containing the DLL is here:
       http://www.winimage.com/zLibDll/zlib125dll.zip

It would be way too easy for us if the zlib geniuses had provided the needed C++ header files with the download, so you also need to download the source for the entire library (if I may make an editorial comment here: Sheesh!).  Get the ZIP file of the entire library source code here:
       http://zlib.net/zlib125.zip
We won't be building the library, but you will need to get the two files:
       zlib.h
and
       zconf.h
and put them into your project directory.

The Code

To make this example be as simple and universal as possible, we'll just throw together a console application program.  Use all of the Visual Studio App Wizard defaults, and then copy the following into the main (only) CPP file:

#include "stdafx.h"   // not actually needed
#define ZLIB_WINAPI   // actually actually needed (for linkage)

#include "windows.h"  // get BYTE et al.
#include "zlib.h"     // declare the external fns -- uses zconf.h, too
#pragma comment(lib, "zlibwapi.lib") // for access to the DLL

int GetMaxCompressedLen( int nLenSrc ) 
{
    int n16kBlocks = (nLenSrc+16383) / 16384; // round up any fraction of a block
    return ( nLenSrc + 6 + (n16kBlocks*5) );
}
int CompressData( const BYTE* abSrc, int nLenSrc, BYTE* abDst, int nLenDst )
{
    z_stream zInfo ={0};
    zInfo.total_in=  zInfo.avail_in=  nLenSrc;
    zInfo.total_out= zInfo.avail_out= nLenDst;
    zInfo.next_in= (BYTE*)abSrc;
    zInfo.next_out= abDst;

    int nErr, nRet= -1;
    nErr= deflateInit( &zInfo, Z_DEFAULT_COMPRESSION ); // zlib function
    if ( nErr == Z_OK ) {
        nErr= deflate( &zInfo, Z_FINISH );              // zlib function
        if ( nErr == Z_STREAM_END ) {
            nRet= zInfo.total_out;
        }
    }
    deflateEnd( &zInfo );    // zlib function
    return( nRet );
}

int UncompressData( const BYTE* abSrc, int nLenSrc, BYTE* abDst, int nLenDst )
{
    z_stream zInfo ={0};
    zInfo.total_in=  zInfo.avail_in=  nLenSrc;
    zInfo.total_out= zInfo.avail_out= nLenDst;
    zInfo.next_in= (BYTE*)abSrc;
    zInfo.next_out= abDst;

    int nErr, nRet= -1;
    nErr= inflateInit( &zInfo );               // zlib function
    if ( nErr == Z_OK ) {
        nErr= inflate( &zInfo, Z_FINISH );     // zlib function
        if ( nErr == Z_STREAM_END ) {
            nRet= zInfo.total_out;
        }
    }
    inflateEnd( &zInfo );   // zlib function
    return( nRet ); // -1 or len of output
}

Select all Open in new window

This defines three functions that simplify using the library for in-memory operation:


Now add the main() function in order to test the compression functions:

int main()
{
    BYTE pbSrc[]="hello hello hello hello there";

    //-------------- compress (save the original length)

    int nLenOrig= strlen( (char*)pbSrc )+1; // include terminating NULL
    int nLenDst= GetMaxCompressedLen( nLenOrig );
    BYTE* pbDst= new BYTE [nLenDst];  // alloc dest buffer

    int nLenPacked= CompressData( pbSrc, nLenOrig, pbDst, nLenDst );
    if ( nLenPacked == -1 ) return(1);  // error

    //-------------- uncompress (uses the saved original length)

    BYTE* pbPacked=   pbDst;
    BYTE* pbUnpacked= new BYTE[ nLenOrig ];

    int nLen= UncompressData( pbPacked, nLenPacked, pbUnpacked, nLenOrig );

    // breakpoint here and view pbUnpacked to confirm
    delete pbDst;            // do some cleanup
    delete pbUnpacked;
    return 0; 
}

Select all Open in new window

Summary

Programmers like zlib because it is unencumbered by patents and there are no royalties to pay.  The source code is freely available.  You can use the DLL as shown here, or link into the static library version to avoid needing to distribute another executable file.  The "deflate" (Huffman/LZ77) algorithm gets quite good compression, certainly adequate for most needs.

zlib is designed to use data streams that nearly always come from and go to disk files.  That's what makes the documentation seem complicated.  For in-memory packing and unpacking, all we needed to do was ignore all of the stream handling logic and set up the z_stream structure so that it would do everything in one go.

There is a lot more you can do with zlib, for instance, you can provide special handling to optimize the output, including selecting a compression strategy (Huffman-only or RLE-only) and you can even set up a compression dictionary in advance -- you might be able to out-perform the on-the-fly algorithms if you know you will be compressing certain specific types of data.

As I said earlier, the library contains many bells and whistles, including a complete set of file handling functions and other things that seem (to me) to be superfluous.  I hope that by providing a cut-to-the-bone example, I've made this tool a little easier for you to use.

References:

zlib Home page:
    http://www.zlib.net/index.html
zlib FAQ
    http://www.zlib.net/zlib_faq.html
zlib programming manual
    http://www.zlib.net/manual.html
gzip Home page
    http://www.gzip.org/index.html

If you want to build the zlib library (e.g., to prune out unwanted functionality), note that there are two VS projects, but you need to look for them:
    (unzip folder)\zlib-1.2.5\contrib\vstudio

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
If you liked this article and want to see more from this author, please click the Yes button near the:
      Was this article helpful?
label that is just below and to the right of this text.   Thanks!
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=