zlib.h function inflateBack

I need to use inflateBack() function from zlib.h to uncompress gz file into a buffer.
zlib.h is in C but I want to utilize c++ to use inflateBack function.  I need a simple example how to use it in C++.  I have seen examples but the code is too unreadable.

I cannot use any other libraries.  I used Boost to do this task but in my  company gz files are created by rsyslog and for some reason the headers of files are broken and non of these libraries parses the entire file.  It only parses the first few block apparently.
LVL 31
farzanjAsked:
Who is Participating?
 
qibdipCommented:
@farzanj  I understand you, but the problem is any sample will have the same code, because almost all of them are  based on gun.c.   How about this (gunzip.cpp)?:

http://code.ohloh.net/file?fid=QJCYw0LOZxI1fWHLeDcPlm6kDYU&cid=jifYzIND9KM

Take a look at "GunzipFile" and "gunpipe", actually this is all that you need. gunpipe reads chunks from one file and writes it to another one. You just need to remove unneeded parts, for example - If you do not have zip-file headers just comment the part that parses headers in gunpipe loop.
0
 
Hugh McCurdyCommented:
Hi,

Since you asked me to look at this, I have.  I don't have an answer.  I have an idea.

Can you share the unreadable code with me or tell me where to find the examples that are too unreadable?  Perhaps I can read them.

Hugh
0
 
evilrixSenior Software Engineer (Avast)Commented:
>> zlib.h is in C but I want to utilize c++ to use inflateBack function
So, what's the problem with doing just that in C++? As a C library zlib works just find in C++. All you should need to is link your code against the zlib library that should be installed as part of your linux distrobution. If you are using another platform building the standard zlib package from source should be as simple as building a tarball from the zlib site.

You mentioned you tried boost. Was it the zlib iostream compressor/decompressor you tried? If not it might be worth taking a look at those.

If you still need help it would be useful if you could be more specific about the issue you are having. The provision of some example could would certainly be helpful.
0
Cloud Class® Course: Certified Penetration Testing

This CPTE Certified Penetration Testing Engineer course covers everything you need to know about becoming a Certified Penetration Testing Engineer. Career Path: Professional roles include Ethical Hackers, Security Consultants, System Administrators, and Chief Security Officers.

 
farzanjAuthor Commented:
Thank you for taking time to take a look at this problem.

I wanted to un-gzip files using zlilb.h.  An example of using the code is also written by the original writer of zlib.h, Mark Adler.
http://svn.ghostscript.com/ghostscript/tags/zlib-1.2.3/examples/gun.c

And here is the actual zlib.h file
ftp://legacy.gsfc.nasa.gov/software/fitsio/c/zlib.h

Documentation is here:
http://www.zlib.net/zlib_how.html

My problems:
I used Boost but due to broken headers of gz files in my company, it doesn't read the entire file.
I used gzread from zlib but you have to inflate the whole file in one attempt because gzseek is too slow--not good for my solution when I have to handle hundreds of TB within hours.  I had to keep my buffer prohibitively large.

Reading up on it, I found that the fast way to read gz files was using inflateBack function.  Using this function was not easy and I was kind of hacking the code so could never make it work.  All I needed was to inflate the gz file into a buffer, process the buffer and then read it again from where I last left.

Again I am very encouraged by your help and am grateful for your attempt to look at the problem because I using to program in C/C++ until 2002 and then left it for script languages.
0
 
evilrixSenior Software Engineer (Avast)Commented:
I'll have to take a look into this before I can respond with anything that is likely to be helpful. Unfortunately, I can't do that right now as I'm currently tied up with RL work stuff. If no one has managed to assist you to reach a solution by tomorrow I'll try and find some time to look at this with you.

Apologies.
0
 
farzanjAuthor Commented:
Well, trust me it will still be very valuable.  Thanks a lot for giving me a lot to think about.
0
 
Hugh McCurdyCommented:
Have you tried compiling your C++ code against the C library?

I didn't follow evilrix's URL's but I have one comment.  You may need to put an
extern "C"
before the C++ defintions of the C library functions

extern "C" foo ( int x );
0
 
farzanjAuthor Commented:
extern "C" is already included in zlib.h and all the code written by Mark Adler.  While I am out of shape with C/C++ anyway, it is extremely hard for me to follow his code and his coding didn't make it any easier for me to use it.  For example his inflateBack function requires two functions to be passed.  Documentation is not very helpful either.
0
 
farzanjAuthor Commented:
Giving up experts?  I hoped you would be able to understand the code I couldn't.  Thanks for your help.
0
 
qibdipCommented:
static int __zip_deflate (Bytef *dstBuf  /*in/out*/, 
                    uLongf *dstLenf  /*in/out*/, 
                    const Bytef *srcBuff /*in*/, 
                    uLong srcLen /*in*/, 
                    int level /*in*/, 
                    int strategy  /*in*/ )

{
    z_stream stream;
    int err;
    stream.next_in = (Bytef*)srcBuf;
    stream.avail_in = (uInt)srcLen;
    stream.next_out = dstBuf;
    stream.avail_out = (uInt)*dstLen;
    if ((uLong)stream.avail_out != *dstLen) return Z_BUF_ERROR;

    stream.zalloc = (alloc_func)0;
    stream.zfree = (free_func)0;
    stream.opaque = (voidpf)0;

    err = deflateInit2(&stream, level, Z_DEFLATED, 
          MAX_WBITS, 8/*DEF_MEM_LEVEL*/, strategy);

    if (err != Z_OK) return err;

    err = deflate(&stream, Z_FINISH);
    if (err != Z_STREAM_END) {
        deflateEnd(&stream);
        return err == Z_OK ? Z_BUF_ERROR : err;
    }
    *dstLen = stream.total_out;

    err = deflateEnd(&stream);
    return err;
}

Open in new window

0
 
farzanjAuthor Commented:
@qibdip:

Kindly explain a little bit about how is works.
0
 
qibdipCommented:
@farzanj sorry I did not notice you need decompression. This function compresses the buffer.  

As I understand you want to utilize less memory amount. Chunked decompression is a little more complicated than simple buffer compression/decompression, and there is not any sense to post full code here. You said you have reviewed gun.c example, but the full logic of inflateBack process is presented over there. inflateBackInit->[inflateBack]->inflateBackEnd.

Can you explain in detail what you do not understand in this logic?

If you do not need to decompress huge data buffers and you can predict final data amount, the process is as simple as:

static bool  __zip_decompress(void *in_buf, 
                                              size_t in_sz, 
                                              (void *out_buf, 
                                              size_t out_sz)
{
   return (Z_OK == uncompress((Bytef *)out_buf,(uLongf *)out_sz,
                 (const Bytef *)in_buf,in_sz));
}

Open in new window

0
 
farzanjAuthor Commented:
Thanks for your response.

Yes, I was able to uncompress using this and some other functions.  I looked at all the examples as well but they were a little too hard to follow for me, mostly because the whole code has different functions but each function is not separate--they appear to be doing all the things at all the various locations.

I wasted a lot of time with it.  All I want to get is uncompressed  data one buffers at a time in such a way that the uncompress part doesn't start all over again instead remembers where it is at.

Makes sense?  Is that code too long?  I know there is too many initializations.
0
 
farzanjAuthor Commented:
Not ok but ok.  This is exactly what I was asking.  Actually the author of zlib is not a computer scientist and has a very strangely designed coding.  I know it is all in there but it was too much trouble for me to separate it out.  Not single function of his code is independent unit.  He is doing everything in every single function.  This is why I was asking this question.  I will reward you anyway.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.