[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

zlib.h function inflateBack

Posted on 2012-08-17
14
Medium Priority
?
754 Views
Last Modified: 2012-12-05
I need to use inflateBack() function from zlib.h to uncompress gz file into a buffer.
zlib.h is in C but I want to utilize c++ to use inflateBack function.  I need a simple example how to use it in C++.  I have seen examples but the code is too unreadable.

I cannot use any other libraries.  I used Boost to do this task but in my  company gz files are created by rsyslog and for some reason the headers of files are broken and non of these libraries parses the entire file.  It only parses the first few block apparently.
0
Comment
Question by:farzanj
  • 7
  • 3
  • 2
  • +1
14 Comments
 
LVL 13

Expert Comment

by:Hugh McCurdy
ID: 38329817
Hi,

Since you asked me to look at this, I have.  I don't have an answer.  I have an idea.

Can you share the unreadable code with me or tell me where to find the examples that are too unreadable?  Perhaps I can read them.

Hugh
0
 
LVL 40

Expert Comment

by:evilrix
ID: 38330126
>> zlib.h is in C but I want to utilize c++ to use inflateBack function
So, what's the problem with doing just that in C++? As a C library zlib works just find in C++. All you should need to is link your code against the zlib library that should be installed as part of your linux distrobution. If you are using another platform building the standard zlib package from source should be as simple as building a tarball from the zlib site.

You mentioned you tried boost. Was it the zlib iostream compressor/decompressor you tried? If not it might be worth taking a look at those.

If you still need help it would be useful if you could be more specific about the issue you are having. The provision of some example could would certainly be helpful.
0
 
LVL 31

Author Comment

by:farzanj
ID: 38330216
Thank you for taking time to take a look at this problem.

I wanted to un-gzip files using zlilb.h.  An example of using the code is also written by the original writer of zlib.h, Mark Adler.
http://svn.ghostscript.com/ghostscript/tags/zlib-1.2.3/examples/gun.c

And here is the actual zlib.h file
ftp://legacy.gsfc.nasa.gov/software/fitsio/c/zlib.h

Documentation is here:
http://www.zlib.net/zlib_how.html

My problems:
I used Boost but due to broken headers of gz files in my company, it doesn't read the entire file.
I used gzread from zlib but you have to inflate the whole file in one attempt because gzseek is too slow--not good for my solution when I have to handle hundreds of TB within hours.  I had to keep my buffer prohibitively large.

Reading up on it, I found that the fast way to read gz files was using inflateBack function.  Using this function was not easy and I was kind of hacking the code so could never make it work.  All I needed was to inflate the gz file into a buffer, process the buffer and then read it again from where I last left.

Again I am very encouraged by your help and am grateful for your attempt to look at the problem because I using to program in C/C++ until 2002 and then left it for script languages.
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 40

Expert Comment

by:evilrix
ID: 38330972
I'll have to take a look into this before I can respond with anything that is likely to be helpful. Unfortunately, I can't do that right now as I'm currently tied up with RL work stuff. If no one has managed to assist you to reach a solution by tomorrow I'll try and find some time to look at this with you.

Apologies.
0
 
LVL 31

Author Comment

by:farzanj
ID: 38330991
Well, trust me it will still be very valuable.  Thanks a lot for giving me a lot to think about.
0
 
LVL 13

Expert Comment

by:Hugh McCurdy
ID: 38331106
Have you tried compiling your C++ code against the C library?

I didn't follow evilrix's URL's but I have one comment.  You may need to put an
extern "C"
before the C++ defintions of the C library functions

extern "C" foo ( int x );
0
 
LVL 31

Author Comment

by:farzanj
ID: 38331219
extern "C" is already included in zlib.h and all the code written by Mark Adler.  While I am out of shape with C/C++ anyway, it is extremely hard for me to follow his code and his coding didn't make it any easier for me to use it.  For example his inflateBack function requires two functions to be passed.  Documentation is not very helpful either.
0
 
LVL 31

Author Comment

by:farzanj
ID: 38467245
Giving up experts?  I hoped you would be able to understand the code I couldn't.  Thanks for your help.
0
 
LVL 1

Expert Comment

by:qibdip
ID: 38562885
static int __zip_deflate (Bytef *dstBuf  /*in/out*/, 
                    uLongf *dstLenf  /*in/out*/, 
                    const Bytef *srcBuff /*in*/, 
                    uLong srcLen /*in*/, 
                    int level /*in*/, 
                    int strategy  /*in*/ )

{
    z_stream stream;
    int err;
    stream.next_in = (Bytef*)srcBuf;
    stream.avail_in = (uInt)srcLen;
    stream.next_out = dstBuf;
    stream.avail_out = (uInt)*dstLen;
    if ((uLong)stream.avail_out != *dstLen) return Z_BUF_ERROR;

    stream.zalloc = (alloc_func)0;
    stream.zfree = (free_func)0;
    stream.opaque = (voidpf)0;

    err = deflateInit2(&stream, level, Z_DEFLATED, 
          MAX_WBITS, 8/*DEF_MEM_LEVEL*/, strategy);

    if (err != Z_OK) return err;

    err = deflate(&stream, Z_FINISH);
    if (err != Z_STREAM_END) {
        deflateEnd(&stream);
        return err == Z_OK ? Z_BUF_ERROR : err;
    }
    *dstLen = stream.total_out;

    err = deflateEnd(&stream);
    return err;
}

Open in new window

0
 
LVL 31

Author Comment

by:farzanj
ID: 38573205
@qibdip:

Kindly explain a little bit about how is works.
0
 
LVL 1

Expert Comment

by:qibdip
ID: 38574932
@farzanj sorry I did not notice you need decompression. This function compresses the buffer.  

As I understand you want to utilize less memory amount. Chunked decompression is a little more complicated than simple buffer compression/decompression, and there is not any sense to post full code here. You said you have reviewed gun.c example, but the full logic of inflateBack process is presented over there. inflateBackInit->[inflateBack]->inflateBackEnd.

Can you explain in detail what you do not understand in this logic?

If you do not need to decompress huge data buffers and you can predict final data amount, the process is as simple as:

static bool  __zip_decompress(void *in_buf, 
                                              size_t in_sz, 
                                              (void *out_buf, 
                                              size_t out_sz)
{
   return (Z_OK == uncompress((Bytef *)out_buf,(uLongf *)out_sz,
                 (const Bytef *)in_buf,in_sz));
}

Open in new window

0
 
LVL 31

Author Comment

by:farzanj
ID: 38576464
Thanks for your response.

Yes, I was able to uncompress using this and some other functions.  I looked at all the examples as well but they were a little too hard to follow for me, mostly because the whole code has different functions but each function is not separate--they appear to be doing all the things at all the various locations.

I wasted a lot of time with it.  All I want to get is uncompressed  data one buffers at a time in such a way that the uncompress part doesn't start all over again instead remembers where it is at.

Makes sense?  Is that code too long?  I know there is too many initializations.
0
 
LVL 1

Accepted Solution

by:
qibdip earned 2000 total points
ID: 38578815
@farzanj  I understand you, but the problem is any sample will have the same code, because almost all of them are  based on gun.c.   How about this (gunzip.cpp)?:

http://code.ohloh.net/file?fid=QJCYw0LOZxI1fWHLeDcPlm6kDYU&cid=jifYzIND9KM

Take a look at "GunzipFile" and "gunpipe", actually this is all that you need. gunpipe reads chunks from one file and writes it to another one. You just need to remove unneeded parts, for example - If you do not have zip-file headers just comment the part that parses headers in gunpipe loop.
0
 
LVL 31

Author Closing Comment

by:farzanj
ID: 38662832
Not ok but ok.  This is exactly what I was asking.  Actually the author of zlib is not a computer scientist and has a very strangely designed coding.  I know it is all in there but it was too much trouble for me to separate it out.  Not single function of his code is independent unit.  He is doing everything in every single function.  This is why I was asking this question.  I will reward you anyway.
0

Featured Post

Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Summary: This tutorial covers some basics of pointer, pointer arithmetic and function pointer. What is a pointer: A pointer is a variable which holds an address. This address might be address of another variable/address of devices/address of fu…
Article by: evilrix
Looking for a way to avoid searching through large data sets for data that doesn't exist? A Bloom Filter might be what you need. This data structure is a probabilistic filter that allows you to avoid unnecessary searches when you know the data defin…
The goal of this video is to provide viewers with basic examples to understand recursion in the C programming language.
The goal of the tutorial is to teach the user how to use functions in C++. The video will cover how to define functions, how to call functions and how to create functions prototypes. Microsoft Visual C++ 2010 Express will be used as a text editor an…
Suggested Courses

873 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question