zlib.h function inflateBack
I need to use inflateBack() function from zlib.h to uncompress gz file into a buffer.
zlib.h is in C but I want to utilize c++ to use inflateBack function. I need a simple example how to use it in C++. I have seen examples but the code is too unreadable.
I cannot use any other libraries. I used Boost to do this task but in my company gz files are created by rsyslog and for some reason the headers of files are broken and non of these libraries parses the entire file. It only parses the first few block apparently.
zlib.h is in C but I want to utilize c++ to use inflateBack function. I need a simple example how to use it in C++. I have seen examples but the code is too unreadable.
I cannot use any other libraries. I used Boost to do this task but in my company gz files are created by rsyslog and for some reason the headers of files are broken and non of these libraries parses the entire file. It only parses the first few block apparently.
>> zlib.h is in C but I want to utilize c++ to use inflateBack function
So, what's the problem with doing just that in C++? As a C library zlib works just find in C++. All you should need to is link your code against the zlib library that should be installed as part of your linux distrobution. If you are using another platform building the standard zlib package from source should be as simple as building a tarball from the zlib site.
You mentioned you tried boost. Was it the zlib iostream compressor/decompressor you tried? If not it might be worth taking a look at those.
If you still need help it would be useful if you could be more specific about the issue you are having. The provision of some example could would certainly be helpful.
So, what's the problem with doing just that in C++? As a C library zlib works just find in C++. All you should need to is link your code against the zlib library that should be installed as part of your linux distrobution. If you are using another platform building the standard zlib package from source should be as simple as building a tarball from the zlib site.
You mentioned you tried boost. Was it the zlib iostream compressor/decompressor you tried? If not it might be worth taking a look at those.
If you still need help it would be useful if you could be more specific about the issue you are having. The provision of some example could would certainly be helpful.
ASKER
Thank you for taking time to take a look at this problem.
I wanted to un-gzip files using zlilb.h. An example of using the code is also written by the original writer of zlib.h, Mark Adler.
http://svn.ghostscript.com/ghostscript/tags/zlib-1.2.3/examples/gun.c
And here is the actual zlib.h file
ftp://legacy.gsfc.nasa.gov/software/fitsio/c/zlib.h
Documentation is here:
http://www.zlib.net/zlib_how.html
My problems:
I used Boost but due to broken headers of gz files in my company, it doesn't read the entire file.
I used gzread from zlib but you have to inflate the whole file in one attempt because gzseek is too slow--not good for my solution when I have to handle hundreds of TB within hours. I had to keep my buffer prohibitively large.
Reading up on it, I found that the fast way to read gz files was using inflateBack function. Using this function was not easy and I was kind of hacking the code so could never make it work. All I needed was to inflate the gz file into a buffer, process the buffer and then read it again from where I last left.
Again I am very encouraged by your help and am grateful for your attempt to look at the problem because I using to program in C/C++ until 2002 and then left it for script languages.
I wanted to un-gzip files using zlilb.h. An example of using the code is also written by the original writer of zlib.h, Mark Adler.
http://svn.ghostscript.com/ghostscript/tags/zlib-1.2.3/examples/gun.c
And here is the actual zlib.h file
ftp://legacy.gsfc.nasa.gov/software/fitsio/c/zlib.h
Documentation is here:
http://www.zlib.net/zlib_how.html
My problems:
I used Boost but due to broken headers of gz files in my company, it doesn't read the entire file.
I used gzread from zlib but you have to inflate the whole file in one attempt because gzseek is too slow--not good for my solution when I have to handle hundreds of TB within hours. I had to keep my buffer prohibitively large.
Reading up on it, I found that the fast way to read gz files was using inflateBack function. Using this function was not easy and I was kind of hacking the code so could never make it work. All I needed was to inflate the gz file into a buffer, process the buffer and then read it again from where I last left.
Again I am very encouraged by your help and am grateful for your attempt to look at the problem because I using to program in C/C++ until 2002 and then left it for script languages.
I'll have to take a look into this before I can respond with anything that is likely to be helpful. Unfortunately, I can't do that right now as I'm currently tied up with RL work stuff. If no one has managed to assist you to reach a solution by tomorrow I'll try and find some time to look at this with you.
Apologies.
Apologies.
ASKER
Well, trust me it will still be very valuable. Thanks a lot for giving me a lot to think about.
Have you tried compiling your C++ code against the C library?
I didn't follow evilrix's URL's but I have one comment. You may need to put an
extern "C"
before the C++ defintions of the C library functions
extern "C" foo ( int x );
I didn't follow evilrix's URL's but I have one comment. You may need to put an
extern "C"
before the C++ defintions of the C library functions
extern "C" foo ( int x );
ASKER
extern "C" is already included in zlib.h and all the code written by Mark Adler. While I am out of shape with C/C++ anyway, it is extremely hard for me to follow his code and his coding didn't make it any easier for me to use it. For example his inflateBack function requires two functions to be passed. Documentation is not very helpful either.
ASKER
Giving up experts? I hoped you would be able to understand the code I couldn't. Thanks for your help.
static int __zip_deflate (Bytef *dstBuf /*in/out*/,
uLongf *dstLenf /*in/out*/,
const Bytef *srcBuff /*in*/,
uLong srcLen /*in*/,
int level /*in*/,
int strategy /*in*/ )
{
z_stream stream;
int err;
stream.next_in = (Bytef*)srcBuf;
stream.avail_in = (uInt)srcLen;
stream.next_out = dstBuf;
stream.avail_out = (uInt)*dstLen;
if ((uLong)stream.avail_out != *dstLen) return Z_BUF_ERROR;
stream.zalloc = (alloc_func)0;
stream.zfree = (free_func)0;
stream.opaque = (voidpf)0;
err = deflateInit2(&stream, level, Z_DEFLATED,
MAX_WBITS, 8/*DEF_MEM_LEVEL*/, strategy);
if (err != Z_OK) return err;
err = deflate(&stream, Z_FINISH);
if (err != Z_STREAM_END) {
deflateEnd(&stream);
return err == Z_OK ? Z_BUF_ERROR : err;
}
*dstLen = stream.total_out;
err = deflateEnd(&stream);
return err;
}ASKER
@qibdip:
Kindly explain a little bit about how is works.
Kindly explain a little bit about how is works.
@farzanj sorry I did not notice you need decompression. This function compresses the buffer.
As I understand you want to utilize less memory amount. Chunked decompression is a little more complicated than simple buffer compression/decompression,
Can you explain in detail what you do not understand in this logic?
If you do not need to decompress huge data buffers and you can predict final data amount, the process is as simple as:
As I understand you want to utilize less memory amount. Chunked decompression is a little more complicated than simple buffer compression/decompression,
Can you explain in detail what you do not understand in this logic?
If you do not need to decompress huge data buffers and you can predict final data amount, the process is as simple as:
static bool __zip_decompress(void *in_buf,
size_t in_sz,
(void *out_buf,
size_t out_sz)
{
return (Z_OK == uncompress((Bytef *)out_buf,(uLongf *)out_sz,
(const Bytef *)in_buf,in_sz));
}ASKER
Thanks for your response.
Yes, I was able to uncompress using this and some other functions. I looked at all the examples as well but they were a little too hard to follow for me, mostly because the whole code has different functions but each function is not separate--they appear to be doing all the things at all the various locations.
I wasted a lot of time with it. All I want to get is uncompressed data one buffers at a time in such a way that the uncompress part doesn't start all over again instead remembers where it is at.
Makes sense? Is that code too long? I know there is too many initializations.
Yes, I was able to uncompress using this and some other functions. I looked at all the examples as well but they were a little too hard to follow for me, mostly because the whole code has different functions but each function is not separate--they appear to be doing all the things at all the various locations.
I wasted a lot of time with it. All I want to get is uncompressed data one buffers at a time in such a way that the uncompress part doesn't start all over again instead remembers where it is at.
Makes sense? Is that code too long? I know there is too many initializations.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Not ok but ok. This is exactly what I was asking. Actually the author of zlib is not a computer scientist and has a very strangely designed coding. I know it is all in there but it was too much trouble for me to separate it out. Not single function of his code is independent unit. He is doing everything in every single function. This is why I was asking this question. I will reward you anyway.
Since you asked me to look at this, I have. I don't have an answer. I have an idea.
Can you share the unreadable code with me or tell me where to find the examples that are too unreadable? Perhaps I can read them.
Hugh