Problem unzipping file on Linux - need PK compat. 4.5

you probably want "gunzip" for a "*.gz" file, nout "unzip"

sorry - that was a mistype on my part - the file ends in .zip and was created with winzip

Try running this on it, and post the output please:

unzip -u
unzip -t


Thanks

~K Black

Hi,

  Could you recompiling unzip with the "-DLARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64" option?
This will allow the utilities to handle uncompressed data files greater than 2 GB in size, as long as the total size of the archive containing them is less than 2 GB. (The operating system also must have support for large files; for Linux, this involves the kernel [2.4 or later], the file system [at least Reiser and ext2 are supported], the C library [glibc 2.x], and possibly other file utilities [ls, rm, etc.] and the shell itself [bash, tcsh, etc.] if redirection is involved.)

   For more details, please check the following URL:
http://www.info-zip.org/pub/infozip/FAQ.html

Wesly

Unzip -u output:

[pascow@server tmp]$ sudo unzip -u lnx111001.zip
Archive:  lnx111001.zip
   skipping: LNX111001-System-flat.vmdk  need PK compat. v4.5 (can do v2.1)
 extracting: Windows Server 2003, Standard Edition.vmx.lck  
  inflating: LNX111001-System.vmdk  
  inflating: nvram                  
  inflating: vmware.log              
  inflating: Windows Server 2003, Standard Edition.vmx

unzip -t output
[pascow@server tmp]$ sudo unzip -t lnx111001.zip
Archive:  lnx111001.zip
   skipping: LNX111001-System-flat.vmdk  need PK compat. v4.5 (can do v2.1)
    testing: Windows Server 2003, Standard Edition.vmx.lck   OK
    testing: LNX111001-System.vmdk    OK
    testing: nvram                    OK
    testing: vmware.log               OK
    testing: Windows Server 2003, Standard Edition.vmx   OK
No errors detected in lnx111001.zip for the 5 files tested.
1 file skipped because of unsupported compression or encoding.

Reply to Wesley Chan:

I have tried compiling with those flags. The site mentioned did not make it clear how this should be done, so I edited the Makefile and changed line 776 to read
CF="-O3 -Wall -I. -DASM_CRC $(LOC) -DLARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64"\

I then compiled unzip, and saw the additional parameters passed to all of the GCC lines, for example:
gcc -c -O3 -Wall -I. -DASM_CRC  -DLARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DSFX unzipsfx.c

Now running this new unzip, firstly it just sits for quite a long time before doing anything. Here is the output of the -t run:

[pascow@server tmp]$ ~pascow/tmp/unzip-5.51/unzip -t lnx111001.zip
Archive:  lnx111001.zip
   skipping: LNX111001-System-flat.vmdk  need PK compat. v4.5 (can do v2.1)
    testing: Windows Server 2003, Standard Edition.vmx.lck   OK
    testing: LNX111001-System.vmdk       testing: nvram                  
  error:  invalid compressed data to inflate
    testing: vmware.log                  testing: Windows Server 2003, Standard Edition.vmx  At least one error was detected in lnx111001.zip.
1 file skipped because of unsupported compression or encoding.

This is different than the output I got before, but two things are notable...

1. It still skips LNX111001-System-flat due to compat issues
2. nvram now fails as being invalid compressed data

-u just does nothing. No processer is being used, and nothing is updated to the screen.

Hi,

   Could you try using gzip for Windows to compress it?
You can download gzip here (version 1.3.5 for files > 4GB):
http://gnuwin32.sourceforge.net/downlinks/gzip-bin.php

You can check the following URL for more details
http://gnuwin32.sourceforge.net/install.html

Wesly

I can't control the compress side of the project unfortunately. These files are being supplied by a different business unit.

Any chance that the binary files in question have been trashed by FTPing them in "text" mode?

You could rule this out by confirming that the size in bytes exactly matches on both systems, or (better) computing checksums.

If you only have access to the unix version, you could check how many <CR> (0x0d) characters it contains. If the answer is zero, then it has probably been corrupted. (Would normally expect a compressed binary file to contain all 256 characters in roughly equal proportions. But text-mode FTP from Windows to unix will mistakenly remove <CR> characters).

The files themselves are fine. It's just a problem with zip files containing files over a certain size (uncompressed). This apparantly has to do with additional flags added by PKWare in PKZip 4.5 in 2001.

I've done some testing, and even Winzip 8.x can't extract these files, but they extract fine on Winzip 9.0 - this appears to be new functionality in 9.

Also, with regards to FTP and text mode, I've actually transfered the files to the machine using FTP, SCP and Samba when I thought I was getting corrupt files. The most recent files were SCP'd :)

Or can you download to Windows machine and use winzip to uzip it. If it fails, then the zip file is corrupted.
If it successes, then upload to Linux machine.

Wesly

That would work, but there are 2 problems...

1. The files are on our dev network. We don't have any windows boxes on that network, so the process would be burn DVD on dev, copy from DVD to windows box, buy external DVD writer for windows box, unzip files, burn to DVD, move back to dev.

2. Even if we could transfer the files to a Windows machine over the network, this would mean moving 3GB over the network to the Windows box, then moving about 8.5GB back from the Windows box to the Linux machine. This is all over a 10Mb network as the machine in question isn't on the main switched network. This would be a little time consuming :)

Ideally, I'd like to get this working on Linux. If that's not possible, it looks like my cheapest solution is a Maxtor external hard drive and doing this by USB. But that's money I'd rather not spend because of a limitation in unzip / a non-standard implementation in Winzip :)

OK, my apologies for distracting everyone with theories about the file being corrupted.

The release notes at http://www.info-zip.org/UnZip.html say:

"There may not ever be another major release of UnZip, although there might well be another minor release to add compatibility with PKWARE's large-file-support kludge"

From the tone of this comment, it seems pretty clear that their version of "unzip" doesn't support the new PKWare extensions and isn't going to be upgraded in the near future.

Anonymouslemming, I think you will have no choice but to push back on the other people in your organisation and demand the files in a different format. You could make noises about the "zip" format being "proprietary", "poorly documented" and "obsolescent".

Hi,
   I agree with Matt_Avery. Push back to ask them to use gzip because of those troubles. ^_^

Wesly

Matt_Avery:

Thanks for the comments. Much as I'd love to push back with this, we're a windows shop, so any such noises would be met with hostility. As I'm responsible for trying to bring Linux onboard, I can't really ruffle too many feathers. I think this problem is going to be best resolved with a combination of a sneakernet and a mastercard :D

Right - now for the important part - How do we close this down and split points ? I'm inclined to split them 50/50 between Matt_Avery and wesley_chen due to the ongoing attention that I've received from yourselves and the advice that you've given. I just don't really want this to become a featured solution.

Would it be possible for you to both post something to the effect that this is not possible on Linux at this time due to the PKWare extensions and then I can accept those.

I suggest you check many of the file splitting utilities available on the web such as http://www.freebyte.com/hjsplit/ HJSplit

You might be able to split larger files into mutliple parts, as a albeit dirty work-a-round for your problem.

I think Winzip also suffers a 2GB limitation on locking a file to uncompress.

Regards,

~K Black
Irvine, Ca.

I think before determining windows has better compression utilities, I'd consider the compression levels achieved by each of the utilities. It's commonly known that Unix compression utilities can reach a much higher level of compression with more options.

Whether it suites you in this case shouldn't be made to reflect on the OS or the community, I think.

My .02

Regards

~KB

p7zip would have worked for this.
http://p7zip.sourceforge.net/