Solved

tar and gzip with pipe not working

Posted on 2008-06-23
21
4,183 Views
Last Modified: 2013-11-17
AIX 5.3 64bit

I am having issues with backing up my Oracle Binaries on my Oracle Home prior to
upcoming upgrade.

Of course the overall directory structure is over 2GB.

I have tried using tar with a gzip pipe, but still cannot open it.

I ran the following

mknod mypipe p
gzip < mypipe > 10201.gz &
tar cvpf mypipe 10201
rm mypipe

Then I copied the file over with SCP.

When I undo all this.
mknod mypipe p
gunzip < filename.gz > mypipe &
tar xvpf mypipe
rm mypipe

I get

$ mknod mypipe p
$ gunzip <10201.gz> mypipe &
[1]     958544
$ tar xvpf mypipe
x 10.2.0
x 10.2.0/db_1
x 10.2.0/db_1/inventory
x 10.2.0/db_1/inventory/Queries21
x 10.2.0/db_1/inventory/Queries21/generalQueries
x 10.2.0/db_1/inventory/Queries21/generalQueries/10.1.0.3.0
x
10.2.0/db_1/inventory/Queries21/generalQueries/10.1.0.3.0/generalQueries.jar,
 89742 bytes, 176 media blocks.
tar: 0511-169 A directory checksum error on media; 804401144 not equal to
71003.
$

Smaller sized files work fine. Any thoughts?
0
Comment
Question by:greatmich
  • 9
  • 7
  • 2
  • +2
21 Comments
 
LVL 14

Expert Comment

by:agriesser
ID: 21848006
What's the target filesystem type? Maybe it's limited to 2GB filesizes...
0
 
LVL 29

Expert Comment

by:MikeOM_DBA
ID: 21848035
You will find that the 2G limit is normal for some utilities.
When dealing with file > 2G use some workaround, try something like this:


mknod mypipe p
gzip -c mypipe > 10201.gz &
tar cvpf mypipe 10201
rm mypipe
 
#-- And --
mknod mypipe p
gunzip -c filename.gz > mypipe &
tar xvpf mypipe
rm mypipe

Open in new window

0
 

Author Comment

by:greatmich
ID: 21848219
oracle@DUBNRTESTAPP1:-> mknod mypipe p
oracle@DUBNRTESTAPP1:-> gzip -c mypipe > 10201.gz &
[1]     618712
oracle@DUBNRTESTAPP1:-> gzip: mypipe is not a directory or a regular file - ignored

I am trying tar and gzip  the following directory structure.

10.2.0 which  it's total size is 5.93g
0
Networking for the Cloud Era

Join Microsoft and Riverbed for a discussion and demonstration of enhancements to SteelConnect:
-One-click orchestration and cloud connectivity in Azure environments
-Tight integration of SD-WAN and WAN optimization capabilities
-Scalability and resiliency equal to a data center

 

Author Comment

by:greatmich
ID: 21848225
how would I know if it has a 2GB limit, agriesser?
0
 

Author Comment

by:greatmich
ID: 21848234
oracle@DUBNRTESTAPP1:-> ulimit -d
unlimited
oracle@DUBNRTESTAPP1:-> ulimit -m
unlimited
oracle@DUBNRTESTAPP1:->
0
 

Author Comment

by:greatmich
ID: 21848248
security/limits say
oracle:
        fsize = -1
        core = -1
        data = -1
        stack = -1
        rss = -1
        nofiles = -1
0
 
LVL 14

Expert Comment

by:agriesser
ID: 21848256
Good question...
You could bruteforce it:

dd if=/dev/zero of=/filesystem/test.bin bs=1M count=2500

If it dies before finishing the block transfer or if the resulting file test.bin is less than 2,5GB in size, you know that your filesystem doesn't support files larger than 2GB.

I've started a tar/gzip run on my AIX 5.3.3 64bit @work a few minutes ago.
The tar file was created successfully (2,8GB in size) on a jffs2 filesystem.
Now it's gzipping - will see how it fairs.
0
 
LVL 14

Expert Comment

by:agriesser
ID: 21848386
OK, works for me:

[root@viuxcl1n1 - /backup] ls -l new.tar.gz
-rw-r-----   1 root     system   2494696065 Jun 23 19:22 new.tar.gz
[root@viuxcl1n1 - /backup]

I just created a tar.gz with 2,5GB of filesize (approx).

The commands I used were:

tar cvpf new.tar some-really-big-directories
gzip new.tar

[root@viuxcl1n1 - /backup] gzip --version
gzip 1.2.4 (18 Aug 93)
Compilation options:
DIRENT UTIME STDC_HEADERS HAVE_UNISTD_H
[root@viuxcl1n1 - /backup]
0
 
LVL 14

Assisted Solution

by:agriesser
agriesser earned 100 total points
ID: 21848439
Oh, sorry, I somehow overlooked that the problem exists after using SCP to transfer the file.
Have you checked the md5sums of the file when it's on the AIX server and when it's on the destination machine?
Do they match?


csum your-file
0
 

Author Comment

by:greatmich
ID: 21848586
I will check these things tomorrow,  thank you for helping me so far.
0
 
LVL 14

Expert Comment

by:agriesser
ID: 21849100
OK, I finally really can't reproduce this problem.
I now copied over the file via SCP to my linux machine and extracted it there without problems.

Maybe your piping is the problem?
Could you try to create the file as I did above, using two separate commands (one that tars and one that gzips)?
0
 

Author Comment

by:greatmich
ID: 21849189
I can do that but because of limited space, I was trying to combined.  I am going to try to clean house and see if I can get alot of space to do your command, will not be able to do till tonight, or tomorrow morning, bear with me.
0
 
LVL 14

Expert Comment

by:agriesser
ID: 21849342
It's your problem, take the time you need to solve it :)
I'm not in a hurry...
0
 
LVL 48

Assisted Solution

by:Tintin
Tintin earned 100 total points
ID: 21849412
Why not do

tar cvf - 10.2.0 | gzip -c >10201.tar.gz
0
 
LVL 29

Accepted Solution

by:
MikeOM_DBA earned 200 total points
ID: 21849955

As I commented, the problem is not that the file system can create files > 2G, but rather some utilities on some unix flavours can only work with files <= 2G; and one case are the compress and gzip programs.

Also, sometimes the /tmp mount point is not configured for large files.

If yo do not have a special requirement for "pipes", then use a command like posted by tintin, which uses the large file work-around I suggested.

0
 
LVL 14

Expert Comment

by:agriesser
ID: 21849994
MikeOM_DBA, on AIX 5.3 there's no 2G file size limit for gzip and tar. tar files are limited to 8GB due to the USTAR header used on AIX, but that's all. gzip can handle larger files too, at least on this version of AIX.
0
 
LVL 16

Assisted Solution

by:Hanno Schröder
Hanno Schröder earned 100 total points
ID: 21854156
many gzip utilies cannot handle file sizes > 2GB

If you use gzip in a pipe, it will not have any of these problems:
   # tar cf - source | gzip -9 > backup.tar.gz
In this case, the filesystem itslef will become the limiting factor.

Also, if you copy using scp, either your scp client or server could
become the limiting factor. To avoid this also, use a pipe again:
   # tar cf - source | ssh server 'cd /path/to/dir ; gzip -9 > backup.tar.gz'
0
 

Author Comment

by:greatmich
ID: 21861511
JustUnix what is the syntax for gunzipping

  # tar cf - source | gzip -9 > backup.tar.gz
0
 

Author Comment

by:greatmich
ID: 21861515
gunzip -q < file.tar.gz | tar xvf -
this works, this all works.
0
 

Author Closing Comment

by:greatmich
ID: 31469828
Everyone helped
0
 
LVL 16

Expert Comment

by:Hanno Schröder
ID: 21863359
To untar a gzipped file locally, use
   gunzip < file.tar.gz | tar xvf -

To untar a remote file (unzipping locally, sending the compressed data accross), use
   ssh 'cat /path/to/dir/file.tar.gz' | gunzip | tar xvf -

To untar a remote file (unzipping on remote system, sedning uncompressed data accross), use
  ssh 'gunzip /path/to/dir/file.tar.gz | gunzip' | tar xvf -
0

Featured Post

Free Tool: ZipGrep

ZipGrep is a utility that can list and search zip (.war, .ear, .jar, etc) archives for text patterns, without the need to extract the archive's contents.

One of a set of tools we're offering as a way to say thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Installing FreeBSD… FreeBSD is a darling of an operating system. The stability and usability make it a clear choice for servers and desktops (for the cunning). Savvy?  The Ports collection makes available every popular FOSS application and packag…
I promised to write further about my project, and here I am.  First, I needed to setup the Primary Server.  You can read how in this article: Setup FreeBSD Server with full HDD encryption (http://www.experts-exchange.com/OS/Unix/BSD/FreeBSD/A_3660-S…
Learn several ways to interact with files and get file information from the bash shell. ls lists the contents of a directory: Using the -a flag displays hidden files: Using the -l flag formats the output in a long list: The file command gives us mor…
Learn how to find files with the shell using the find and locate commands. Use locate to find a needle in a haystack.: With locate, check if the file still exists.: Use find to get the actual location of the file.:

837 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question