• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 4301
  • Last Modified:

tar and gzip with pipe not working

AIX 5.3 64bit

I am having issues with backing up my Oracle Binaries on my Oracle Home prior to
upcoming upgrade.

Of course the overall directory structure is over 2GB.

I have tried using tar with a gzip pipe, but still cannot open it.

I ran the following

mknod mypipe p
gzip < mypipe > 10201.gz &
tar cvpf mypipe 10201
rm mypipe

Then I copied the file over with SCP.

When I undo all this.
mknod mypipe p
gunzip < filename.gz > mypipe &
tar xvpf mypipe
rm mypipe

I get

$ mknod mypipe p
$ gunzip <10201.gz> mypipe &
[1]     958544
$ tar xvpf mypipe
x 10.2.0
x 10.2.0/db_1
x 10.2.0/db_1/inventory
x 10.2.0/db_1/inventory/Queries21
x 10.2.0/db_1/inventory/Queries21/generalQueries
x 10.2.0/db_1/inventory/Queries21/generalQueries/10.1.0.3.0
x
10.2.0/db_1/inventory/Queries21/generalQueries/10.1.0.3.0/generalQueries.jar,
 89742 bytes, 176 media blocks.
tar: 0511-169 A directory checksum error on media; 804401144 not equal to
71003.
$

Smaller sized files work fine. Any thoughts?
0
greatmich
Asked:
greatmich
  • 9
  • 7
  • 2
  • +2
4 Solutions
 
agriesserCommented:
What's the target filesystem type? Maybe it's limited to 2GB filesizes...
0
 
MikeOM_DBACommented:
You will find that the 2G limit is normal for some utilities.
When dealing with file > 2G use some workaround, try something like this:


mknod mypipe p
gzip -c mypipe > 10201.gz &
tar cvpf mypipe 10201
rm mypipe
 
#-- And --
mknod mypipe p
gunzip -c filename.gz > mypipe &
tar xvpf mypipe
rm mypipe

Open in new window

0
 
greatmichAuthor Commented:
oracle@DUBNRTESTAPP1:-> mknod mypipe p
oracle@DUBNRTESTAPP1:-> gzip -c mypipe > 10201.gz &
[1]     618712
oracle@DUBNRTESTAPP1:-> gzip: mypipe is not a directory or a regular file - ignored

I am trying tar and gzip  the following directory structure.

10.2.0 which  it's total size is 5.93g
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
greatmichAuthor Commented:
how would I know if it has a 2GB limit, agriesser?
0
 
greatmichAuthor Commented:
oracle@DUBNRTESTAPP1:-> ulimit -d
unlimited
oracle@DUBNRTESTAPP1:-> ulimit -m
unlimited
oracle@DUBNRTESTAPP1:->
0
 
greatmichAuthor Commented:
security/limits say
oracle:
        fsize = -1
        core = -1
        data = -1
        stack = -1
        rss = -1
        nofiles = -1
0
 
agriesserCommented:
Good question...
You could bruteforce it:

dd if=/dev/zero of=/filesystem/test.bin bs=1M count=2500

If it dies before finishing the block transfer or if the resulting file test.bin is less than 2,5GB in size, you know that your filesystem doesn't support files larger than 2GB.

I've started a tar/gzip run on my AIX 5.3.3 64bit @work a few minutes ago.
The tar file was created successfully (2,8GB in size) on a jffs2 filesystem.
Now it's gzipping - will see how it fairs.
0
 
agriesserCommented:
OK, works for me:

[root@viuxcl1n1 - /backup] ls -l new.tar.gz
-rw-r-----   1 root     system   2494696065 Jun 23 19:22 new.tar.gz
[root@viuxcl1n1 - /backup]

I just created a tar.gz with 2,5GB of filesize (approx).

The commands I used were:

tar cvpf new.tar some-really-big-directories
gzip new.tar

[root@viuxcl1n1 - /backup] gzip --version
gzip 1.2.4 (18 Aug 93)
Compilation options:
DIRENT UTIME STDC_HEADERS HAVE_UNISTD_H
[root@viuxcl1n1 - /backup]
0
 
agriesserCommented:
Oh, sorry, I somehow overlooked that the problem exists after using SCP to transfer the file.
Have you checked the md5sums of the file when it's on the AIX server and when it's on the destination machine?
Do they match?


csum your-file
0
 
greatmichAuthor Commented:
I will check these things tomorrow,  thank you for helping me so far.
0
 
agriesserCommented:
OK, I finally really can't reproduce this problem.
I now copied over the file via SCP to my linux machine and extracted it there without problems.

Maybe your piping is the problem?
Could you try to create the file as I did above, using two separate commands (one that tars and one that gzips)?
0
 
greatmichAuthor Commented:
I can do that but because of limited space, I was trying to combined.  I am going to try to clean house and see if I can get alot of space to do your command, will not be able to do till tonight, or tomorrow morning, bear with me.
0
 
agriesserCommented:
It's your problem, take the time you need to solve it :)
I'm not in a hurry...
0
 
TintinCommented:
Why not do

tar cvf - 10.2.0 | gzip -c >10201.tar.gz
0
 
MikeOM_DBACommented:

As I commented, the problem is not that the file system can create files > 2G, but rather some utilities on some unix flavours can only work with files <= 2G; and one case are the compress and gzip programs.

Also, sometimes the /tmp mount point is not configured for large files.

If yo do not have a special requirement for "pipes", then use a command like posted by tintin, which uses the large file work-around I suggested.

0
 
agriesserCommented:
MikeOM_DBA, on AIX 5.3 there's no 2G file size limit for gzip and tar. tar files are limited to 8GB due to the USTAR header used on AIX, but that's all. gzip can handle larger files too, at least on this version of AIX.
0
 
Hanno P.S.IT Consultant and Infrastructure ArchitectCommented:
many gzip utilies cannot handle file sizes > 2GB

If you use gzip in a pipe, it will not have any of these problems:
   # tar cf - source | gzip -9 > backup.tar.gz
In this case, the filesystem itslef will become the limiting factor.

Also, if you copy using scp, either your scp client or server could
become the limiting factor. To avoid this also, use a pipe again:
   # tar cf - source | ssh server 'cd /path/to/dir ; gzip -9 > backup.tar.gz'
0
 
greatmichAuthor Commented:
JustUnix what is the syntax for gunzipping

  # tar cf - source | gzip -9 > backup.tar.gz
0
 
greatmichAuthor Commented:
gunzip -q < file.tar.gz | tar xvf -
this works, this all works.
0
 
greatmichAuthor Commented:
Everyone helped
0
 
Hanno P.S.IT Consultant and Infrastructure ArchitectCommented:
To untar a gzipped file locally, use
   gunzip < file.tar.gz | tar xvf -

To untar a remote file (unzipping locally, sending the compressed data accross), use
   ssh 'cat /path/to/dir/file.tar.gz' | gunzip | tar xvf -

To untar a remote file (unzipping on remote system, sedning uncompressed data accross), use
  ssh 'gunzip /path/to/dir/file.tar.gz | gunzip' | tar xvf -
0

Featured Post

[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

  • 9
  • 7
  • 2
  • +2
Tackle projects and never again get stuck behind a technical roadblock.
Join Now