Solved

tar and gzip with pipe not working

Posted on 2008-06-23
21
4,217 Views
Last Modified: 2013-11-17
AIX 5.3 64bit

I am having issues with backing up my Oracle Binaries on my Oracle Home prior to
upcoming upgrade.

Of course the overall directory structure is over 2GB.

I have tried using tar with a gzip pipe, but still cannot open it.

I ran the following

mknod mypipe p
gzip < mypipe > 10201.gz &
tar cvpf mypipe 10201
rm mypipe

Then I copied the file over with SCP.

When I undo all this.
mknod mypipe p
gunzip < filename.gz > mypipe &
tar xvpf mypipe
rm mypipe

I get

$ mknod mypipe p
$ gunzip <10201.gz> mypipe &
[1]     958544
$ tar xvpf mypipe
x 10.2.0
x 10.2.0/db_1
x 10.2.0/db_1/inventory
x 10.2.0/db_1/inventory/Queries21
x 10.2.0/db_1/inventory/Queries21/generalQueries
x 10.2.0/db_1/inventory/Queries21/generalQueries/10.1.0.3.0
x
10.2.0/db_1/inventory/Queries21/generalQueries/10.1.0.3.0/generalQueries.jar,
 89742 bytes, 176 media blocks.
tar: 0511-169 A directory checksum error on media; 804401144 not equal to
71003.
$

Smaller sized files work fine. Any thoughts?
0
Comment
Question by:greatmich
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 9
  • 7
  • 2
  • +2
21 Comments
 
LVL 14

Expert Comment

by:agriesser
ID: 21848006
What's the target filesystem type? Maybe it's limited to 2GB filesizes...
0
 
LVL 29

Expert Comment

by:MikeOM_DBA
ID: 21848035
You will find that the 2G limit is normal for some utilities.
When dealing with file > 2G use some workaround, try something like this:


mknod mypipe p
gzip -c mypipe > 10201.gz &
tar cvpf mypipe 10201
rm mypipe
 
#-- And --
mknod mypipe p
gunzip -c filename.gz > mypipe &
tar xvpf mypipe
rm mypipe

Open in new window

0
 

Author Comment

by:greatmich
ID: 21848219
oracle@DUBNRTESTAPP1:-> mknod mypipe p
oracle@DUBNRTESTAPP1:-> gzip -c mypipe > 10201.gz &
[1]     618712
oracle@DUBNRTESTAPP1:-> gzip: mypipe is not a directory or a regular file - ignored

I am trying tar and gzip  the following directory structure.

10.2.0 which  it's total size is 5.93g
0
The Ultimate Checklist to Optimize Your Website

Websites are getting bigger and complicated by the day. Video, images, custom fonts are all great for showcasing your product/service. But the price to pay in terms of reduced page load times and ultimately, decreased sales, can lead to some difficult decisions about what to cut.

 

Author Comment

by:greatmich
ID: 21848225
how would I know if it has a 2GB limit, agriesser?
0
 

Author Comment

by:greatmich
ID: 21848234
oracle@DUBNRTESTAPP1:-> ulimit -d
unlimited
oracle@DUBNRTESTAPP1:-> ulimit -m
unlimited
oracle@DUBNRTESTAPP1:->
0
 

Author Comment

by:greatmich
ID: 21848248
security/limits say
oracle:
        fsize = -1
        core = -1
        data = -1
        stack = -1
        rss = -1
        nofiles = -1
0
 
LVL 14

Expert Comment

by:agriesser
ID: 21848256
Good question...
You could bruteforce it:

dd if=/dev/zero of=/filesystem/test.bin bs=1M count=2500

If it dies before finishing the block transfer or if the resulting file test.bin is less than 2,5GB in size, you know that your filesystem doesn't support files larger than 2GB.

I've started a tar/gzip run on my AIX 5.3.3 64bit @work a few minutes ago.
The tar file was created successfully (2,8GB in size) on a jffs2 filesystem.
Now it's gzipping - will see how it fairs.
0
 
LVL 14

Expert Comment

by:agriesser
ID: 21848386
OK, works for me:

[root@viuxcl1n1 - /backup] ls -l new.tar.gz
-rw-r-----   1 root     system   2494696065 Jun 23 19:22 new.tar.gz
[root@viuxcl1n1 - /backup]

I just created a tar.gz with 2,5GB of filesize (approx).

The commands I used were:

tar cvpf new.tar some-really-big-directories
gzip new.tar

[root@viuxcl1n1 - /backup] gzip --version
gzip 1.2.4 (18 Aug 93)
Compilation options:
DIRENT UTIME STDC_HEADERS HAVE_UNISTD_H
[root@viuxcl1n1 - /backup]
0
 
LVL 14

Assisted Solution

by:agriesser
agriesser earned 100 total points
ID: 21848439
Oh, sorry, I somehow overlooked that the problem exists after using SCP to transfer the file.
Have you checked the md5sums of the file when it's on the AIX server and when it's on the destination machine?
Do they match?


csum your-file
0
 

Author Comment

by:greatmich
ID: 21848586
I will check these things tomorrow,  thank you for helping me so far.
0
 
LVL 14

Expert Comment

by:agriesser
ID: 21849100
OK, I finally really can't reproduce this problem.
I now copied over the file via SCP to my linux machine and extracted it there without problems.

Maybe your piping is the problem?
Could you try to create the file as I did above, using two separate commands (one that tars and one that gzips)?
0
 

Author Comment

by:greatmich
ID: 21849189
I can do that but because of limited space, I was trying to combined.  I am going to try to clean house and see if I can get alot of space to do your command, will not be able to do till tonight, or tomorrow morning, bear with me.
0
 
LVL 14

Expert Comment

by:agriesser
ID: 21849342
It's your problem, take the time you need to solve it :)
I'm not in a hurry...
0
 
LVL 48

Assisted Solution

by:Tintin
Tintin earned 100 total points
ID: 21849412
Why not do

tar cvf - 10.2.0 | gzip -c >10201.tar.gz
0
 
LVL 29

Accepted Solution

by:
MikeOM_DBA earned 200 total points
ID: 21849955

As I commented, the problem is not that the file system can create files > 2G, but rather some utilities on some unix flavours can only work with files <= 2G; and one case are the compress and gzip programs.

Also, sometimes the /tmp mount point is not configured for large files.

If yo do not have a special requirement for "pipes", then use a command like posted by tintin, which uses the large file work-around I suggested.

0
 
LVL 14

Expert Comment

by:agriesser
ID: 21849994
MikeOM_DBA, on AIX 5.3 there's no 2G file size limit for gzip and tar. tar files are limited to 8GB due to the USTAR header used on AIX, but that's all. gzip can handle larger files too, at least on this version of AIX.
0
 
LVL 16

Assisted Solution

by:Hanno P.S.
Hanno P.S. earned 100 total points
ID: 21854156
many gzip utilies cannot handle file sizes > 2GB

If you use gzip in a pipe, it will not have any of these problems:
   # tar cf - source | gzip -9 > backup.tar.gz
In this case, the filesystem itslef will become the limiting factor.

Also, if you copy using scp, either your scp client or server could
become the limiting factor. To avoid this also, use a pipe again:
   # tar cf - source | ssh server 'cd /path/to/dir ; gzip -9 > backup.tar.gz'
0
 

Author Comment

by:greatmich
ID: 21861511
JustUnix what is the syntax for gunzipping

  # tar cf - source | gzip -9 > backup.tar.gz
0
 

Author Comment

by:greatmich
ID: 21861515
gunzip -q < file.tar.gz | tar xvf -
this works, this all works.
0
 

Author Closing Comment

by:greatmich
ID: 31469828
Everyone helped
0
 
LVL 16

Expert Comment

by:Hanno P.S.
ID: 21863359
To untar a gzipped file locally, use
   gunzip < file.tar.gz | tar xvf -

To untar a remote file (unzipping locally, sending the compressed data accross), use
   ssh 'cat /path/to/dir/file.tar.gz' | gunzip | tar xvf -

To untar a remote file (unzipping on remote system, sedning uncompressed data accross), use
  ssh 'gunzip /path/to/dir/file.tar.gz | gunzip' | tar xvf -
0

Featured Post

On Demand Webinar: Networking for the Cloud Era

Did you know SD-WANs can improve network connectivity? Check out this webinar to learn how an SD-WAN simplified, one-click tool can help you migrate and manage data in the cloud.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When you do backups in the Solaris Operating System, the file system must be inactive. Otherwise, the output may be inconsistent. A file system is inactive when it's unmounted or it's write-locked by the operating system. Although the fssnap utility…
Let's say you need to move the data of a file system from one partition to another. This generally involves dismounting the file system, backing it up to tapes, and restoring it to a new partition. You may also copy the file system from one place to…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
Learn how to navigate the file tree with the shell. Use pwd to print the current working directory: Use ls to list a directory's contents: Use cd to change to a new directory: Use wildcards instead of typing out long directory names: Use ../ to move…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question