Solved

tar and gzip with pipe not working

Posted on 2008-06-23
21
4,162 Views
Last Modified: 2013-11-17
AIX 5.3 64bit

I am having issues with backing up my Oracle Binaries on my Oracle Home prior to
upcoming upgrade.

Of course the overall directory structure is over 2GB.

I have tried using tar with a gzip pipe, but still cannot open it.

I ran the following

mknod mypipe p
gzip < mypipe > 10201.gz &
tar cvpf mypipe 10201
rm mypipe

Then I copied the file over with SCP.

When I undo all this.
mknod mypipe p
gunzip < filename.gz > mypipe &
tar xvpf mypipe
rm mypipe

I get

$ mknod mypipe p
$ gunzip <10201.gz> mypipe &
[1]     958544
$ tar xvpf mypipe
x 10.2.0
x 10.2.0/db_1
x 10.2.0/db_1/inventory
x 10.2.0/db_1/inventory/Queries21
x 10.2.0/db_1/inventory/Queries21/generalQueries
x 10.2.0/db_1/inventory/Queries21/generalQueries/10.1.0.3.0
x
10.2.0/db_1/inventory/Queries21/generalQueries/10.1.0.3.0/generalQueries.jar,
 89742 bytes, 176 media blocks.
tar: 0511-169 A directory checksum error on media; 804401144 not equal to
71003.
$

Smaller sized files work fine. Any thoughts?
0
Comment
Question by:greatmich
  • 9
  • 7
  • 2
  • +2
21 Comments
 
LVL 14

Expert Comment

by:agriesser
Comment Utility
What's the target filesystem type? Maybe it's limited to 2GB filesizes...
0
 
LVL 29

Expert Comment

by:MikeOM_DBA
Comment Utility
You will find that the 2G limit is normal for some utilities.
When dealing with file > 2G use some workaround, try something like this:


mknod mypipe p

gzip -c mypipe > 10201.gz &

tar cvpf mypipe 10201

rm mypipe
 

#-- And --

mknod mypipe p

gunzip -c filename.gz > mypipe &

tar xvpf mypipe

rm mypipe

Open in new window

0
 

Author Comment

by:greatmich
Comment Utility
oracle@DUBNRTESTAPP1:-> mknod mypipe p
oracle@DUBNRTESTAPP1:-> gzip -c mypipe > 10201.gz &
[1]     618712
oracle@DUBNRTESTAPP1:-> gzip: mypipe is not a directory or a regular file - ignored

I am trying tar and gzip  the following directory structure.

10.2.0 which  it's total size is 5.93g
0
 

Author Comment

by:greatmich
Comment Utility
how would I know if it has a 2GB limit, agriesser?
0
 

Author Comment

by:greatmich
Comment Utility
oracle@DUBNRTESTAPP1:-> ulimit -d
unlimited
oracle@DUBNRTESTAPP1:-> ulimit -m
unlimited
oracle@DUBNRTESTAPP1:->
0
 

Author Comment

by:greatmich
Comment Utility
security/limits say
oracle:
        fsize = -1
        core = -1
        data = -1
        stack = -1
        rss = -1
        nofiles = -1
0
 
LVL 14

Expert Comment

by:agriesser
Comment Utility
Good question...
You could bruteforce it:

dd if=/dev/zero of=/filesystem/test.bin bs=1M count=2500

If it dies before finishing the block transfer or if the resulting file test.bin is less than 2,5GB in size, you know that your filesystem doesn't support files larger than 2GB.

I've started a tar/gzip run on my AIX 5.3.3 64bit @work a few minutes ago.
The tar file was created successfully (2,8GB in size) on a jffs2 filesystem.
Now it's gzipping - will see how it fairs.
0
 
LVL 14

Expert Comment

by:agriesser
Comment Utility
OK, works for me:

[root@viuxcl1n1 - /backup] ls -l new.tar.gz
-rw-r-----   1 root     system   2494696065 Jun 23 19:22 new.tar.gz
[root@viuxcl1n1 - /backup]

I just created a tar.gz with 2,5GB of filesize (approx).

The commands I used were:

tar cvpf new.tar some-really-big-directories
gzip new.tar

[root@viuxcl1n1 - /backup] gzip --version
gzip 1.2.4 (18 Aug 93)
Compilation options:
DIRENT UTIME STDC_HEADERS HAVE_UNISTD_H
[root@viuxcl1n1 - /backup]
0
 
LVL 14

Assisted Solution

by:agriesser
agriesser earned 100 total points
Comment Utility
Oh, sorry, I somehow overlooked that the problem exists after using SCP to transfer the file.
Have you checked the md5sums of the file when it's on the AIX server and when it's on the destination machine?
Do they match?


csum your-file
0
 

Author Comment

by:greatmich
Comment Utility
I will check these things tomorrow,  thank you for helping me so far.
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 
LVL 14

Expert Comment

by:agriesser
Comment Utility
OK, I finally really can't reproduce this problem.
I now copied over the file via SCP to my linux machine and extracted it there without problems.

Maybe your piping is the problem?
Could you try to create the file as I did above, using two separate commands (one that tars and one that gzips)?
0
 

Author Comment

by:greatmich
Comment Utility
I can do that but because of limited space, I was trying to combined.  I am going to try to clean house and see if I can get alot of space to do your command, will not be able to do till tonight, or tomorrow morning, bear with me.
0
 
LVL 14

Expert Comment

by:agriesser
Comment Utility
It's your problem, take the time you need to solve it :)
I'm not in a hurry...
0
 
LVL 48

Assisted Solution

by:Tintin
Tintin earned 100 total points
Comment Utility
Why not do

tar cvf - 10.2.0 | gzip -c >10201.tar.gz
0
 
LVL 29

Accepted Solution

by:
MikeOM_DBA earned 200 total points
Comment Utility

As I commented, the problem is not that the file system can create files > 2G, but rather some utilities on some unix flavours can only work with files <= 2G; and one case are the compress and gzip programs.

Also, sometimes the /tmp mount point is not configured for large files.

If yo do not have a special requirement for "pipes", then use a command like posted by tintin, which uses the large file work-around I suggested.

0
 
LVL 14

Expert Comment

by:agriesser
Comment Utility
MikeOM_DBA, on AIX 5.3 there's no 2G file size limit for gzip and tar. tar files are limited to 8GB due to the USTAR header used on AIX, but that's all. gzip can handle larger files too, at least on this version of AIX.
0
 
LVL 16

Assisted Solution

by:Hanno Schröder
Hanno Schröder earned 100 total points
Comment Utility
many gzip utilies cannot handle file sizes > 2GB

If you use gzip in a pipe, it will not have any of these problems:
   # tar cf - source | gzip -9 > backup.tar.gz
In this case, the filesystem itslef will become the limiting factor.

Also, if you copy using scp, either your scp client or server could
become the limiting factor. To avoid this also, use a pipe again:
   # tar cf - source | ssh server 'cd /path/to/dir ; gzip -9 > backup.tar.gz'
0
 

Author Comment

by:greatmich
Comment Utility
JustUnix what is the syntax for gunzipping

  # tar cf - source | gzip -9 > backup.tar.gz
0
 

Author Comment

by:greatmich
Comment Utility
gunzip -q < file.tar.gz | tar xvf -
this works, this all works.
0
 

Author Closing Comment

by:greatmich
Comment Utility
Everyone helped
0
 
LVL 16

Expert Comment

by:Hanno Schröder
Comment Utility
To untar a gzipped file locally, use
   gunzip < file.tar.gz | tar xvf -

To untar a remote file (unzipping locally, sending the compressed data accross), use
   ssh 'cat /path/to/dir/file.tar.gz' | gunzip | tar xvf -

To untar a remote file (unzipping on remote system, sedning uncompressed data accross), use
  ssh 'gunzip /path/to/dir/file.tar.gz | gunzip' | tar xvf -
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

This tech tip describes how to install the Solaris Operating System from a tape backup that was created using the Solaris flash archive utility. I have used this procedure on the Solaris 8 and 9 OS, and it shoudl also work well on the Solaris 10 rel…
FreeBSD on EC2 FreeBSD (https://www.freebsd.org) is a robust Unix-like operating system that has been around for many years. FreeBSD is available on Amazon EC2 through Amazon Machine Images (AMIs) provided by FreeBSD developer and security office…
Learn how to get help with Linux/Unix bash shell commands. Use help to read help documents for built in bash shell commands.: Use man to interface with the online reference manuals for shell commands.: Use man to search man pages for unknown command…
This video shows how to set up a shell script to accept a positional parameter when called, pass that to a SQL script, accept the output from the statement back and then manipulate it in the Shell.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now