How to check when zip function completed?

Hi,
I have a perl scrip to zip a folder:
-------------------------------------
tar -cf folder.tar folder;
gzip -f folder.tar;
-------------------------------------
After that I remove the folder and all the files in that folder. However, when there are a lot of files in the folder and need some time to gzip the folder but the process continue running to the following steps to remove the files and folder. This caused the gzip process zip the empty folder.
How to make sure that the gzip is complete before removing the files and folder?
Please help.
Thanks.

Best Regards,
Ivonne
ivonne1094Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

shivsaCommented:
$check = `gzip -f folder.tar`;
if ($check > 0 ) {
    print "zip is complete \n";
}
0
ivonne1094Author Commented:
Hi shivsa,
Thanks....but how about if the gzip function still running? How should I change my code to let the gzip finish then only removing the folder?
Thanks.
0
shivsaCommented:
u can do more check

$check = `ps -aef | grep gzip`;
if ( $check ~= /folder.tar/ ) {
   print " gzip still running \n";
}
0
Cloud Class® Course: Microsoft Exchange Server

The MCTS: Microsoft Exchange Server 2010 certification validates your skills in supporting the maintenance and administration of the Exchange servers in an enterprise environment. Learn everything you need to know with this course.

jmcgOwnerCommented:
I have the impression there is something more going on that you are not telling us. Normally, whether executed from a perl script or from a shell script or from a command line, these commands run to completion before the next command is started or before returning control to the caller. What environment are you running on (or what code are you actually using?) that would permit sequential commands to overlap in time?
0
ivonne1094Author Commented:
Hi jmcg,
The perl script is running in Unix environment.
Please refer to the perl script below:
-----------------------------------------
system "tar -cf folder.tar folder";
system "gzip -f folder.tar";

system "rm folder/*";
rmdir folder;
------------------------------------------
When the folder size is very big, the gzip need some times to zip the folder. However, the result returns invalid zip file because the files in the folder have been removed before the zip process finished.
Now I add "sleep 60" into the perl script to give one minute for zip process.
-----------------------------------------
system "tar -cf folder.tar folder";
system "gzip -f folder.tar";

sleep 60;

system "rm folder/*";
rmdir folder;
------------------------------------------
But this is no good solution to wait for the zip process.
Maybe I shouldn't write the gzip and remove file in above method, please let me know if there
is better solution.
Thanks.

0
jmcgOwnerCommented:
This isn't yet making sense to me.

The tar process creates 'folder.tar' from the contents of 'folder'. It does not exit until it has finished. But there has to be sufficient disk space on the device to hold both copies, since the tar file takes up just as much space as the files that go into it.

The gzip command operates on the folder.tar file, which is not affected by any later attempts to remove the contents of folder. But it again requires more space to write the gzipped file.

Could it be that you are running out of space? If you are not checking for errors along the way, that might account for having an incomplete or invalid zip file at the end.

0
merphleCommented:
Instead of

---
system "tar -cf folder.tar folder";
system "gzip -f folder.tar";
sleep 60;
system "rm folder/*";
rmdir folder;
---

try using:

---
system "tar -cf - folder | gzip -9 > folder.tar.gz && rm -rf folder";
---

That should tar the folder and pipe the output to stdout, which is then read by gzip, and compressed into folder.tar.gz. If that whole operation succeeds, then rm -rf folder and all subfolder/directories.
0
merphleCommented:
Note that you will not have the intermediary folder.tar taking up space during the tar/gzip process. Also note that the -9 option to gzip will cause it to take a longer time to process, but the resulting gz file will almost certainly be smaller. You can safely remove -9 if you need the speed (as opposed to the disk space savings).
0
ivonne1094Author Commented:
Hi merphle,
I have tried the command:
---
system "tar -cf - folder | gzip -9 > folder.tar.gz && rm -rf folder";
---
but the forder seems not to be compressed properly and cannot open the zip file.

I found that the perl script terminated without waiting for the gzip process running because the "system" command will execute the process and the perl script continue running.

Is there any other function used to run or execute the any process and the following perl function can wait for the process finish?

Please advise. Thanks a lot.
0
jmcgOwnerCommented:
Perl's "system" command on UNIX should not (and _does not_ on any system I've encountered) return before the subprocess exits. Do you have a 'gzip' script that is executed in place of the normal gzip command?

Please give us some supporting evidence for your claims. What happens if you leave out the 'rm' step? Do you then get a working gzip file? How have you determined that gzip continues to run after the "system" call has returned? When you say "cannot open the zip file", what are you using to try to open it? Are there any error messages?

0
ivonne1094Author Commented:
Hi,
Below is the original complete perl script:
------------------------------
#!/usr/local/bin/perl
#script 'create_dir.pl'
#

   $WEBSERVER_NAME="pngedms2";
   $LOCALTIME=localtime;

   open(LOGFILE,">>auto_zip.log") or die "Can't find the auto_zip.log file, Please check the auto zip log file in current dir";
   print LOGFILE "************************************* \n";
   print LOGFILE "$LOCALTIME \n\n";

$ENV{ORACLE_BASE} = "/oracle/app/oracle";
$ENV{ORACLE_SID} = "ppap";
$ENV{ORACLE_HOME} = "/oracle/app/oracle/product/8.1.7";

$ctxdir = "/oracle/app/dl25/web/cgi/eppap";

#chop($WEBSERVER_NAME=`uname -n`);

   if($ENV{'REQUEST_METHOD'} eq "GET" ) {
        $_ = $ENV{'QUERY_STRING'};
        ($one,$two) = split /&/,$_,2;
        ($temp1,$ctg_id )= split /=/,$one,2;
        ($temp2,$user_id)= split /=/,$two,2;

   print "Content-type: text/html\n";
   print "Refresh:1;  URL=http://$WEBSERVER_NAME/eppap/plsql/amd_zip.page_waiting?v_ctg_id=$ctg_id\n\n";

#########################################################
#eval 'use Oraperl; 1;' || die $@ if $] >= 5;
use Oraperl;
#$ENV{'ORACLE_SID'};
$local_db = $ENV{'ORACLE_SID'};
$lda = &ora_login('','dl@eppap','dlbest') || die $ora_errstr;
$csr = &ora_open($lda,"select f.oid, f.upper_file_name
                 from cr_files f
                 where f.it_ctg_id = $ctg_id");

#($oid, $file_name) = &ora_fetch($csr);
 
#########################################################

            $_ = $ctg_id;
            chdir zip;
            mkdir $_;

    print LOGFILE "Folder id: $ctg_id \n";
    print LOGFILE "User: $user_id \n";
    print LOGFILE "Documents: \n";

    while (($oid, $file_name) = &ora_fetch($csr))
    {
        system "$ctxdir/ctx_getfile $file_name";
        print LOGFILE " $file_name \n";
    }

    system "tar -cf $_.tar $_ ";
    system "gzip -f $_.tar";

    # Remove files and folder
    system "rm $_/*";
    rmdir $_;

    print "<HTML>";
    print "<body bgcolor=linen>";
    print "<Strong>";
    print "Please wait while system download and zip the files...";

   }

 close(LOGFILE);

------------------------------
When the command " system "$ctxdir/ctx_getfile $file_name "; " still running (not all the files have been extracted to file system), the perl script already generate the refresh URL :
-----------------------------
print "Refresh:1;  URL=http://$WEBSERVER_NAME/eppap/plsql/amd_zip.page_waiting?v_ctg_id=$ctg_id\n\n";
-----------------------------
and the log file is closed.
SO I suspect the system through the process running outside the perl script and the perl script terminated without waiting for the program running under "system".
0
jmcgOwnerCommented:
Yes, that refresh line is printed by the script BEFORE it calls out to the $ctxdir/ctx_getfile" command.

Is there anything interesting in the LOGFILE? Is there anything interesting in the web server log files?

Since it is running as a CGI, are there limits imposed by your httpd daemon on how long a CGI script is allowed to run?
0
ivonne1094Author Commented:
Although the refresh line comes first, but we no problem in other script.
The problem only comes for the system"..."
The logfile will keep the file and folder that have been compressed, and I found the logfile and page refresh before all the files extracted into file system.
The cgi timeout has been set to 30 minutes.
Any command can replace the "system" and wait for the page running?
Please help. THanks a lot.
0
jmcgOwnerCommented:
You hold all the high cards in this guessing game. Your model for how "system" works and my model for how it works are apparently completely at odds. I cannot force you to accept my model, that is, that "system" only returns when the command has completed. I must therefore guess at the evidence you've seen that causes you to conclude that "system" returns before the command is completed.

So far, you have been extremely stingy in sharing data that would help us explain the disrepancy. Unless you are more forthcoming, I am unable to help you.

The LOGFILE is created and the refresh line is sent BEFORE any "system" call is made. It is completely consistent that you should see them and also see that the file extraction is going on. On UNIX systems, a file and its contents are observable as soon as the file is opened. You do not have to wait, as on some systems, for the file to be closed.

Please post the contents of the logfile, if it's not too large. Please verify how the amount of free space on the disk in question compares to the total size of the folder contents.

Is it a downloaded ZIP file that is unusable? Do you have access to check the gzipped archive that is on the web server?


0
jmcgOwnerCommented:
One more question. Is the refresh URL that you've given

print "Refresh:1;  URL=http://$WEBSERVER_NAME/eppap/plsql/amd_zip.page_waiting?v_ctg_id=$ctg_id\n\n";

pointing back to this same script? I'm wondering if you are repeatedly doing the extract/archive/zip/rm process with successive refreshes.



0
ivonne1094Author Commented:

Sorry if I didn't explain in details about my problem. I am not quite familiar with perl script and facing problem when use the script above and need help to solve some problems.

After few testing and I found out more and moreIf I moved the refresh URL to bottom, the program seems like will wait for the zip files process running. The refresh URL is just opening a page to inform user to download the zip but not extract and zip the folder again.

I thought the "system" caused some problem because from some testing of below code:
--------------------------------
while (($oid, $file_name) = &ora_fetch($csr))
    {
        system "$ctxdir/ctx_getfile $file_name";
        print LOGFILE " $file_name \n";
    }

    system "tar -cf $_.tar $_ ";
    system "gzip -f $_.tar";

    # Remove files and folder
    system "rm $_/*";
    rmdir $_;

----------------------------------
when the program still running the gzip process, some of the files already been removed - system "rm $_/*"; and caused the gzip process return invalid zip files (empty files), so I thought the "system" caused the problem.

The log file only contains information about the zip time, folder id, and files name :
*************************************
Thu Dec 11 00:49:21 2003

Folder id: 423
User: DELPHI
Documents:
 DELPHI ACN 2080 WAIVER PAGE.DOC
 ACN CHANGE NOTICE 12_15_00.DOC
 DELPHI QUALIFIED LETTER LEVEL1.AMDDOC.DOC
 09386968 DL400B90EI PSW ACN 2080.PDF
 ACN NUMBER 2080.TXT
 09386968 4_12_02 APPRVL.PDF
 2077 DELPHI ACN 2080.PDF
 09386968_A DELPHI.PDF
*************************************
Thu Dec 11 00:55:56 2003
 
Folder id: 422
User: DELPHI
Documents:
 09386968 4_12_02 APPRVL.PDF
 DELPHI ACN 2040 WAIVER PAGE.DOC
 DELPHI QUALIFIED LETTER LEVEL1.AMDDOC.DOC
 ACN 2040 SECTIONR LETTER DL400B90EI.DOC
 PSW_00_LEVEL1 0936968 ACN 2040.XLS
 2040X_LETTER.DOC
 ACN 2040 QUAL SUMMARY.PDF
*************************************

0
jmcgOwnerCommented:
In the original script, the screen directing the user to download the zip file would appear after a second or two, regardless of the state of the script on the server continuing to first get the files, then archive them, then zip them. Now, with the refresh URL moved to after the "system" calls, the various file operations should be completed before that screen appears. (How long does it typically take now for that screen to appear?)  Do you still need to prevent the user from resending the original request that started the extraction while the files are still being worked on?
0
ivonne1094Author Commented:
The time for page refresh to the URL depends on the size and total no of files to be compressed. If the file is small then the time needed will be little.
Yes, I would like to know how to prevent the user do the extraction again while waiting for the program running. Very appreciate your time and help.
Thanks a lot.
0
jmcgOwnerCommented:
You may need to create a lockfile using the value of $ctg_id. This can be removed at the time you remove the folder contents. By checking for the existence of this file before beginning the extraction/archive process, you can avoid overlapping requests that would interfere with each other.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Perl

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.