Solved

exec background process, capture output, and return value in an anonymous array

Posted on 2006-11-10
22
277 Views
Last Modified: 2008-02-01
what i wish to do is really simple. I don't know how complex the solution is in perl. being the pismerl misaster that you are, i figured i'd ask you.

i need a function. the function is passed 2 arguments.  1st a string, which is a program/script name. 2nd a string containing the program/script's arguments. The function should be non-blocking, and promptly return($reference) while executing the (program + arguments) in the background also capturing the output and return value of the process.

upon completion of execution the return/output values should be set in hash reference as follows:

$reference->{PID}->{return} = "return value";
$reference->{PID}->{output} = "program output";


0
Comment
Question by:tejinashi
  • 11
  • 9
  • 2
22 Comments
 
LVL 39

Expert Comment

by:Adam314
ID: 17915251
When the function first returns, there will be no output and no return value from the program.  Those wouldn't be available until the program was finished executing.  Is this what you want?
0
 
LVL 1

Author Comment

by:tejinashi
ID: 17918796
Yes, I guess the function should return the PID right away, and update an external hash when the process completes
0
 
LVL 8

Expert Comment

by:jhurst
ID: 17927571
sub yourFunction {
  my $pid=fork();
   if ($pid) {return $pid}
   my $command="$_[0] $_[1]";
    exec($command);
    }
0
 
LVL 1

Author Comment

by:tejinashi
ID: 17933220
so since this is actually forking would the only way to get the output of the command being execed back to the program who called the function be via IPC or sockets of some sort?
0
 
LVL 39

Expert Comment

by:Adam314
ID: 17938756
I was thinking using threads, and placing the data in a shared variable.  If you are using any non thread safe modules though, this won't work.
0
 
LVL 1

Author Comment

by:tejinashi
ID: 17940693
I am worried about using threaded perl because I have heard of many issues with it. I am comfortable compiling perl with thread support. Can you tell me more about modules which aren't thread safe? Basically at this time I am using no external perl modules. Unless you call "strict" a module ;).

0
 
LVL 39

Expert Comment

by:Adam314
ID: 17941473
If you aren't using any modules, then using threads will be okay.  Some modules, in their documentation, will state whether they are thread safe.  

Using threads, getting the data from the process running the external command to the main process will be easy (simply place it in a shared variable).
Using fork, getting the data back to the main process will be more difficult.  But if you are concerned about threads based on other things, using fork might be the "safest" way.
0
 
LVL 1

Author Comment

by:tejinashi
ID: 17943206
Using fork what is the best way to get the return output of the executed programs back to the main program? (I will raise the point value of this topic)

Cheers
0
 
LVL 8

Expert Comment

by:jhurst
ID: 17947019
if you use the fork then you will need to have the forked copy send its output to a file, someything like $command="$_[0] $_[1] >/tmp/outputFromCommand.$$"

Now, the output will go to named file.  The one thing that I suspect that we are all missing here is why do you want to do this in the background if you want the script to wait for it to happen.  If you want to wait, then why not just do it.  If you can not wait then cleraly the outut will not be available.
0
 
LVL 1

Author Comment

by:tejinashi
ID: 17947285
Sorry I didn't explain that, the reason is because I could be executing as many as 30 to 40 different processes per second.
0
 
LVL 39

Expert Comment

by:Adam314
ID: 17947888
There are several methods to get the output back to the main process.  See here for your options:
http://perldoc.perl.org/perlipc.html

Whatever method, you'll have to be sure to put the output into the right variable (eg: don't put child 1's output in the variable for child 2).  Once you have the method to get the data, this shouldn't be too difficult, you could have the child send the PID before the actual output.

Again, this is why I was considering threads.  The output could be placed in a shared variable.  The main program wouldn't have to do anything to get the data, it would simply be in the shared variable.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 1

Author Comment

by:tejinashi
ID: 17953370
Using threads that same shared variable could be a hash or any other data structure that perl provides correct?

I will attempt to compile a threaded perl.
0
 
LVL 39

Expert Comment

by:Adam314
ID: 17956007
Yes, it should work.
0
 
LVL 1

Author Comment

by:tejinashi
ID: 17956308
I have just received word that this should work without a thread enabled perl. After some research I can see that based on your recommendation that a threaded implementation should be much easier to work with.

I believe that in this case, we should provide a solution for both threaded and non-threaded implementations of perl. I have tripled the point value. 125 for the original question using a threaded implementation, and 250 for the non-threaded example. Does this sound about right?

I have succesfully compiled a threaded version of perl using:
sh Configure -Dprefix=/usr/local/wansecurity -Dusethreads -de

Make test ran fine.
0
 
LVL 39

Expert Comment

by:Adam314
ID: 17957291
The solutions will be very different.  With the threaded implementation, you would put all of the output data into a shared variable - and it will be available in the main thread.

With the non-threaded implementation, you will have to use one of the IPC methods to get the data back into your main program.  As long as you have to do all of the development work to make it work with the non-threaded, there is no need to make it work with threads (as the non-threaded method will work on a threaded compiled perl).
0
 
LVL 1

Author Comment

by:tejinashi
ID: 17957445
I guess the best bet is to go with the non-threaded route.

I was just thinking though for the bennifit of the list, that it would be nice to post a solution for both methods. Hence the reason that I raised the point value.
0
 
LVL 1

Author Comment

by:tejinashi
ID: 17971723
Well, how about the threaded version then?
0
 
LVL 39

Accepted Solution

by:
Adam314 earned 440 total points
ID: 17980748
Here is the threaded version.


#!/usr/bin/perl
use strict;
use threads;
use threads::shared;
use Data::Dumper;

my %reference: shared;

for(my $i=0; $i<10; $i++){
      my $newtid=do_command("program","$i");
      print "TID: $newtid\n";
}
print Dumper(\%reference);
#If it is possible for the main program to finish before all it's threads are finished, they should be joined first

sub do_command {
      my $thread;
      unless($thread=threads->create(\&do_command_thread,@_)){
            print STDERR "ERROR creating thread: $!\n";
      }
      return $thread->tid;
}

sub do_command_thread {
      my $program=shift;
      my $args=join(" ",@_);
      
      #To actually call the external program
      my @lines=`$program $args`;
      my $retvalue=$?;
      
      #For debugging, don't actually call external program
      #my @lines=("Running $program $args\n", "line1\n", "line2\n", "line3\n");
      #my $retvalue=10;
      
      my $tid=threads->self->tid;
      $reference{$tid}=&share({});
      $reference{$tid}->{'return'}=$retvalue;
      $reference{$tid}->{'complete'}=1;
      $reference{$tid}->{'output'}="@lines";
}

0
 
LVL 1

Author Comment

by:tejinashi
ID: 17999141
Thank you, my son was just born, so I need just a little bit of time to review this
0
 
LVL 39

Expert Comment

by:Adam314
ID: 18020230
congratulations!

no problems, take as much time as needed.
0
 
LVL 1

Author Comment

by:tejinashi
ID: 18055125
I got a chance to test this out, and it works with a threaded perl to get return output back to the main process. Is there any way we can get an IPC version?
0
 
LVL 39

Expert Comment

by:Adam314
ID: 18056847
I don't have much experience with IPC.

If you want to try it, the first thing you should do is read up on it:
http://perldoc.perl.org/perlipc.html

Then decide which method would work best for what you need (maybe post another question to get help with that)
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (http://www.mongodb.org/downloads),  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
Checking the Alert Log in AWS RDS Oracle can be a pain through their user interface.  I made a script to download the Alert Log, look for errors, and email me the trace files.  In this article I'll describe what I did and share my script.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Excel styles will make formatting consistent and let you apply and change formatting faster. In this tutorial, you'll learn how to use Excel's built-in styles, how to modify styles, and how to create your own. You'll also learn how to use your custo…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now