Solved

Piping output of subprocess.Popen to stdout

Posted on 2006-11-03
6
806 Views
Last Modified: 2012-08-13
Here is what I would like to do.  I would like to execute a command using subprocess.Popen(), capture the output and write it to sys.stdout.  I know I can get stdout and stderr easily enough like this:

    p = Popen(cmd, shell=True, bufsize=bufsize, stdout=PIPE, stderr=PIPE, close_fds=True)
    (child_stdout, child_stderr) = p.communicate()
    for line in child_stdin:
        print line

I want to do something a bit different and likely a bit more complicated.  In the code above (if I understand things correctly), the for loop will not execute until the command executes.  For a long running command this is unnaceptable to me as I would like some feedback to the user.  In addition, I would like to preface every line sent from the subprocess with something along the lines of:

   username@host stdout ==>

To distinguish this output from normal program output.  Essentially what I need is a rewrite pipe that takes stdout from the subprocess prepends stuff to each line and outputs it to stdout, or even more generically to any FILE I give it as an agrument.

It seems like this should be fairly straightforward, but I'm not sure if I need to mess with threads or anything like that.

Derek
0
Comment
Question by:derekl
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
6 Comments
 
LVL 7

Expert Comment

by:Chatable
ID: 17868425
Use the os' module popen function.

import os
spam = popen("<command>")
for i in spam:
        print "<whateveryoulike>" + i
0
 

Author Comment

by:derekl
ID: 17868631
When will the for loop be executed?  I would like to perform this on the fly assuming I have a command that may take several minutes, I don't want to wait this long for any input to appear.
0
 
LVL 7

Accepted Solution

by:
Chatable earned 500 total points
ID: 17873721
Oh, sorry, in that example the loop will probably be executed only after the process has ended.
You can still use the popen function. In order to be able to read from the pipe during the execution there are several important issues:
First of all, the sub-process you are running should use unbuffered IO. Many programs use buffered IO which means that if you attempt to redirect their IO, they will hold their output until they have a minimal amount of odata buffered (on my computer this is 4k, meaning that until the sub-process has printed 4k of data to the pipe, yu will not be able to get it). If the sub-process is itself a python script you can disable buffering by running "python -u" instead of just "python". Other programs may or may not have a similar option. Unfortunately if the sub-process does not have such an option and it is closed-source then there isn't much you can do.
A further problem is that even with unbuffered output, you can't know how much data the sub-process has written back to you. for instance, suppose the sub-process has written the word "Hello" (5 bytes), then if you try to read at most 5 bytes from the pipe [eggs = spam.read(5)] you will get the data immediately but if you attempt to read more [eggs = spam.read(6)] or [eggs = spam.read()] then it will block.
For your specific question, since you want to prefix each LINE of output, you can use the readline() method, which (provided that the output is not buffered) will return when a newline is encountered.
This code should function as excepted:

import os
spam = popen("<command>")
eggs = spam.readline()
while eggs != "":
        sys.stdout.write("<whateveryoulike>" + i)  # I replaced print because the line being read already contains a '\n'.
        eggs = spam.readline()

In addition, you can completely disable the blocking, with the following command, but that will work only on UNIX platforms:

import os
import fcntl
spam = popen("<command>")
fcntl.fcntl(spam.fileno(), fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)
0
On Demand Webinar: Networking for the Cloud Era

Did you know SD-WANs can improve network connectivity? Check out this webinar to learn how an SD-WAN simplified, one-click tool can help you migrate and manage data in the cloud.

 

Author Comment

by:derekl
ID: 17891783
Thanks for the response.  I took the lazy way out, and rather than implement this myself I used the pexpect module which (except for a few minor bugs) is quite nice.

http://pexpect.sourceforge.net/
0
 
LVL 7

Expert Comment

by:Chatable
ID: 17893073
Well, if you can assume that this module will be available on any system your script is going to run on, it's a good solution.
0
 

Author Comment

by:derekl
ID: 17893757
The claim is that it's pure Python...
0

Featured Post

[Webinar] Learn How Hackers Steal Your Credentials

Do You Know How Hackers Steal Your Credentials? Join us and Skyport Systems to learn how hackers steal your credentials and why Active Directory must be secure to stop them. Thursday, July 13, 2017 10:00 A.M. PDT

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Strings in Python are the set of characters that, once defined, cannot be changed by any other method like replace. Even if we use the replace method it still does not modify the original string that we use, but just copies the string and then modif…
Dictionaries contain key:value pairs. Which means a collection of tuples with an attribute name and an assigned value to it. The semicolon present in between each key and values and attribute with values are delimited with a comma.  In python we can…
Learn the basics of lists in Python. Lists, as their name suggests, are a means for ordering and storing values. : Lists are declared using brackets; for example: t = [1, 2, 3]: Lists may contain a mix of data types; for example: t = ['string', 1, T…
Learn the basics of if, else, and elif statements in Python 2.7. Use "if" statements to test a specified condition.: The structure of an if statement is as follows: (CODE) Use "else" statements to allow the execution of an alternative, if the …

696 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question