Solved

Piping output of subprocess.Popen to stdout

Posted on 2006-11-03
6
717 Views
Last Modified: 2012-08-13
Here is what I would like to do.  I would like to execute a command using subprocess.Popen(), capture the output and write it to sys.stdout.  I know I can get stdout and stderr easily enough like this:

    p = Popen(cmd, shell=True, bufsize=bufsize, stdout=PIPE, stderr=PIPE, close_fds=True)
    (child_stdout, child_stderr) = p.communicate()
    for line in child_stdin:
        print line

I want to do something a bit different and likely a bit more complicated.  In the code above (if I understand things correctly), the for loop will not execute until the command executes.  For a long running command this is unnaceptable to me as I would like some feedback to the user.  In addition, I would like to preface every line sent from the subprocess with something along the lines of:

   username@host stdout ==>

To distinguish this output from normal program output.  Essentially what I need is a rewrite pipe that takes stdout from the subprocess prepends stuff to each line and outputs it to stdout, or even more generically to any FILE I give it as an agrument.

It seems like this should be fairly straightforward, but I'm not sure if I need to mess with threads or anything like that.

Derek
0
Comment
Question by:derekl
  • 3
  • 3
6 Comments
 
LVL 7

Expert Comment

by:Chatable
ID: 17868425
Use the os' module popen function.

import os
spam = popen("<command>")
for i in spam:
        print "<whateveryoulike>" + i
0
 

Author Comment

by:derekl
ID: 17868631
When will the for loop be executed?  I would like to perform this on the fly assuming I have a command that may take several minutes, I don't want to wait this long for any input to appear.
0
 
LVL 7

Accepted Solution

by:
Chatable earned 500 total points
ID: 17873721
Oh, sorry, in that example the loop will probably be executed only after the process has ended.
You can still use the popen function. In order to be able to read from the pipe during the execution there are several important issues:
First of all, the sub-process you are running should use unbuffered IO. Many programs use buffered IO which means that if you attempt to redirect their IO, they will hold their output until they have a minimal amount of odata buffered (on my computer this is 4k, meaning that until the sub-process has printed 4k of data to the pipe, yu will not be able to get it). If the sub-process is itself a python script you can disable buffering by running "python -u" instead of just "python". Other programs may or may not have a similar option. Unfortunately if the sub-process does not have such an option and it is closed-source then there isn't much you can do.
A further problem is that even with unbuffered output, you can't know how much data the sub-process has written back to you. for instance, suppose the sub-process has written the word "Hello" (5 bytes), then if you try to read at most 5 bytes from the pipe [eggs = spam.read(5)] you will get the data immediately but if you attempt to read more [eggs = spam.read(6)] or [eggs = spam.read()] then it will block.
For your specific question, since you want to prefix each LINE of output, you can use the readline() method, which (provided that the output is not buffered) will return when a newline is encountered.
This code should function as excepted:

import os
spam = popen("<command>")
eggs = spam.readline()
while eggs != "":
        sys.stdout.write("<whateveryoulike>" + i)  # I replaced print because the line being read already contains a '\n'.
        eggs = spam.readline()

In addition, you can completely disable the blocking, with the following command, but that will work only on UNIX platforms:

import os
import fcntl
spam = popen("<command>")
fcntl.fcntl(spam.fileno(), fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)
0
Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

 

Author Comment

by:derekl
ID: 17891783
Thanks for the response.  I took the lazy way out, and rather than implement this myself I used the pexpect module which (except for a few minor bugs) is quite nice.

http://pexpect.sourceforge.net/
0
 
LVL 7

Expert Comment

by:Chatable
ID: 17893073
Well, if you can assume that this module will be available on any system your script is going to run on, it's a good solution.
0
 

Author Comment

by:derekl
ID: 17893757
The claim is that it's pure Python...
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

The really strange introduction Once upon a time there were individuals who intentionally put the grass seeds to the soil with anticipation of solving their nutrition problems. Or they maybe only played with seeds and noticed what happened... Som…
Introduction On September 29, 2012, the Python 3.3.0 was released; nothing extremely unexpected,  yet another, better version of Python. But, if you work in Microsoft Windows, you should notice that the Python Launcher for Windows was introduced wi…
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
Learn the basics of while and for loops in Python.  while loops are used for testing while, or until, a condition is met: The structure of a while loop is as follows:     while <condition>:         do something         repeate: The break statement m…

895 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now