Solved

Piping output of subprocess.Popen to stdout

Posted on 2006-11-03
6
713 Views
Last Modified: 2012-08-13
Here is what I would like to do.  I would like to execute a command using subprocess.Popen(), capture the output and write it to sys.stdout.  I know I can get stdout and stderr easily enough like this:

    p = Popen(cmd, shell=True, bufsize=bufsize, stdout=PIPE, stderr=PIPE, close_fds=True)
    (child_stdout, child_stderr) = p.communicate()
    for line in child_stdin:
        print line

I want to do something a bit different and likely a bit more complicated.  In the code above (if I understand things correctly), the for loop will not execute until the command executes.  For a long running command this is unnaceptable to me as I would like some feedback to the user.  In addition, I would like to preface every line sent from the subprocess with something along the lines of:

   username@host stdout ==>

To distinguish this output from normal program output.  Essentially what I need is a rewrite pipe that takes stdout from the subprocess prepends stuff to each line and outputs it to stdout, or even more generically to any FILE I give it as an agrument.

It seems like this should be fairly straightforward, but I'm not sure if I need to mess with threads or anything like that.

Derek
0
Comment
Question by:derekl
  • 3
  • 3
6 Comments
 
LVL 7

Expert Comment

by:Chatable
ID: 17868425
Use the os' module popen function.

import os
spam = popen("<command>")
for i in spam:
        print "<whateveryoulike>" + i
0
 

Author Comment

by:derekl
ID: 17868631
When will the for loop be executed?  I would like to perform this on the fly assuming I have a command that may take several minutes, I don't want to wait this long for any input to appear.
0
 
LVL 7

Accepted Solution

by:
Chatable earned 500 total points
ID: 17873721
Oh, sorry, in that example the loop will probably be executed only after the process has ended.
You can still use the popen function. In order to be able to read from the pipe during the execution there are several important issues:
First of all, the sub-process you are running should use unbuffered IO. Many programs use buffered IO which means that if you attempt to redirect their IO, they will hold their output until they have a minimal amount of odata buffered (on my computer this is 4k, meaning that until the sub-process has printed 4k of data to the pipe, yu will not be able to get it). If the sub-process is itself a python script you can disable buffering by running "python -u" instead of just "python". Other programs may or may not have a similar option. Unfortunately if the sub-process does not have such an option and it is closed-source then there isn't much you can do.
A further problem is that even with unbuffered output, you can't know how much data the sub-process has written back to you. for instance, suppose the sub-process has written the word "Hello" (5 bytes), then if you try to read at most 5 bytes from the pipe [eggs = spam.read(5)] you will get the data immediately but if you attempt to read more [eggs = spam.read(6)] or [eggs = spam.read()] then it will block.
For your specific question, since you want to prefix each LINE of output, you can use the readline() method, which (provided that the output is not buffered) will return when a newline is encountered.
This code should function as excepted:

import os
spam = popen("<command>")
eggs = spam.readline()
while eggs != "":
        sys.stdout.write("<whateveryoulike>" + i)  # I replaced print because the line being read already contains a '\n'.
        eggs = spam.readline()

In addition, you can completely disable the blocking, with the following command, but that will work only on UNIX platforms:

import os
import fcntl
spam = popen("<command>")
fcntl.fcntl(spam.fileno(), fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)
0
Why You Should Analyze Threat Actor TTPs

After years of analyzing threat actor behavior, it’s become clear that at any given time there are specific tactics, techniques, and procedures (TTPs) that are particularly prevalent. By analyzing and understanding these TTPs, you can dramatically enhance your security program.

 

Author Comment

by:derekl
ID: 17891783
Thanks for the response.  I took the lazy way out, and rather than implement this myself I used the pexpect module which (except for a few minor bugs) is quite nice.

http://pexpect.sourceforge.net/
0
 
LVL 7

Expert Comment

by:Chatable
ID: 17893073
Well, if you can assume that this module will be available on any system your script is going to run on, it's a good solution.
0
 

Author Comment

by:derekl
ID: 17893757
The claim is that it's pure Python...
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Plenty of writing has gone on the web trying to compare Python with other competitive programming languages and vice versa. However, not much has been put into a wholistic perspective. This article should help you decide whether to adopt Python as a…
Variable is a place holder or reserved memory locations to store any value. Which means whenever we create a variable, indirectly we are reserving some space in the memory. The interpreter assigns or allocates some space in the memory based on the d…
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
Learn the basics of modules and packages in Python. Every Python file is a module, ending in the suffix: .py: Modules are a collection of functions and variables.: Packages are a collection of modules.: Module functions and variables are accessed us…

757 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now