Piping output of subprocess.Popen to stdout

Posted on 2006-11-03
Last Modified: 2012-08-13
Here is what I would like to do.  I would like to execute a command using subprocess.Popen(), capture the output and write it to sys.stdout.  I know I can get stdout and stderr easily enough like this:

    p = Popen(cmd, shell=True, bufsize=bufsize, stdout=PIPE, stderr=PIPE, close_fds=True)
    (child_stdout, child_stderr) = p.communicate()
    for line in child_stdin:
        print line

I want to do something a bit different and likely a bit more complicated.  In the code above (if I understand things correctly), the for loop will not execute until the command executes.  For a long running command this is unnaceptable to me as I would like some feedback to the user.  In addition, I would like to preface every line sent from the subprocess with something along the lines of:

   username@host stdout ==>

To distinguish this output from normal program output.  Essentially what I need is a rewrite pipe that takes stdout from the subprocess prepends stuff to each line and outputs it to stdout, or even more generically to any FILE I give it as an agrument.

It seems like this should be fairly straightforward, but I'm not sure if I need to mess with threads or anything like that.

Question by:derekl
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3

Expert Comment

ID: 17868425
Use the os' module popen function.

import os
spam = popen("<command>")
for i in spam:
        print "<whateveryoulike>" + i

Author Comment

ID: 17868631
When will the for loop be executed?  I would like to perform this on the fly assuming I have a command that may take several minutes, I don't want to wait this long for any input to appear.

Accepted Solution

Chatable earned 500 total points
ID: 17873721
Oh, sorry, in that example the loop will probably be executed only after the process has ended.
You can still use the popen function. In order to be able to read from the pipe during the execution there are several important issues:
First of all, the sub-process you are running should use unbuffered IO. Many programs use buffered IO which means that if you attempt to redirect their IO, they will hold their output until they have a minimal amount of odata buffered (on my computer this is 4k, meaning that until the sub-process has printed 4k of data to the pipe, yu will not be able to get it). If the sub-process is itself a python script you can disable buffering by running "python -u" instead of just "python". Other programs may or may not have a similar option. Unfortunately if the sub-process does not have such an option and it is closed-source then there isn't much you can do.
A further problem is that even with unbuffered output, you can't know how much data the sub-process has written back to you. for instance, suppose the sub-process has written the word "Hello" (5 bytes), then if you try to read at most 5 bytes from the pipe [eggs =] you will get the data immediately but if you attempt to read more [eggs =] or [eggs =] then it will block.
For your specific question, since you want to prefix each LINE of output, you can use the readline() method, which (provided that the output is not buffered) will return when a newline is encountered.
This code should function as excepted:

import os
spam = popen("<command>")
eggs = spam.readline()
while eggs != "":
        sys.stdout.write("<whateveryoulike>" + i)  # I replaced print because the line being read already contains a '\n'.
        eggs = spam.readline()

In addition, you can completely disable the blocking, with the following command, but that will work only on UNIX platforms:

import os
import fcntl
spam = popen("<command>")
fcntl.fcntl(spam.fileno(), fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)
Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.


Author Comment

ID: 17891783
Thanks for the response.  I took the lazy way out, and rather than implement this myself I used the pexpect module which (except for a few minor bugs) is quite nice.

Expert Comment

ID: 17893073
Well, if you can assume that this module will be available on any system your script is going to run on, it's a good solution.

Author Comment

ID: 17893757
The claim is that it's pure Python...

Featured Post

Announcing the Most Valuable Experts of 2016

MVEs are more concerned with the satisfaction of those they help than with the considerable points they can earn. They are the types of people you feel privileged to call colleagues. Join us in honoring this amazing group of Experts.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Python tuples 2 133
Parse csv file and generate graphs in HTML in bash 8 277
linux crontab output 3 80
Getting Variable not defined error in Python 1 45
The purpose of this article is to demonstrate how we can use conditional statements using Python.
When we want to run, execute or repeat a statement multiple times, a loop is necessary. This article covers the two types of loops in Python: the while loop and the for loop.
Learn the basics of strings in Python: declaration, operations, indices, and slicing. Strings are declared with quotations; for example: s = "string": Strings are immutable.: Strings may be concatenated or multiplied using the addition and multiplic…
Learn the basics of while and for loops in Python.  while loops are used for testing while, or until, a condition is met: The structure of a while loop is as follows:     while <condition>:         do something         repeate: The break statement m…

726 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question