We help IT Professionals succeed at work.

Piping output of subprocess.Popen to stdout

derekl asked
Medium Priority
Last Modified: 2012-08-13
Here is what I would like to do.  I would like to execute a command using subprocess.Popen(), capture the output and write it to sys.stdout.  I know I can get stdout and stderr easily enough like this:

    p = Popen(cmd, shell=True, bufsize=bufsize, stdout=PIPE, stderr=PIPE, close_fds=True)
    (child_stdout, child_stderr) = p.communicate()
    for line in child_stdin:
        print line

I want to do something a bit different and likely a bit more complicated.  In the code above (if I understand things correctly), the for loop will not execute until the command executes.  For a long running command this is unnaceptable to me as I would like some feedback to the user.  In addition, I would like to preface every line sent from the subprocess with something along the lines of:

   username@host stdout ==>

To distinguish this output from normal program output.  Essentially what I need is a rewrite pipe that takes stdout from the subprocess prepends stuff to each line and outputs it to stdout, or even more generically to any FILE I give it as an agrument.

It seems like this should be fairly straightforward, but I'm not sure if I need to mess with threads or anything like that.

Watch Question

Use the os' module popen function.

import os
spam = popen("<command>")
for i in spam:
        print "<whateveryoulike>" + i


When will the for loop be executed?  I would like to perform this on the fly assuming I have a command that may take several minutes, I don't want to wait this long for any input to appear.
Oh, sorry, in that example the loop will probably be executed only after the process has ended.
You can still use the popen function. In order to be able to read from the pipe during the execution there are several important issues:
First of all, the sub-process you are running should use unbuffered IO. Many programs use buffered IO which means that if you attempt to redirect their IO, they will hold their output until they have a minimal amount of odata buffered (on my computer this is 4k, meaning that until the sub-process has printed 4k of data to the pipe, yu will not be able to get it). If the sub-process is itself a python script you can disable buffering by running "python -u" instead of just "python". Other programs may or may not have a similar option. Unfortunately if the sub-process does not have such an option and it is closed-source then there isn't much you can do.
A further problem is that even with unbuffered output, you can't know how much data the sub-process has written back to you. for instance, suppose the sub-process has written the word "Hello" (5 bytes), then if you try to read at most 5 bytes from the pipe [eggs = spam.read(5)] you will get the data immediately but if you attempt to read more [eggs = spam.read(6)] or [eggs = spam.read()] then it will block.
For your specific question, since you want to prefix each LINE of output, you can use the readline() method, which (provided that the output is not buffered) will return when a newline is encountered.
This code should function as excepted:

import os
spam = popen("<command>")
eggs = spam.readline()
while eggs != "":
        sys.stdout.write("<whateveryoulike>" + i)  # I replaced print because the line being read already contains a '\n'.
        eggs = spam.readline()

In addition, you can completely disable the blocking, with the following command, but that will work only on UNIX platforms:

import os
import fcntl
spam = popen("<command>")
fcntl.fcntl(spam.fileno(), fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)

Not the solution you were looking for? Getting a personalized solution is easy.

Ask the Experts


Thanks for the response.  I took the lazy way out, and rather than implement this myself I used the pexpect module which (except for a few minor bugs) is quite nice.


Well, if you can assume that this module will be available on any system your script is going to run on, it's a good solution.


The claim is that it's pure Python...
Access more of Experts Exchange with a free account
Thanks for using Experts Exchange.

Create a free account to continue.

Limited access with a free account allows you to:

  • View three pieces of content (articles, solutions, posts, and videos)
  • Ask the experts questions (counted toward content limit)
  • Customize your dashboard and profile

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.


Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.