We help IT Professionals succeed at work.

Executing a subprocess and reading its output on windows

cxt
cxt asked
on
7,791 Views
Last Modified: 2008-01-09
This should be fairly easy but I can't get things to work correctly. Basically I am just trying to launch a command line argument from python and read the standard output. I first tried:

res = os.popen(cmdline, 'r')
for line in res.readlines():
   ...

The problem is that when the res.readlines() is executed, the process may still be running, and thus I am reading a partial output. I have no way to know whether I should wait or if I should proceed.

I then stumbled upon subprocess.Popen which has a wait() member. However, for some reason this does not capture the stdout very well. For example, a subprocess.Popen('cmd /C dir') will print the result on screen rather than capturing it in its result.

Finally, there is a win32pipe module with a popen and a WaitNamedPipe, but I couldn't make this to work and the documentation seems to be sparse.

Surely there must be a simple solution to this simple and common task..

Any help greatly appreciated!

cxt
Comment
Watch Question

readlines() will block while the subprocess is still running and there's no output left to consume, and will resume when output appears or the process exits.  When the process exits, your loop will exit.

I'm not sure whether that answers your question or not... I think there may be some requirement hidden in the "..." piece of your code snippet that I don't understand.  8-)
cxt

Author

Commented:
Well, the program I call in os.popen takes around 1 minute to finish. During that time, it prints various stuff on stdout.

Quite often (~30% of the times), the readlines will actually trigger after the first output.

Suppose the "..." is this:

for line in res.readlines():
  print line

and that the output of the cmdline is this:

A. <pause for ~20s>
B.

then what I observe is that sometimes only "A." is captured by readlines() and printed out by "print line". I should mention that this is not deterministic. Re-running the very same command line may succeed. Is this really in the spec of readlines() that it should wait for the end of the command? Then what am I doing wrong?

Oh, another thing I tried but turned out a disapointment: os.spawnv(os.P_WAIT,...) does not capture the output. I wonder btw if there is a way to 'recover' the output of this command (or os.system), to somehow emulate a working os.popen...

Do you mean that sometimes "B." is never printed, even when you leave it running for the required minute?  I can't reproduce that with my simple script:

ee-popen.py:

import os
res = os.popen("cmd /C pausing-output.bat")
for line in res.readlines():
    print line
print "Done"

pausing-output.bat:

echo "Line 1"
sleep 5
echo "Line 2"
sleep 10
echo "Done

That works, in that all the output of the batch script is captured by the Python script and printed.  Bear in mind that the subprocess might be buffering its output, so you might not receive the lines in real time.  That happens with my example - it takes 15 seconds before anything is printed, then the whole lot appears at once.
cxt

Author

Commented:
Okay, so it seems that my problem is a bit more complicated than that. I tried your example and it works fine. It seems that the problem originates from my use of psexec ( http://www.microsoft.com/technet/sysinternals/utilities/psexec.mspx ). If you have now

res = os.popen('psexec c:\my_path\pausing-output.bat')

then, sometimes, you will get only part of the output or none at all.

CERTIFIED EXPERT

Commented:
Just a guess In your  
  res = os.popen(cmdline, 'r')
try to add the bufsize argument. In Python 2.5 documentation it is said to have the same meaning as the same argument of the built-in open(), and the doc for open() says:
  The optional bufsize argument specifies the file's desired buffer size:
      * 0 means unbuffered,
      * 1 means line buffered,
      * any other positive value means use a buffer of (approximately) that size.
      * A negative bufsize means to use the system default, which is usually line buffered
         for tty devices and fully buffered for other files.
      * If omitted, the system default is used.

So, you should try:

  res = os.popen(cmdline, 'r', bufsize=0)

The next problem may be hidden in res.readlines() as it returns the list of strings constructed from the file-like object. Try

  for line in res:
      ...

Which uses iterator through the lines of the res. In other words, the output line should be processed as soon as the next line is ready.

This is just my blind guess. I am not sure if it solves your problem. Also, os.popen() did not work in Windows for older Python versions. However, it probably works now.
This one is on us!
(Get your first solution completely free - no credit card required)
UNLOCK SOLUTION
cxt

Author

Commented:
I am realizing I am using an old version of psexec... I downloaded the latest one, and now commands are executed in another cmd window -- and it doesn't produce any output as you say.

You're right, a good old dump into a file is probably all I need. At least it hacks me into getting things working.

Thanks!

cxt
cxt

Author

Commented:
To get back to the comment of pepr:

On my version of python (2.4.3 on Windows), os.popen doesn't really accept a third argument: if I add a third it says it must be -1. Setting it to -1 did not solve my problem.

I also tried not to use an iterator, the problem is that it finds an EOL before the end of the execution of psexec -- probably the same pb readlines() has.

I still don't know how psexec managed to throw things to popen before it ends or how popen is fooled into thinking psexec is finished before it actually is. But dumping into a file works fine for me.

Gain unlimited access to on-demand training courses with an Experts Exchange subscription.

Get Access
Why Experts Exchange?

Experts Exchange always has the answer, or at the least points me in the correct direction! It is like having another employee that is extremely experienced.

Jim Murphy
Programmer at Smart IT Solutions

When asked, what has been your best career decision?

Deciding to stick with EE.

Mohamed Asif
Technical Department Head

Being involved with EE helped me to grow personally and professionally.

Carl Webster
CTP, Sr Infrastructure Consultant
Empower Your Career
Did You Know?

We've partnered with two important charities to provide clean water and computer science education to those who need it most. READ MORE

Ask ANY Question

Connect with Certified Experts to gain insight and support on specific technology challenges including:

  • Troubleshooting
  • Research
  • Professional Opinions
Unlock the solution to this question.
Join our community and discover your potential

Experts Exchange is the only place where you can interact directly with leading experts in the technology field. Become a member today and access the collective knowledge of thousands of technology experts.

*This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

OR

Please enter a first name

Please enter a last name

8+ characters (letters, numbers, and a symbol)

By clicking, you agree to the Terms of Use and Privacy Policy.