asked on

Getting the correct arguments for Python Subprocess

I'm trying to figure out how to use the subprocess module in Python to run a command to execute edx-dl. Edx-dl is a git project to download video and course content from Edx.org.

In the windows shell (usually powershell), I would do the following without error:

python E:\edx-dl\edx-dl.py -u my@email.com -p "mypw" --ignore-errors

Open in new window

In python, I have tried a number of different ways of inputting the arguments, all to no avail. I got the furthest with this:

import os
import subprocess

#subprocess.call(["ls", "-alt"])
#subprocess.call(["grep", "six", "requirements.txt"])

os.chdir('E:\edx-dl')
subprocess.call(["python", "E:\edx-dl\edx-dl.py", "-u my@email.com -p mypw"])

Open in new window

In the shell, this would give:

edx_dl version 0.1.7

Building initial headers for future requests.
Getting initial CSRF token.
Found CSRF token.
Logging into Open edX site: https://courses.edx.org/login_ajax
Extracting course information from dashboard.
You can access 12 courses
...{LIST OF COURSES}

But from Python all I get is that first line, and then it hangs.

What is the proper argument format for subprocess? Do the '-u' and the 'my@email.com' belong in the same or different strings? 'Python' is the actual command, and the edx-dl.py is the python file.

What do I do?

ASKER CERTIFIED SOLUTION

ugeb

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

Apurva Bhandari

issue resolved??

ugeb

ASKER

Well, I answered this particular question, but if you want to answer the driver behind this question, that would be great, but certainly more complicated.

Essentially I am trying to run the edx_dl.py code and output everything into a file instead of stdout. My attempts and redirecting stdout don't seem to affect any output from edx_dl.py.

If you think can answer that question, I'd be happy to expound on it. Maybe the title of the question needs changing or a new question needs asking.

SOLUTION

pepr

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

ugeb

ASKER

Thank you for your response. I had read that shell=True was insecure, but it may not make a difference for me.

I fully agree that it doesn't make sense to call the python interpreter from within the script, but I couldn't figure out how to make stdout go to a file, since I don't control the code that outputs within edx_dl. My intention was to use the subprocess for an actual redirect. Inefficient yes, but I was in a hurry. This is not production code, so I'm not concerned with efficiency, but I would still like to do it the best way.

I tried changing stdout as follows:

sys.stdout = open('myfile.txt', 'w')
print("one two three")
edx_dl.main()
print("That's all folks.")

Open in new window

But edx_dl still prints everything to the console. The only thing in myfile.txt is "one two three".

Do you know how I can direct all stdout to a file when I can't control the output statements?

(I need this to run in both 2.7 and 3.5+)

pepr

I have looked briefly at the edx_dl code. You are right that it may not be easy to use it from within a Python script. In my opinion, the tool should have been written better (with better interface) to be usefull also as a module, not only as a utility.

If you are willing to do that, it may be reasonable to refactor the edx_dl code.

ugeb

ASKER

Thank you for taking a look at the edx code.

Yeah, that sounds like too much work for my bandwidth. In the meantime I've set up a shell script to run edx and redirect into a file which I then read with Python. Shlex is really good for identifying the proper tokenization, so I'm glad I found that.

ugeb

ASKER

The solution I found was the only thing that got me close to being able to run the command without error.