troubleshooting Question

Logon to Web Site and Download File Programmatically using Python, urllib2 module

Avatar of DoveTails
DoveTailsFlag for United States of America asked on
PythonScripting LanguagesProgramming
6 Comments3 Solutions3623 ViewsLast Modified:
Wish to logon to web site to programmatically download data.
Looking for assistance with Python and the urllib2 module.

At this page:
  https://www.nifc.blm.gov/cgi/WfmiHome.cgi
there is a Logon button which uses a GET (not a Post) referring to:
    https://www.nifc.blm.gov/cgi/WfmiHome.cgi/Page/Logon
which Redirects to: https://www.nifc.blm.gov/cgi/WfmiHome.cgi/Page/DoiMonitor

I'm hoping to understand how to navigate through web pages to eventually logon with my account.  
I've tried many variations of the below code without success.
Need help understand urllib2 to navigate to page with "I Agree" button and then to the actual page to enter username and password.
mport urllib
import urllib2
import cookielib

#cookie storage
cj = cookielib.CookieJar()
opener = urllib2.build_opener(
    urllib2.HTTPCookieProcessor(cj),
    urllib2.HTTPRedirectHandler
    )

#### First page
url = 'https://www.nifc.blm.gov/cgi/WfmiHome.cgi'

request = urllib2.Request(url)

response = urllib2.urlopen(request)

html = response.read()

# Print to screen
print html 

Appreciate help ...  Thanks !
ASKER CERTIFIED SOLUTION
Walter Ritzel
Senior Software Engineer

Our community of experts have been thoroughly vetted for their expertise and industry experience.

Join our community to see this answer!
Unlock 3 Answers and 6 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 3 Answers and 6 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros