asked on

PYTHON CLIENT SIDE CGI SCRIPT

I need Major Help, I have this webpage called www.peirce.com That I need to log into and extract information from. First of all it it a cgi script that I need to end up in to get the information.

So My question consist of tree things.
1.) How do I log into the CGI
2.) How do I realize that I am logged in correctly
3.) How do I know if I need cookie support
4.) And how issue a command to do the search I need done

The last part i beileve would be easy is parsing the text from the webpage to get the data needed.

Here is the Source Data from right After I login:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3c.org/TR/1999/REC-html401-19991224/loose.dtd">

<HTML><HEAD><TITLE>DEALERS</TITLE>
<META http-equiv=Content-Type content="text/html; charset=utf-8">
<STYLE type=text/css>BODY {
      MARGIN-TOP: 0px; MARGIN-LEFT: 0px
}
INPUT {
      FONT-SIZE: 11px; BACKGROUND: #efefef; FONT-FAMILY: verdana
}
SELECT {
      FONT-SIZE: 11px; BACKGROUND: #efefef; FONT-FAMILY: verdana
}
TEXTAREA {
      FONT-SIZE: 11px; BACKGROUND: #efefef; FONT-FAMILY: verdana
}
A:link {
      COLOR: #003366; TEXT-DECORATION: none
}
A:visited {
      COLOR: #003366; TEXT-DECORATION: none
}
A:hover {
      COLOR: #990000; TEXT-DECORATION: none
}
A {
      TEXT-DECORATION: none
}
TD {
      FONT-SIZE: 11px; FONT-FAMILY: verdana
}
.links {
      FONT-SIZE: 11px; COLOR: #000000; FONT-FAMILY: verdana
}
.button {

}
.toolTip {
      BORDER-RIGHT: #c0c0c0 1px solid; PADDING-RIGHT: 10px; BORDER-TOP: #c0c0c0 1px solid; DISPLAY: block; PADDING-LEFT: 10px; BACKGROUND: #eff3b0; PADDING-BOTTOM: 10px; BORDER-LEFT: #c0c0c0 1px solid; WIDTH: 250px; PADDING-TOP: 10px; BORDER-BOTTOM: #c0c0c0 1px solid; POSITION: absolute
}
.infoTipShow {
      PADDING-RIGHT: 10px; DISPLAY: none; PADDING-LEFT: 10px; PADDING-BOTTOM: 10px; WIDTH: 100%; PADDING-TOP: 10px; POSITION: absolute
}
.toolTipHide {
      BORDER-RIGHT: #c0c0c0 1px solid; PADDING-RIGHT: 10px; BORDER-TOP: #c0c0c0 1px solid; DISPLAY: none; PADDING-LEFT: 10px; BACKGROUND: #eff3b0; PADDING-BOTTOM: 10px; BORDER-LEFT: #c0c0c0 1px solid; WIDTH: 250px; PADDING-TOP: 10px; BORDER-BOTTOM: #c0c0c0 1px solid; POSITION: absolute
}
</STYLE>

<SCRIPT>
thisApp = 'https://www.peirce.com/com/peircephelps/functions.cfm';
</SCRIPT>

<SCRIPT src="DEALERS_files/engine.js" type=text/javascript></SCRIPT>

<SCRIPT src="DEALERS_files/util.js" type=text/javascript></SCRIPT>

<SCRIPT src="DEALERS_files/settings.js" type=text/javascript></SCRIPT>

<SCRIPT src="DEALERS_files/dealer_js.js" type=text/javascript></SCRIPT>

<META content="MSHTML 6.00.2900.3157" name=GENERATOR></HEAD>
<BODY bgColor=#d5d5d5
onload="init();MM_preloadImages('images/company_o.jpg','images/products_o.jpg','images/dealerlocator_o.jpg','images/hvac_o.jpg','images/dealers_o.jpg','images/training_courses_o.jpg','images/contact_o.jpg','images/employment_o.jpg','images/dealer_profile_o.jpg')">
<TABLE cellSpacing=0 cellPadding=0 width=790 align=center border=0>
<TBODY>
<TR>
<TD width=1 bgColor=#999999><IMG height=1 src="DEALERS_files/spacer.gif"
width=1></TD>
<TD width=790 bgColor=#ffffff>
<TABLE cellSpacing=0 cellPadding=0 width=790 border=0>
<TBODY>
<TR>
<TD>
<TABLE cellSpacing=0 cellPadding=0 width=790 border=0>
<TBODY>
<TR>
<TD width=197><A href="https://www.peirce.com/index.cfm"><IMG
height=83 alt=logo src="DEALERS_files/logo.jpg" width=189
border=0></A></TD>
<TD width=197><IMG height=83 alt=topbanner
src="DEALERS_files/top2.jpg" width=416></TD>
<TD width=197><IMG height=83 alt=news
src="DEALERS_files/news.jpg" width=119></TD>
<TD width=199><A href="https://www.peirce.com/index.cfm"><IMG
height=83 alt=home src="DEALERS_files/home.jpg" width=66
border=0></A></TD></TR></TBODY></TABLE></TD></TR>
<TR>
<TD vAlign=top>
<TABLE cellSpacing=0 cellPadding=0 width=790 border=0>
<TBODY>
<TR>
<TD vAlign=top width=181>
<SCRIPT language=JavaScript type=text/JavaScript>

</SCRIPT>

<TABLE cellSpacing=0 cellPadding=0 width=181 border=0>
<TBODY>
<TR>
<TD vAlign=top width=181>
<TABLE cellSpacing=0 cellPadding=0 width=181 border=0>
<TBODY>
<TR>
<TD height=31><A
href="https://www.peirce.com/about.cfm"><IMG
id=Image1
onmouseover="MM_swapImage('Image1','','images/company_o.jpg',1)"
onmouseout=MM_swapImgRestore() height=31
alt=company src="DEALERS_files/company.jpg"
width=181 border=0 name=Image1></A></TD></TR>
<TR>
<TD height=25><A
href="https://www.peirce.com/products.cfm"><IMG
id=Image2
onmouseover="MM_swapImage('Image2','','images/products_o.jpg',1)"
onmouseout=MM_swapImgRestore() height=25
alt=products src="DEALERS_files/products.jpg"
width=181 border=0 name=Image2></A></TD></TR>
<TR>
<TD height=23><A
href="http://www.peirce.com/dealer_locator.cfm"><IMG
id=Image4
onmouseover="MM_swapImage('Image4','','images/dealerlocator_o.jpg',1)"
onmouseout=MM_swapImgRestore() height=23
alt=dealerlocator
src="DEALERS_files/dealerlocator.jpg" width=181
border=0 name=Image4></A></TD></TR>
<TR>
<TD height=26><A
href="https://www.peirce.com/howhvacworks.cfm"><IMG
id=Image8
onmouseover="MM_swapImage('Image81','','images/hvac_o.jpg',1)"
onmouseout=MM_swapImgRestore() height=26
alt=surveys src="DEALERS_files/hvac.jpg" width=181
border=0 name=Image81></A></TD></TR>
<TR>
<TD height=23><A
href="https://www.peirce.com/dealer_register.cfm"><IMG
id=Image3
onmouseover="MM_swapImage('Image3','','images/dealers_o.jpg',1)"
onmouseout=MM_swapImgRestore() height=23
alt=dealers src="DEALERS_files/dealers.jpg"
width=181 border=0 name=Image3></A></TD></TR>
<TR>
<TD height=24><A
href="https://www.peirce.com/training.cfm"><IMG
id=Image5
onmouseover="MM_swapImage('Image5','','images/training_courses_o.jpg',1)"
onmouseout=MM_swapImgRestore() height=24
alt=courses
src="DEALERS_files/training_courses.jpg" width=181
border=0 name=Image5></A></TD></TR>
<TR>
<TD height=23><A
href="https://www.peirce.com/contact.cfm"><IMG
id=Image7
onmouseover="MM_swapImage('Image7','','images/contact_o.jpg',1)"
onmouseout=MM_swapImgRestore() height=23
alt=contact src="DEALERS_files/contact.jpg"
width=181 border=0 name=Image7></A></TD></TR>
<TR>
<TD height=24><A
href="https://www.peirce.com/news.cfm"><IMG
id=Image66
onmouseover="MM_swapImage('Image66','','images/employment_o.jpg',1)"
onmouseout=MM_swapImgRestore() height=24 alt=news
src="DEALERS_files/employment.jpg" width=181
border=0
name=Image66></A></TD></TR></TBODY></TABLE></TD></TR>
<TR>
<TD><BR><BR></TD></TR></TBODY></TABLE></TD>
<TD vAlign=top width=609>
<TABLE cellSpacing=0 cellPadding=0 width=609 border=0>
<TBODY>
<TR>
<TD><IMG height=80
src="DEALERS_files/general_top_image.jpg"
width=608></TD></TR>
<TR>
<TD width=609>
<TABLE cellSpacing=0 cellPadding=0 width=606 border=0>
<TBODY>
<TR align=middle bgColor=#d3d7d0>
<TD noWrap align=left width=606><IMG height=20
src="DEALERS_files/grey_navigation_bar.jpg"
width=606 useMap=#Map
border=0></TD></TR></TBODY></TABLE><MAP id=Map
name=Map><AREA shape=RECT coords=3,1,88,18
href="https://www.peirce.com/products_hvac.cfm"><AREA
shape=RECT coords=94,2,226,22
href="https://www.peirce.com/products_buildingautomation.cfm"><AREA
shape=RECT coords=236,1,363,20
href="https://www.peirce.com/products_hearth.cfm"><AREA
shape=RECT coords=374,0,487,30
href="https://www.peirce.com/products_kitchen_bath.cfm"><AREA
shape=RECT coords=496,2,601,29
href="https://www.peirce.com/products_GRILL.cfm"></MAP></TD></TR>
<TR>
<TD bgColor=#7c7d77><IMG height=1
src="DEALERS_files/spacer.gif" width=22></TD></TR>
<TR>
<TD vAlign=top>
<TABLE cellSpacing=0 cellPadding=10 width="100%"
border=0>
<TBODY>
<TR>
<TD bgColor=#efefef>
<TABLE cellSpacing=0 cellPadding=0 width=590
border=0>
<TBODY>
<TR>
<TD width=20>
<TABLE borderColor=#efefef height=12
cellSpacing=0 cellPadding=0 width=12 border=1>
<TBODY>
<TR>
<TD bgColor=#143874><IMG height=10
src="DEALERS_files/spacer.gif" width=10
border=1></TD></TR></TBODY></TABLE></TD>
<TD width=140><STRONG><FONT
face="Verdana, Arial, Helvetica, sans-serif"
color=#143874
size=4> DEALERS</FONT></STRONG></TD>
<TD align=right
width=429><STRONG></STRONG></TD></TR></TBODY></TABLE></TD></TR>
<TR>
<TD vAlign=top><BR><FONT color=#143874>
<TABLE width="100%">
<TBODY>
<TR>
<TD width="50%"><STRONG>Welcome, Dawn Marie
McDonnell</STRONG><BR><EM>offenbachers</EM> [<A
href="https://www.peirce.com/dealer_main.cfm?logout=1">Logout</A>]</TD>
<TD align=right
width="50%"> </TD></TR></TBODY></TABLE>
<TABLE cellSpacing=0 cellPadding=7 width="100%"
border=0>
<TBODY>
<TR>
<TD colSpan=2>You have access to the following
areas. Click on a link below to
continue</TD></TR>
<TR vAlign=top>
<TD align=right>
<TABLE borderColor=#ffffff height=7
cellSpacing=0 cellPadding=0 width=7 border=1>
<TBODY>
<TR>
<TD bgColor=#9ec21a><IMG height=7
src="DEALERS_files/spacer.gif" width=7
border=1></TD></TR></TBODY></TABLE></TD>
<FORM name=frmPPI
action=https://outer.peirce.com/cgi-bin/CGIRPC32.DLL/wb500.acu
method=post target=_new><INPUT type=hidden
maxLength=8 size=8 value=16086dmm
name=WW_USERID> <INPUT type=hidden maxLength=10
size=10 value=ppiweb name=WW_PASSWORD> <INPUT
type=hidden value=Submit> </FORM>
<TD><STRONG><A
onclick=document.forms.frmPPI.submit();
href="https://www.peirce.com/dealer_main.cfm#">PPI
Online</A></STRONG><STRONG><BR></STRONG>Click on
the link above to connect to PPI</TD></TR>
<TR vAlign=top>
<TD align=right>
<TABLE borderColor=#ffffff height=7
cellSpacing=0 cellPadding=0 width=7 border=1>
<TBODY>
<TR>
<TD bgColor=#9ec21a><IMG height=7
src="DEALERS_files/spacer.gif" width=7
border=1></TD></TR></TBODY></TABLE></TD>
<TD><STRONG><A
href="https://www.peirce.com/dealer_main.cfm?action=dealer_reports">Dealer
Content</A><BR></STRONG>In this area you have
access to reports uploaded by Peirce Phelps.
These are word, pdf or Access files. You can
click on the file to view the report.</TD></TR>
<TR vAlign=top>
<TD align=right>
<TABLE borderColor=#ffffff height=7
cellSpacing=0 cellPadding=0 width=7 border=1>
<TBODY>
<TR>
<TD bgColor=#9ec21a><IMG height=7
src="DEALERS_files/spacer.gif" width=7
border=1></TD></TR></TBODY></TABLE></TD>
<TD><STRONG><A
href="https://www.peirce.com/training.cfm">Training</A><BR></STRONG>Peirce
Phelps offers a variety of Instructor-led
classes. Take advantage of the training courses
offered. This section includes information about
the course and well as registration
information.</TD></TR>
<TR vAlign=top>
<TD align=right>
<TABLE borderColor=#ffffff height=7
cellSpacing=0 cellPadding=0 width=7 border=1>
<TBODY>
<TR>
<TD bgColor=#9ec21a><IMG height=7
src="DEALERS_files/spacer.gif" width=7
border=1></TD></TR></TBODY></TABLE></TD>
<TD><STRONG><A
href="http://www.hvacpartners.com/"
target=_new>HVAC Partners</A><BR></STRONG>View
the HVAC Partners web site.</TD></TR>
<TR vAlign=top>
<TD align=right>
<TABLE borderColor=#ffffff height=7
cellSpacing=0 cellPadding=0 width=7 border=1>
<TBODY>
<TR>
<TD bgColor=#9ec21a><IMG height=7
src="DEALERS_files/spacer.gif" width=7
border=1></TD></TR></TBODY></TABLE></TD>
<TD><STRONG><A
href="https://www.peirce.com/dealer_main.cfm?action=dealer_As_Is">As-Is
Products</A><BR></STRONG>This section has a
listing of all the As-Is items that you can
purchase. All items here are sold in As-Is
condition. </TD></TR>
<TR vAlign=top>
<TD align=right width="7%">
<TABLE borderColor=#ffffff height=7
cellSpacing=0 cellPadding=0 width=7 border=1>
<TBODY>
<TR>
<TD bgColor=#9ec21a><IMG height=7
src="DEALERS_files/spacer.gif" width=7
border=1></TD></TR></TBODY></TABLE></TD>
<TD width="93%"><STRONG><A
href="https://www.peirce.com/dealer_main.cfm?action=dealer_profile">Edit
Dealer Profile</A><BR></STRONG>You can update
your profile in this section. </TD></TR>
<TR vAlign=top>
<TD align=right>
<TABLE borderColor=#ffffff height=7
cellSpacing=0 cellPadding=0 width=7 border=1>
<TBODY>
<TR>
<TD bgColor=#9ec21a><IMG height=7
src="DEALERS_files/spacer.gif" width=7
border=1></TD></TR></TBODY></TABLE></TD>
<TD><STRONG><A
href="https://www.peirce.com/dealer_main.cfm?action=dealer_sub">Sub-Accounts</A><BR></STRONG>Use
this administrative tool to view and add new sub
logins for your account.</TD></TR></TBODY></TABLE>
<P><BR><BR></P>
<P> </P></FONT></TD></TR></TBODY></TABLE></TD></TR></TBODY></TABLE></TD></TR></TBODY></TABLE></TD></TR>
<TR bgColor=#7e8776>
<TD vAlign=top><IMG height=1 src="DEALERS_files/spacer.gif"
width=22></TD></TR>
<TR>
<TD height=25><FONT face="Verdana, Arial, Helvetica, sans-serif"
color=#7e8776 size=1> Copyright © 2007, Peirce Phelps, All
Rights Reserved | Terms and Conditions</FONT></TD></TR>
<TR bgColor=#7e8776>
<TD><IMG height=1 src="DEALERS_files/spacer.gif"
width=22></TD></TR></TBODY></TABLE></TD>
<TD width=1 bgColor=#999999><IMG height=1 src="DEALERS_files/spacer.gif"
width=1></TD></TR></TBODY></TABLE>
<SCRIPT src="DEALERS_files/urchin.js" type=text/javascript>
      </SCRIPT>

<SCRIPT type=text/javascript>
      _uacct = "UA-870264-1";
      urchinTracker();
      </SCRIPT>
</BODY></HTML>

From there I click on PPI ONLINE. That is when it takes me to the CGI Page where I click Search Then type my item number in and it gives me a price for that ITEM.

And thats what I need done using python. I am familiar with python and have written things like this before but I never had to login using a cgi script .. Only ever did it from the address bar.

Any help at all would be great.

Thanks
Scog

Tintin

What is your Python scripting knowledge/level?

scogger1974

ASKER

I would say Mid Level. About 3 Linux Server Programs in python and I have been studying and working with other aspects such as gui building for about 1 year.

scogger1974

ASKER

Please can anyone help at all ???

Tintin

Unfortunately my python skills are very limited. I could do it in Perl, but I guess you don't want a Perl solution.

mish33

Look at urllib2 in standard library for CGI authentication
and at http://www.crummy.com/software/BeautifulSoup/ for HTML parsing.

scogger1974

ASKER

mish33: I see that Beautiful Soup is good for reading webpages but that isnt really my problem. I can do that without trouble. What I really need is to be able to send my username and password and get a response. Then send the correct command to get the information I need then parsing it is a breeze.

Tintin: If you could do it in perl I am interested. There are a couple of things that I have converted in perl to python that I didnt know how to do and I could only find it in perl.

scogger1974

ASKER

Anyone have any Idea on this one???

signedlongint

You can use the httplib, htmllib and Cookie modules to do everything you need. And they're all pretty simply to use, too.

ASKER CERTIFIED SOLUTION

nochkin

membership

This solution is only available to members.

To access this solution, you must be a member of Experts Exchange.

Start Free Trial

scogger1974

ASKER

Thanks, I have been trying to solve it for a while... put it down and when i got your response I tried again and it worked.... I appreciate it..

scog