• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 1137
  • Last Modified:

PYTHON CLIENT SIDE CGI SCRIPT

I need Major Help,  I have this webpage called www.peirce.com That I need to log into and extract information from.  First of all it it a cgi script that I need to end up in to get the information.

So My question consist of tree things.
1.) How do I log into the CGI
2.) How do I realize that I am logged in correctly
3.) How do I know if I need cookie support
4.) And how issue a command to do the search I need done

The last part i beileve would be easy is parsing the text from the webpage to get the data needed.  

Here is the Source Data from right After I login:


<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3c.org/TR/1999/REC-html401-19991224/loose.dtd">
<!-- saved from url=(0038)https://www.peirce.com/dealer_main.cfm -->
<HTML><HEAD><TITLE>DEALERS</TITLE>
<META http-equiv=Content-Type content="text/html; charset=utf-8">
<STYLE type=text/css>BODY {
      MARGIN-TOP: 0px; MARGIN-LEFT: 0px
}
INPUT {
      FONT-SIZE: 11px; BACKGROUND: #efefef; FONT-FAMILY: verdana
}
SELECT {
      FONT-SIZE: 11px; BACKGROUND: #efefef; FONT-FAMILY: verdana
}
TEXTAREA {
      FONT-SIZE: 11px; BACKGROUND: #efefef; FONT-FAMILY: verdana
}
A:link {
      COLOR: #003366; TEXT-DECORATION: none
}
A:visited {
      COLOR: #003366; TEXT-DECORATION: none
}
A:hover {
      COLOR: #990000; TEXT-DECORATION: none
}
A {
      TEXT-DECORATION: none
}
TD {
      FONT-SIZE: 11px; FONT-FAMILY: verdana
}
.links {
      FONT-SIZE: 11px; COLOR: #000000; FONT-FAMILY: verdana
}
.button {
      
}
.toolTip {
      BORDER-RIGHT: #c0c0c0 1px solid; PADDING-RIGHT: 10px; BORDER-TOP: #c0c0c0 1px solid; DISPLAY: block; PADDING-LEFT: 10px; BACKGROUND: #eff3b0; PADDING-BOTTOM: 10px; BORDER-LEFT: #c0c0c0 1px solid; WIDTH: 250px; PADDING-TOP: 10px; BORDER-BOTTOM: #c0c0c0 1px solid; POSITION: absolute
}
.infoTipShow {
      PADDING-RIGHT: 10px; DISPLAY: none; PADDING-LEFT: 10px; PADDING-BOTTOM: 10px; WIDTH: 100%; PADDING-TOP: 10px; POSITION: absolute
}
.toolTipHide {
      BORDER-RIGHT: #c0c0c0 1px solid; PADDING-RIGHT: 10px; BORDER-TOP: #c0c0c0 1px solid; DISPLAY: none; PADDING-LEFT: 10px; BACKGROUND: #eff3b0; PADDING-BOTTOM: 10px; BORDER-LEFT: #c0c0c0 1px solid; WIDTH: 250px; PADDING-TOP: 10px; BORDER-BOTTOM: #c0c0c0 1px solid; POSITION: absolute
}
</STYLE>

<SCRIPT>
thisApp = 'https://www.peirce.com/com/peircephelps/functions.cfm';
</SCRIPT>

<SCRIPT src="DEALERS_files/engine.js" type=text/javascript></SCRIPT>

<SCRIPT src="DEALERS_files/util.js" type=text/javascript></SCRIPT>

<SCRIPT src="DEALERS_files/settings.js" type=text/javascript></SCRIPT>

<SCRIPT src="DEALERS_files/dealer_js.js" type=text/javascript></SCRIPT>

<META content="MSHTML 6.00.2900.3157" name=GENERATOR></HEAD>
<BODY bgColor=#d5d5d5
onload="init();MM_preloadImages('images/company_o.jpg','images/products_o.jpg','images/dealerlocator_o.jpg','images/hvac_o.jpg','images/dealers_o.jpg','images/training_courses_o.jpg','images/contact_o.jpg','images/employment_o.jpg','images/dealer_profile_o.jpg')">
<TABLE cellSpacing=0 cellPadding=0 width=790 align=center border=0>
  <TBODY>
  <TR>
    <TD width=1 bgColor=#999999><IMG height=1 src="DEALERS_files/spacer.gif"
      width=1></TD>
    <TD width=790 bgColor=#ffffff>
      <TABLE cellSpacing=0 cellPadding=0 width=790 border=0>
        <TBODY>
        <TR>
          <TD>
            <TABLE cellSpacing=0 cellPadding=0 width=790 border=0>
              <TBODY>
              <TR>
                <TD width=197><A href="https://www.peirce.com/index.cfm"><IMG
                  height=83 alt=logo src="DEALERS_files/logo.jpg" width=189
                  border=0></A></TD>
                <TD width=197><IMG height=83 alt=topbanner
                  src="DEALERS_files/top2.jpg" width=416></TD>
                <TD width=197><IMG height=83 alt=news
                  src="DEALERS_files/news.jpg" width=119></TD>
                <TD width=199><A href="https://www.peirce.com/index.cfm"><IMG
                  height=83 alt=home src="DEALERS_files/home.jpg" width=66
                  border=0></A></TD></TR></TBODY></TABLE></TD></TR>
        <TR>
          <TD vAlign=top>
            <TABLE cellSpacing=0 cellPadding=0 width=790 border=0>
              <TBODY>
              <TR>
                <TD vAlign=top width=181>
                  <SCRIPT language=JavaScript type=text/JavaScript>
<!--
function MM_preloadImages() { //v3.0
  var d=document; if(d.images){ if(!d.MM_p) d.MM_p=new Array();
    var i,j=d.MM_p.length,a=MM_preloadImages.arguments; for(i=0; i<a.length; i++)
    if (a[i].indexOf("#")!=0){ d.MM_p[j]=new Image; d.MM_p[j++].src=a[i];}}
}

function MM_swapImgRestore() { //v3.0
  var i,x,a=document.MM_sr; for(i=0;a&&i<a.length&&(x=a[i])&&x.oSrc;i++) x.src=x.oSrc;
}

function MM_findObj(n, d) { //v4.01
  var p,i,x;  if(!d) d=document; if((p=n.indexOf("?"))>0&&parent.frames.length) {
    d=parent.frames[n.substring(p+1)].document; n=n.substring(0,p);}
  if(!(x=d[n])&&d.all) x=d.all[n]; for (i=0;!x&&i<d.forms.length;i++) x=d.forms[i][n];
  for(i=0;!x&&d.layers&&i<d.layers.length;i++) x=MM_findObj(n,d.layers[i].document);
  if(!x && d.getElementById) x=d.getElementById(n); return x;
}

function MM_swapImage() { //v3.0
  var i,j=0,x,a=MM_swapImage.arguments; document.MM_sr=new Array; for(i=0;i<(a.length-2);i+=3)
   if ((x=MM_findObj(a[i]))!=null){document.MM_sr[j++]=x; if(!x.oSrc) x.oSrc=x.src; x.src=a[i+2];}
}
//-->
</SCRIPT>

                  <TABLE cellSpacing=0 cellPadding=0 width=181 border=0>
                    <TBODY>
                    <TR>
                      <TD vAlign=top width=181>
                        <TABLE cellSpacing=0 cellPadding=0 width=181 border=0>
                          <TBODY>
                          <TR>
                            <TD height=31><A
                              href="https://www.peirce.com/about.cfm"><IMG
                              id=Image1
                              onmouseover="MM_swapImage('Image1','','images/company_o.jpg',1)"
                              onmouseout=MM_swapImgRestore() height=31
                              alt=company src="DEALERS_files/company.jpg"
                              width=181 border=0 name=Image1></A></TD></TR>
                          <TR>
                            <TD height=25><A
                              href="https://www.peirce.com/products.cfm"><IMG
                              id=Image2
                              onmouseover="MM_swapImage('Image2','','images/products_o.jpg',1)"
                              onmouseout=MM_swapImgRestore() height=25
                              alt=products src="DEALERS_files/products.jpg"
                              width=181 border=0 name=Image2></A></TD></TR>
                          <TR>
                            <TD height=23><A
                              href="http://www.peirce.com/dealer_locator.cfm"><IMG
                              id=Image4
                              onmouseover="MM_swapImage('Image4','','images/dealerlocator_o.jpg',1)"
                              onmouseout=MM_swapImgRestore() height=23
                              alt=dealerlocator
                              src="DEALERS_files/dealerlocator.jpg" width=181
                              border=0 name=Image4></A></TD></TR>
                          <TR>
                            <TD height=26><A
                              href="https://www.peirce.com/howhvacworks.cfm"><IMG
                              id=Image8
                              onmouseover="MM_swapImage('Image81','','images/hvac_o.jpg',1)"
                              onmouseout=MM_swapImgRestore() height=26
                              alt=surveys src="DEALERS_files/hvac.jpg" width=181
                              border=0 name=Image81></A></TD></TR>
                          <TR>
                            <TD height=23><A
                              href="https://www.peirce.com/dealer_register.cfm"><IMG
                              id=Image3
                              onmouseover="MM_swapImage('Image3','','images/dealers_o.jpg',1)"
                              onmouseout=MM_swapImgRestore() height=23
                              alt=dealers src="DEALERS_files/dealers.jpg"
                              width=181 border=0 name=Image3></A></TD></TR>
                          <TR>
                            <TD height=24><A
                              href="https://www.peirce.com/training.cfm"><IMG
                              id=Image5
                              onmouseover="MM_swapImage('Image5','','images/training_courses_o.jpg',1)"
                              onmouseout=MM_swapImgRestore() height=24
                              alt=courses
                              src="DEALERS_files/training_courses.jpg" width=181
                              border=0 name=Image5></A></TD></TR>
                          <TR>
                            <TD height=23><A
                              href="https://www.peirce.com/contact.cfm"><IMG
                              id=Image7
                              onmouseover="MM_swapImage('Image7','','images/contact_o.jpg',1)"
                              onmouseout=MM_swapImgRestore() height=23
                              alt=contact src="DEALERS_files/contact.jpg"
                              width=181 border=0 name=Image7></A></TD></TR>
                          <TR>
                            <TD height=24><A
                              href="https://www.peirce.com/news.cfm"><IMG
                              id=Image66
                              onmouseover="MM_swapImage('Image66','','images/employment_o.jpg',1)"
                              onmouseout=MM_swapImgRestore() height=24 alt=news
                              src="DEALERS_files/employment.jpg" width=181
                              border=0
                    name=Image66></A></TD></TR></TBODY></TABLE></TD></TR>
                    <TR>
                      <TD><BR><BR></TD></TR></TBODY></TABLE></TD>
                <TD vAlign=top width=609>
                  <TABLE cellSpacing=0 cellPadding=0 width=609 border=0>
                    <TBODY>
                    <TR>
                      <TD><IMG height=80
                        src="DEALERS_files/general_top_image.jpg"
                    width=608></TD></TR>
                    <TR>
                      <TD width=609>
                        <TABLE cellSpacing=0 cellPadding=0 width=606 border=0>
                          <TBODY>
                          <TR align=middle bgColor=#d3d7d0>
                            <TD noWrap align=left width=606><IMG height=20
                              src="DEALERS_files/grey_navigation_bar.jpg"
                              width=606 useMap=#Map
                        border=0></TD></TR></TBODY></TABLE><MAP id=Map
                          name=Map><AREA shape=RECT coords=3,1,88,18
                          href="https://www.peirce.com/products_hvac.cfm"><AREA
                          shape=RECT coords=94,2,226,22
                          href="https://www.peirce.com/products_buildingautomation.cfm"><AREA
                          shape=RECT coords=236,1,363,20
                          href="https://www.peirce.com/products_hearth.cfm"><AREA
                          shape=RECT coords=374,0,487,30
                          href="https://www.peirce.com/products_kitchen_bath.cfm"><AREA
                          shape=RECT coords=496,2,601,29
                          href="https://www.peirce.com/products_GRILL.cfm"></MAP></TD></TR>
                    <TR>
                      <TD bgColor=#7c7d77><IMG height=1
                        src="DEALERS_files/spacer.gif" width=22></TD></TR>
                    <TR>
                      <TD vAlign=top>
                        <TABLE cellSpacing=0 cellPadding=10 width="100%"
                        border=0>
                          <TBODY>
                          <TR>
                            <TD bgColor=#efefef>
                              <TABLE cellSpacing=0 cellPadding=0 width=590
                              border=0>
                                <TBODY>
                                <TR>
                                <TD width=20>
                                <TABLE borderColor=#efefef height=12
                                cellSpacing=0 cellPadding=0 width=12 border=1>
                                <TBODY>
                                <TR>
                                <TD bgColor=#143874><IMG height=10
                                src="DEALERS_files/spacer.gif" width=10
                                border=1></TD></TR></TBODY></TABLE></TD>
                                <TD width=140><STRONG><FONT
                                face="Verdana, Arial, Helvetica, sans-serif"
                                color=#143874
                                size=4>&nbsp;DEALERS</FONT></STRONG></TD>
                                <TD align=right
                                width=429><STRONG></STRONG></TD></TR></TBODY></TABLE></TD></TR>
                          <TR>
                            <TD vAlign=top><BR><FONT color=#143874>
                              <TABLE width="100%">
                                <TBODY>
                                <TR>
                                <TD width="50%"><STRONG>Welcome, Dawn Marie
                                McDonnell</STRONG><BR><EM>offenbachers</EM> [<A
                                href="https://www.peirce.com/dealer_main.cfm?logout=1">Logout</A>]</TD>
                                <TD align=right
                                width="50%">&nbsp;</TD></TR></TBODY></TABLE>
                              <TABLE cellSpacing=0 cellPadding=7 width="100%"
                              border=0>
                                <TBODY>
                                <TR>
                                <TD colSpan=2>You have access to the following
                                areas. Click on a link below to
                                continue</TD></TR>
                                <TR vAlign=top>
                                <TD align=right>
                                <TABLE borderColor=#ffffff height=7
                                cellSpacing=0 cellPadding=0 width=7 border=1>
                                <TBODY>
                                <TR>
                                <TD bgColor=#9ec21a><IMG height=7
                                src="DEALERS_files/spacer.gif" width=7
                                border=1></TD></TR></TBODY></TABLE></TD>
                                <FORM name=frmPPI
                                action=https://outer.peirce.com/cgi-bin/CGIRPC32.DLL/wb500.acu
                                method=post target=_new><INPUT type=hidden
                                maxLength=8 size=8 value=16086dmm
                                name=WW_USERID> <INPUT type=hidden maxLength=10
                                size=10 value=ppiweb name=WW_PASSWORD> <INPUT
                                type=hidden value=Submit> </FORM>
                                <TD><STRONG><A
                                onclick=document.forms.frmPPI.submit();
                                href="https://www.peirce.com/dealer_main.cfm#">PPI
                                Online</A></STRONG><STRONG><BR></STRONG>Click on
                                the link above to connect to PPI</TD></TR>
                                <TR vAlign=top>
                                <TD align=right>
                                <TABLE borderColor=#ffffff height=7
                                cellSpacing=0 cellPadding=0 width=7 border=1>
                                <TBODY>
                                <TR>
                                <TD bgColor=#9ec21a><IMG height=7
                                src="DEALERS_files/spacer.gif" width=7
                                border=1></TD></TR></TBODY></TABLE></TD>
                                <TD><STRONG><A
                                href="https://www.peirce.com/dealer_main.cfm?action=dealer_reports">Dealer
                                Content</A><BR></STRONG>In this area you have
                                access to reports uploaded by Peirce Phelps.
                                These are word, pdf or Access files. You can
                                click on the file to view the report.</TD></TR>
                                <TR vAlign=top>
                                <TD align=right>
                                <TABLE borderColor=#ffffff height=7
                                cellSpacing=0 cellPadding=0 width=7 border=1>
                                <TBODY>
                                <TR>
                                <TD bgColor=#9ec21a><IMG height=7
                                src="DEALERS_files/spacer.gif" width=7
                                border=1></TD></TR></TBODY></TABLE></TD>
                                <TD><STRONG><A
                                href="https://www.peirce.com/training.cfm">Training</A><BR></STRONG>Peirce
                                Phelps offers a variety of Instructor-led
                                classes. Take advantage of the training courses
                                offered. This section includes information about
                                the course and well as registration
                                information.</TD></TR>
                                <TR vAlign=top>
                                <TD align=right>
                                <TABLE borderColor=#ffffff height=7
                                cellSpacing=0 cellPadding=0 width=7 border=1>
                                <TBODY>
                                <TR>
                                <TD bgColor=#9ec21a><IMG height=7
                                src="DEALERS_files/spacer.gif" width=7
                                border=1></TD></TR></TBODY></TABLE></TD>
                                <TD><STRONG><A
                                href="http://www.hvacpartners.com/"
                                target=_new>HVAC Partners</A><BR></STRONG>View
                                the HVAC Partners web site.</TD></TR>
                                <TR vAlign=top>
                                <TD align=right>
                                <TABLE borderColor=#ffffff height=7
                                cellSpacing=0 cellPadding=0 width=7 border=1>
                                <TBODY>
                                <TR>
                                <TD bgColor=#9ec21a><IMG height=7
                                src="DEALERS_files/spacer.gif" width=7
                                border=1></TD></TR></TBODY></TABLE></TD>
                                <TD><STRONG><A
                                href="https://www.peirce.com/dealer_main.cfm?action=dealer_As_Is">As-Is
                                Products</A><BR></STRONG>This section has a
                                listing of all the As-Is items that you can
                                purchase. All items here are sold in As-Is
                                condition. </TD></TR>
                                <TR vAlign=top>
                                <TD align=right width="7%">
                                <TABLE borderColor=#ffffff height=7
                                cellSpacing=0 cellPadding=0 width=7 border=1>
                                <TBODY>
                                <TR>
                                <TD bgColor=#9ec21a><IMG height=7
                                src="DEALERS_files/spacer.gif" width=7
                                border=1></TD></TR></TBODY></TABLE></TD>
                                <TD width="93%"><STRONG><A
                                href="https://www.peirce.com/dealer_main.cfm?action=dealer_profile">Edit
                                Dealer Profile</A><BR></STRONG>You can update
                                your profile in this section. </TD></TR>
                                <TR vAlign=top>
                                <TD align=right>
                                <TABLE borderColor=#ffffff height=7
                                cellSpacing=0 cellPadding=0 width=7 border=1>
                                <TBODY>
                                <TR>
                                <TD bgColor=#9ec21a><IMG height=7
                                src="DEALERS_files/spacer.gif" width=7
                                border=1></TD></TR></TBODY></TABLE></TD>
                                <TD><STRONG><A
                                href="https://www.peirce.com/dealer_main.cfm?action=dealer_sub">Sub-Accounts</A><BR></STRONG>Use
                                this administrative tool to view and add new sub
                                logins for your account.</TD></TR></TBODY></TABLE>
                              <P><BR><BR></P>
                              <P>&nbsp;</P></FONT></TD></TR></TBODY></TABLE></TD></TR></TBODY></TABLE></TD></TR></TBODY></TABLE></TD></TR>
        <TR bgColor=#7e8776>
          <TD vAlign=top><IMG height=1 src="DEALERS_files/spacer.gif"
          width=22></TD></TR>
        <TR>
          <TD height=25><FONT face="Verdana, Arial, Helvetica, sans-serif"
            color=#7e8776 size=1>&nbsp;Copyright © 2007, Peirce Phelps, All
            Rights Reserved | Terms and Conditions</FONT></TD></TR>
        <TR bgColor=#7e8776>
          <TD><IMG height=1 src="DEALERS_files/spacer.gif"
        width=22></TD></TR></TBODY></TABLE></TD>
    <TD width=1 bgColor=#999999><IMG height=1 src="DEALERS_files/spacer.gif"
      width=1></TD></TR></TBODY></TABLE>
<SCRIPT src="DEALERS_files/urchin.js" type=text/javascript>
      </SCRIPT>

<SCRIPT type=text/javascript>
      _uacct = "UA-870264-1";
      urchinTracker();
      </SCRIPT>
</BODY></HTML>

From there I click on PPI ONLINE.  That is when it takes me to the CGI Page where I click Search Then type my item number in and it gives me a price for that ITEM.  

And thats what I need done using python.  I am familiar with python and have written things like this before but I never had to login using a cgi script .. Only ever did it from the address bar.

Any help at all would be great.

Thanks
Scog
0
scogger1974
Asked:
scogger1974
1 Solution
 
TintinCommented:
What is your Python scripting knowledge/level?
0
 
scogger1974Author Commented:
I would say Mid Level.  About 3  Linux Server Programs in python and I have been studying and working with other aspects such as gui building for about 1 year.
0
 
scogger1974Author Commented:
Please can anyone help at all ???
0
Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

 
TintinCommented:
Unfortunately my python skills are very limited.  I could do it in Perl, but I guess you don't want a Perl solution.
0
 
mish33Commented:
Look at urllib2 in standard library for CGI authentication
and at http://www.crummy.com/software/BeautifulSoup/ for HTML parsing.
0
 
scogger1974Author Commented:
mish33:  I see that Beautiful Soup is good for reading webpages but that isnt really my problem.  I can do that without trouble.  What I really need is to be able to send my username and password and get a response.  Then send the correct command to get the information I need then parsing it is a breeze.  

Tintin: If you could do it in perl I am interested.   There are a couple of things that I have converted in perl to python that I didnt know how to do and I could only find it in perl.
0
 
scogger1974Author Commented:
Anyone have any Idea on this one???
0
 
signedlongintCommented:
You can use the httplib, htmllib and Cookie modules to do everything you need. And they're all pretty simply to use, too.
0
 
nochkinCommented:
I'm not going to make a full source code, because this job won't simply fit in here, but I can give you some ideas to start.

1.) Simple form with user and password will do the job on the client side (browser).
Server-side (cgi script) will take information using python "cgi" module and decide if login info is good or not.
If you have many users then I would recommend to use a database (python has modules for each database so it's your choice)

2.) Cookies is the best bet here. Once you accepted login information then you can assign a session cookie and store in in the database assigned to some specific userid. Then on every request which is suppose to be password protected you can check that cookie to make sure it belongs to someone and then load the page. If cookie is wrong (someone is trying to fake it) then you can erase it and redirect to the login page. Python has "Cookie" module to process http cookies.

3.) Cookies will give you more flexibility than passing an argument with each request. If you don't want to use cookies then you can supply "sid=sessionidcookie" argument where "sessionidcookie" is your session cookie. But in this case your urls will not look nice and there are more chances someone will try to fake it.

4.) If you store your data in the database then simple "SELECT" query will do the job.
If you don't store in the database (which is not recommended) then it all depends on the way you store it. Text files can be opened, read and then parsed with information you need.
0
 
scogger1974Author Commented:
Thanks,  I have been trying to solve it for a while... put it down and when i got your response I tried again and it worked.... I appreciate it..

scog
0

Featured Post

Upgrade your Question Security!

Add Premium security features to your question to ensure its privacy or anonymity. Learn more about your ability to control Question Security today.

Tackle projects and never again get stuck behind a technical roadblock.
Join Now