Signing in using Java

Folks,
I am writing a program for people to manage their ebay auctions.  However I have come to a hurdle.

If you look at this link:

http://cgi.ebay.co.uk/ws/eBayISAPI.dll?ViewItem&item=4451836193

If you scroll down you see in the summary of the auction "History: Purchases" which is a link to:

http://offer.ebay.co.uk/ws/eBayISAPI.dll?ViewBidsLogin&item=4451836193

However to view this page it is required to sign in.  At the moment I am downloading details of the auctions using this code, however it does not work to view the links which require sign in.

How can I by pass this?  The success of this software package greatly depends on the ability to full manage the auctions for the owner - i.e all the purchases etc.. etc..  Any suggestions??

I have created test ebay id:

id: expertsexchangetest
password: experts123

  String readString = "";
  String searchPageURL  = "http://cgi.ebay.co.uk/ws/eBayISAPI.dll?ViewItem&item=4451836193";
  BufferedReader HTMLpage;
  URL url;
  URLConnection urlConnection;
  boolean stop = false;
  try
  {
      //Connecting to URL
      url = new URL(searchPageURL);
      urlConnection = url.openConnection();
      HTMLpage = new BufferedReader(new InputStreamReader(url.openStream()));
      //Looping through Web Page
      while((readString = HTMLpage.readLine()) != null && !stop)
      {
        System.out.println(readString);
      }
      HTMLpage.close();
    }
LVL 2
amacfarlAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

gauravkrtomarCommented:
That's a very primitive way of doing the task. Try a latest open-source project by Apache:-
http://jakarta.apache.org/commons/httpclient/

Using this u can do whatever u can do using a browser. It maintains Sessions as well, thus making it feasible to use Signin on sites.
U can do something as follows:-
                HttpClient client = new HttpClient();
            
            GetMethod method = new GetMethod("http://www.xyz.com");
            PostMethod loginMethod = new PostMethod(
                        "http://www.xyz.com/login");
            postMethod.addParameter("username", username);
            postMethod.addParameter("password", password);

            GetMethod anotherMethod = new GetMethod(
                        "http://www.xyz.com/dosomething");
                 try {
                  client.executeMethod(method);
                  System.out.println("started...");
                  client.executeMethod(loginMethod);
                  System.out.println("authenticaed...");
                  client.executeMethod(anotherMethod);
            } catch (HttpException e) {
                  e.printStackTrace();
            } catch (IOException e) {
                  e.printStackTrace();
            }
I think this will solve ur problem.
0
Siva Prasanna KumarPrincipal Solutions ArchitectCommented:
Can you please be clear what you wanted is to pass the required parameters for signing in?????

or some thing else???

I think you are intending to make auto signing kind of stuff if i am correct.
0
gauravkrtomarCommented:
I forgot to tell you that as the HttpClient maintains the Session information you need not provide the signin info again and again (almost similar like in the case of the Browser). So HttpClient is a browser and all the PostMethod/GetMethod are the form submits and page requests respectively that u will be making to the site.
0
Cloud Class® Course: CompTIA Cloud+

The CompTIA Cloud+ Basic training course will teach you about cloud concepts and models, data storage, networking, and network infrastructure.

amacfarlAuthor Commented:
Wow - Thanks for the prompt response!

Shivaspk,
In a nut shell what I am trying to do is download all the auction information into a database.  For the first link (below) there is no log in required and I simply parse the data and write to a db.  For the (2) link a login is required and this is the point that I am stuck on.

(1)http://cgi.ebay.co.uk/ws/eBayISAPI.dll?ViewItem&item=4451836193
(2)http://offer.ebay.co.uk/ws/eBayISAPI.dll?ViewBidsLogin&item=4451836193

Gauravkrtomar,
Thanks for the info.  I am going to review the web site and work out how to implement your code.  I am using JBuilder 9 Enterprise to develop this application and need to work out how to install the open-source project.  I also need to work out how to parse it.  Any tips would be most welcome.

Cheers
0
gauravkrtomarCommented:
for (1)http://cgi.ebay.co.uk/ws/eBayISAPI.dll?ViewItem&item=4451836193 u can directly use the GetMethod

for (2)http://offer.ebay.co.uk/ws/eBayISAPI.dll?ViewBidsLogin&item=4451836193 before issuing a getmethod to this link just issue a getmethod/postmethod to the login page (whichever is applicable, u can that info by viewing the source of the html, if form is using post method use the postmethod otherwise use the getmethod).

>> I am using JBuilder 9 Enterprise to develop this application and need to work out how to install the open-source project
U just need to download the jar from the site and put that in the project classpath of the Jbuilder, surely jbuilder should have something like add libraries or something like that. After that u can start working on your classes .
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.methods.GetMethod;
import org.apache.commons.httpclient.methods.PostMethod;
.................................................................

>> I also need to work out how to parse it.  Any tips would be most welcome.
U can use http://tidy.sourceforge.net/
though it is used to tidy the Html page but u can use it to parse the html page into DOM :).


import java.io.InputStream;
import java.io.PrintWriter;
import java.util.Vector;

import org.apache.log4j.Category;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.NodeList;
import org.w3c.tidy.Tidy;

/**
 * This class implements an HTML document
 *
 * It uses JTidy to parse the given HTML code to an internal DOM representation.
 *
 */
public class HtmlDocument {

      /** Content text as an array of bytes (this is how we get it from HTTP !) */
      private InputStream contentStream = null;

      /** the DOM representation of this HTML document */
      private Document domDoc = null;

      /** Log4J category for logging purposes */
      private Category log;

      /**
       * Initializes an HTML document with the given content.
       *
       */
      public HtmlDocument(InputStream inStream) {
            contentStream = inStream;
            parseToDOM(contentStream);
      }

      /**
       * gets all Element nodes of a given type as a Vector
       *
       * @param type
       *            the type of elements to return. e.g. type="a" will return all
       *            <A> tags. type must be lowercase
       * @return a Vector containing all element nodes of the given type
       */
      public Vector getElements(String type) {
            if (domDoc == null) {
                  parseToDOM(contentStream);
            }

            Vector links = new Vector();
            extractElements(domDoc.getDocumentElement(), type, links);

            return links;
      }

      /**
       * Extract elements from the given DOM subtree and put it into the given
       * vector.
       *
       * @param element
       *            the top level DOM element of the DOM tree to parse
       * @param type
       *            HTML tag to extract (e.g. "a", "form", "head" ...)
       * @param elementList
       *            the vector that will store the elements
       */
      protected void extractElements(Element element, String type,
                  Vector elementList) {

            // this should not happen !
            if (element == null) {
                  log.error("got a null element");
                  return;
            }

            String name = element.getNodeName();

            if (name.equals(type)) {
                  elementList.add(element);
            }

            // recursive travel through all childs
            NodeList childs = element.getChildNodes();

            for (int i = 0; i < childs.getLength(); i++) {
                  if (childs.item(i) instanceof Element) {
                        extractElements((Element) childs.item(i), type, elementList);
                  }
            }

      }

      /**
       * parses the document to a DOM tree using Tidy
       */
      private void parseToDOM(InputStream is) {
            // set tidy parameters
            Tidy tidy = new Tidy();
            tidy.setUpperCaseTags(false);
            tidy.setUpperCaseAttrs(false);
            try {
                  tidy.setErrout(new PrintWriter(System.out));
            } catch (Exception e) {
                  e.printStackTrace();
            }

            domDoc = tidy.parseDOM(is, null);
      }

}


And the other method is hard core string parsing using regular expressions and BufferedReader.
i.e. reading each line and getting wat u want all programmaticaly.
It all depends on ur need ;)
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
amacfarlAuthor Commented:
Wow.... I am getting somewhere fast.  I have successfully implemented the Apache code that you suggested and the following code works:


    HttpClient client = new HttpClient();
    PostMethod loginMethod = new PostMethod("https://signin.ebay.co.uk/ws/eBayISAPI.dll?co_partnerid=2&siteid=3&UsingSSL=1");
    loginMethod.addParameter("userid", "expertsexchangetest");
    loginMethod.addParameter("pass", "experts123");
    GetMethod method = new GetMethod("http://offer.ebay.co.uk/ws/eBayISAPI.dll?ViewBidsLogin&item=4451836193");
    try
    {
      client.executeMethod(loginMethod);
      System.out.println("authenticaed...");
      client.executeMethod(method);
      System.out.println("successful");
    }
    catch (Exception ex)
    {
      ex.printStackTrace();
    }

I have a couple of questions though:

1) The code is printing out the following log text: how can I disable this.  Some people who will be using this product have 20,000 auctions and it is best not to have the logging.
--------------------------------------------------------
13-Apr-2006 15:11:24 org.apache.commons.httpclient.HttpMethodBase readResponse
INFO: Discarding unexpected response: HTTP/1.1 100 Continue
13-Apr-2006 15:11:24 org.apache.commons.httpclient.HttpMethodDirector isRedirectNeeded
INFO: Redirect requested but followRedirects is disabled
13-Apr-2006 15:11:24 org.apache.commons.httpclient.SimpleHttpConnectionManager getConnectionWithTimeout
WARNING: SimpleHttpConnectionManager being used incorrectly.  Be sure that HttpMethod.releaseConnection() is always called and that only one thread and/or method is using this connection manager at a time.
authenticaed...
successful

2) Parsing.  I have already written a class that uses BufferedReader to parse the code.  I know this is ugly and hard core, however it works.  How can I output the source code of the GetMethod to the BufferedReader or to a simple String.  It would be simpler/quicker than rewriting the code using http://tidy.sourceforge.net/

THANKS THANKS AND THANKS for all your help!
0
gauravkrtomarCommented:
>>How can I output the source code of the GetMethod to the BufferedReader or to a simple String.  It would be simpler/quicker than rewriting the code using http://tidy.sourceforge.net/

U can use getMethod.getResponseBodyAsStream()
or            getMethod.getResponseBodyAsString().

for setting the logging off try:-
 System.setProperty("org.apache.commons.logging.Log", "org.apache.commons.logging.impl.NoOpLog");
0
amacfarlAuthor Commented:
thanks again for all your help.  Your recommendations have worked.  The issue that I am having now is that with the fact that the Log On is via SSL.

Therefore the logon is not successful and the reading of the (2) link is redirecting back to the logon screen.

Trying to fix it.
0
amacfarlAuthor Commented:
hmmm.  puzzling... I tried simply this:

    HttpClient httpclient = new HttpClient();
    GetMethod httpget = new GetMethod("https://www.verisign.com/");
    try {
      httpclient.executeMethod(httpget);
      System.out.println(httpget.getStatusLine());
    }
      catch (Exception exe)
      {
        exe.printStackTrace();
      }
    finally {
      httpget.releaseConnection();
    }

And get this error

javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: CA certificate does not include basic constraints extension

      at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.a(DashoA6275)

0
gauravkrtomarCommented:
http://jakarta.apache.org/commons/httpclient/sslguide.html

Look at the above link. It has all the details about how to connect to secure HTTP communication over SSL.
0
eNarcCommented:
haha I just noticed expertsexchange is expert sex change!!! hahaha! funny never saw that before :D
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Java

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.