Solved

Java http client - automated web browsing

Posted on 2003-11-27
8
355 Views
Last Modified: 2008-03-17
Hi,

I asked this question in the Web Languages section previosuly, but didn't get any replies.

I decided in any case that I would like to go with a Java implementation if possible, since I am familiar with this language.  Does anyone know of any existing libraries or classes that would make something like this easier?  In particular classes that allow establishment of an http connection, storing of cookies, etc.  I was thinking about using httpUnit.  Is this a good choice?

Original question:

I need to automate browsing of a particular site.  That is, I must be able to programatically download the pages associated with the site for parsing and analysis.  Additionally, I need to be able to fill out and submit forms in an automated way, as well as support cookies (the site might require the information in the cookie in order to provide context for certain pages).

There are probably several ways to crack this egg, so in the case where there are several arguably equal solutions, I would prefer the ones that leverage the languages or technologies that I am familar with: C++, Java, Sockets.  I have superficial familarity with html, javascript, VB etc.  This application needs to run on the Win32 platform.

Thanks in advance for your suggestions.
0
Comment
Question by:travd
  • 3
  • 3
  • 2
8 Comments
 
LVL 92

Expert Comment

by:objects
ID: 9833749
Yes httpunit seems like a good match
http://www.junit.org/news/extension/web/index.htm
0
 
LVL 1

Author Comment

by:travd
ID: 9833984
Looking at those links, it seems there are three good possibilites - jWebUnit, htmlUnit and httpUnit.   Does anyone have any experience with any of these?  Is it possible to use these without JUnit - I'm not really running a unit test, but instead trying to have automatic form submission, combined with harvesting and processing the returned data.
0
 
LVL 15

Accepted Solution

by:
jimmack earned 125 total points
ID: 9838623
I've used httpUnit to do the first part of what you are asking (automatically accessing a web site and then parsing the result to see if I've got the data I want).  It works fine for this.

The WebResponse object that you get back (in response to a WebRequest), allows you to identify Forms, Links, Tables etc. fairly easily.

It can also handle Cookies and you should be able to submit Form data back to the server as necessary.

httpUnit requires JUnit, since it is effectively an extension to it (but you should look into this too because it's very useful to automatically test your Java applications ;-))

I haven't used jWebUnit or htmlUnit.
0
Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

 
LVL 1

Author Comment

by:travd
ID: 9840977
OK, so I need to have JUnit, but I just want to run this as a normal application - I don't need to actually run it as a unit test, do I?

Thanks,
T.
0
 
LVL 92

Expert Comment

by:objects
ID: 9840989
> I don't need to actually run it as a unit test

just a matter of semantics isn't it.
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9841869
That's right, just because you have the facilities availble to do "testing", doesn't mean you need to use them.  The application should just run via a normal "main" method.  Just don't use the "assert..." methods ;-)
0
 
LVL 1

Author Comment

by:travd
ID: 9848504
Thanks for your help, I'll go ahead and use http unit. By concern over using the unit test framework was that I didn't want to invoke it using junit and have to see the graphical fail/pass GUI or anything like that - I just want this to be a straight java app.

Thanks.
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9848680
;-)
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
servlet example 17 53
Java basic valueOf question 1 29
running on tomcat not jboss eap 7.0 3 18
Java class and jar 3 25
Introduction Java can be integrated with native programs using an interface called JNI(Java Native Interface). Native programs are programs which can directly run on the processor. JNI is simply a naming and calling convention so that the JVM (Java…
Java Flight Recorder and Java Mission Control together create a complete tool chain to continuously collect low level and detailed runtime information enabling after-the-fact incident analysis. Java Flight Recorder is a profiling and event collectio…
This tutorial covers a step-by-step guide to install VisualVM launcher in eclipse.
This tutorial explains how to use the VisualVM tool for the Java platform application. This video goes into detail on the Threads, Sampler, and Profiler tabs.

828 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question