Solved

Java http client - automated web browsing

Posted on 2003-11-27
8
352 Views
Last Modified: 2008-03-17
Hi,

I asked this question in the Web Languages section previosuly, but didn't get any replies.

I decided in any case that I would like to go with a Java implementation if possible, since I am familiar with this language.  Does anyone know of any existing libraries or classes that would make something like this easier?  In particular classes that allow establishment of an http connection, storing of cookies, etc.  I was thinking about using httpUnit.  Is this a good choice?

Original question:

I need to automate browsing of a particular site.  That is, I must be able to programatically download the pages associated with the site for parsing and analysis.  Additionally, I need to be able to fill out and submit forms in an automated way, as well as support cookies (the site might require the information in the cookie in order to provide context for certain pages).

There are probably several ways to crack this egg, so in the case where there are several arguably equal solutions, I would prefer the ones that leverage the languages or technologies that I am familar with: C++, Java, Sockets.  I have superficial familarity with html, javascript, VB etc.  This application needs to run on the Win32 platform.

Thanks in advance for your suggestions.
0
Comment
Question by:travd
  • 3
  • 3
  • 2
8 Comments
 
LVL 92

Expert Comment

by:objects
ID: 9833749
Yes httpunit seems like a good match
http://www.junit.org/news/extension/web/index.htm
0
 
LVL 1

Author Comment

by:travd
ID: 9833984
Looking at those links, it seems there are three good possibilites - jWebUnit, htmlUnit and httpUnit.   Does anyone have any experience with any of these?  Is it possible to use these without JUnit - I'm not really running a unit test, but instead trying to have automatic form submission, combined with harvesting and processing the returned data.
0
 
LVL 15

Accepted Solution

by:
jimmack earned 125 total points
ID: 9838623
I've used httpUnit to do the first part of what you are asking (automatically accessing a web site and then parsing the result to see if I've got the data I want).  It works fine for this.

The WebResponse object that you get back (in response to a WebRequest), allows you to identify Forms, Links, Tables etc. fairly easily.

It can also handle Cookies and you should be able to submit Form data back to the server as necessary.

httpUnit requires JUnit, since it is effectively an extension to it (but you should look into this too because it's very useful to automatically test your Java applications ;-))

I haven't used jWebUnit or htmlUnit.
0
 
LVL 1

Author Comment

by:travd
ID: 9840977
OK, so I need to have JUnit, but I just want to run this as a normal application - I don't need to actually run it as a unit test, do I?

Thanks,
T.
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 
LVL 92

Expert Comment

by:objects
ID: 9840989
> I don't need to actually run it as a unit test

just a matter of semantics isn't it.
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9841869
That's right, just because you have the facilities availble to do "testing", doesn't mean you need to use them.  The application should just run via a normal "main" method.  Just don't use the "assert..." methods ;-)
0
 
LVL 1

Author Comment

by:travd
ID: 9848504
Thanks for your help, I'll go ahead and use http unit. By concern over using the unit test framework was that I didn't want to invoke it using junit and have to see the graphical fail/pass GUI or anything like that - I just want this to be a straight java app.

Thanks.
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9848680
;-)
0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Java contains several comparison operators (e.g., <, <=, >, >=, ==, !=) that allow you to compare primitive values. However, these operators cannot be used to compare the contents of objects. Interface Comparable is used to allow objects of a cl…
Introduction This article is the last of three articles that explain why and how the Experts Exchange QA Team does test automation for our web site. This article covers our test design approach and then goes through a simple test case example, how …
Video by: Michael
Viewers learn about how to reduce the potential repetitiveness of coding in main by developing methods to perform specific tasks for their program. Additionally, objects are introduced for the purpose of learning how to call methods in Java. Define …
This tutorial covers a practical example of lazy loading technique and early loading technique in a Singleton Design Pattern.

708 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now