Solved

Java http client - automated web browsing

Posted on 2003-11-27
8
357 Views
Last Modified: 2008-03-17
Hi,

I asked this question in the Web Languages section previosuly, but didn't get any replies.

I decided in any case that I would like to go with a Java implementation if possible, since I am familiar with this language.  Does anyone know of any existing libraries or classes that would make something like this easier?  In particular classes that allow establishment of an http connection, storing of cookies, etc.  I was thinking about using httpUnit.  Is this a good choice?

Original question:

I need to automate browsing of a particular site.  That is, I must be able to programatically download the pages associated with the site for parsing and analysis.  Additionally, I need to be able to fill out and submit forms in an automated way, as well as support cookies (the site might require the information in the cookie in order to provide context for certain pages).

There are probably several ways to crack this egg, so in the case where there are several arguably equal solutions, I would prefer the ones that leverage the languages or technologies that I am familar with: C++, Java, Sockets.  I have superficial familarity with html, javascript, VB etc.  This application needs to run on the Win32 platform.

Thanks in advance for your suggestions.
0
Comment
Question by:travd
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
  • 2
8 Comments
 
LVL 92

Expert Comment

by:objects
ID: 9833749
Yes httpunit seems like a good match
http://www.junit.org/news/extension/web/index.htm
0
 
LVL 1

Author Comment

by:travd
ID: 9833984
Looking at those links, it seems there are three good possibilites - jWebUnit, htmlUnit and httpUnit.   Does anyone have any experience with any of these?  Is it possible to use these without JUnit - I'm not really running a unit test, but instead trying to have automatic form submission, combined with harvesting and processing the returned data.
0
 
LVL 15

Accepted Solution

by:
jimmack earned 125 total points
ID: 9838623
I've used httpUnit to do the first part of what you are asking (automatically accessing a web site and then parsing the result to see if I've got the data I want).  It works fine for this.

The WebResponse object that you get back (in response to a WebRequest), allows you to identify Forms, Links, Tables etc. fairly easily.

It can also handle Cookies and you should be able to submit Form data back to the server as necessary.

httpUnit requires JUnit, since it is effectively an extension to it (but you should look into this too because it's very useful to automatically test your Java applications ;-))

I haven't used jWebUnit or htmlUnit.
0
Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

 
LVL 1

Author Comment

by:travd
ID: 9840977
OK, so I need to have JUnit, but I just want to run this as a normal application - I don't need to actually run it as a unit test, do I?

Thanks,
T.
0
 
LVL 92

Expert Comment

by:objects
ID: 9840989
> I don't need to actually run it as a unit test

just a matter of semantics isn't it.
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9841869
That's right, just because you have the facilities availble to do "testing", doesn't mean you need to use them.  The application should just run via a normal "main" method.  Just don't use the "assert..." methods ;-)
0
 
LVL 1

Author Comment

by:travd
ID: 9848504
Thanks for your help, I'll go ahead and use http unit. By concern over using the unit test framework was that I didn't want to invoke it using junit and have to see the graphical fail/pass GUI or anything like that - I just want this to be a straight java app.

Thanks.
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9848680
;-)
0

Featured Post

PeopleSoft Has Never Been Easier

PeopleSoft Adoption Made Smooth & Simple!

On-The-Job Training Is made Intuitive & Easy With WalkMe's On-Screen Guidance Tool.  Claim Your Free WalkMe Account Now

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

An old method to applying the Singleton pattern in your Java code is to check if a static instance, defined in the same class that needs to be instantiated once and only once, is null and then create a new instance; otherwise, the pre-existing insta…
Java Flight Recorder and Java Mission Control together create a complete tool chain to continuously collect low level and detailed runtime information enabling after-the-fact incident analysis. Java Flight Recorder is a profiling and event collectio…
Viewers learn about the “for” loop and how it works in Java. By comparing it to the while loop learned before, viewers can make the transition easily. You will learn about the formatting of the for loop as we write a program that prints even numbers…
The viewer will learn how to implement Singleton Design Pattern in Java.

695 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question