?
Solved

Java http client - automated web browsing

Posted on 2003-11-27
8
Medium Priority
?
360 Views
Last Modified: 2008-03-17
Hi,

I asked this question in the Web Languages section previosuly, but didn't get any replies.

I decided in any case that I would like to go with a Java implementation if possible, since I am familiar with this language.  Does anyone know of any existing libraries or classes that would make something like this easier?  In particular classes that allow establishment of an http connection, storing of cookies, etc.  I was thinking about using httpUnit.  Is this a good choice?

Original question:

I need to automate browsing of a particular site.  That is, I must be able to programatically download the pages associated with the site for parsing and analysis.  Additionally, I need to be able to fill out and submit forms in an automated way, as well as support cookies (the site might require the information in the cookie in order to provide context for certain pages).

There are probably several ways to crack this egg, so in the case where there are several arguably equal solutions, I would prefer the ones that leverage the languages or technologies that I am familar with: C++, Java, Sockets.  I have superficial familarity with html, javascript, VB etc.  This application needs to run on the Win32 platform.

Thanks in advance for your suggestions.
0
Comment
Question by:travd
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 3
  • 2
8 Comments
 
LVL 92

Expert Comment

by:objects
ID: 9833749
Yes httpunit seems like a good match
http://www.junit.org/news/extension/web/index.htm
0
 
LVL 1

Author Comment

by:travd
ID: 9833984
Looking at those links, it seems there are three good possibilites - jWebUnit, htmlUnit and httpUnit.   Does anyone have any experience with any of these?  Is it possible to use these without JUnit - I'm not really running a unit test, but instead trying to have automatic form submission, combined with harvesting and processing the returned data.
0
 
LVL 15

Accepted Solution

by:
jimmack earned 500 total points
ID: 9838623
I've used httpUnit to do the first part of what you are asking (automatically accessing a web site and then parsing the result to see if I've got the data I want).  It works fine for this.

The WebResponse object that you get back (in response to a WebRequest), allows you to identify Forms, Links, Tables etc. fairly easily.

It can also handle Cookies and you should be able to submit Form data back to the server as necessary.

httpUnit requires JUnit, since it is effectively an extension to it (but you should look into this too because it's very useful to automatically test your Java applications ;-))

I haven't used jWebUnit or htmlUnit.
0
Get real performance insights from real users

Key features:
- Total Pages Views and Load times
- Top Pages Viewed and Load Times
- Real Time Site Page Build Performance
- Users’ Browser and Platform Performance
- Geographic User Breakdown
- And more

 
LVL 1

Author Comment

by:travd
ID: 9840977
OK, so I need to have JUnit, but I just want to run this as a normal application - I don't need to actually run it as a unit test, do I?

Thanks,
T.
0
 
LVL 92

Expert Comment

by:objects
ID: 9840989
> I don't need to actually run it as a unit test

just a matter of semantics isn't it.
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9841869
That's right, just because you have the facilities availble to do "testing", doesn't mean you need to use them.  The application should just run via a normal "main" method.  Just don't use the "assert..." methods ;-)
0
 
LVL 1

Author Comment

by:travd
ID: 9848504
Thanks for your help, I'll go ahead and use http unit. By concern over using the unit test framework was that I didn't want to invoke it using junit and have to see the graphical fail/pass GUI or anything like that - I just want this to be a straight java app.

Thanks.
0
 
LVL 15

Expert Comment

by:jimmack
ID: 9848680
;-)
0

Featured Post

Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

INTRODUCTION Working with files is a moderately common task in Java.  For most projects hard coding the file names, using parameters in configuration files, or using command-line arguments is sufficient.   However, when your application has vi…
Introduction Java can be integrated with native programs using an interface called JNI(Java Native Interface). Native programs are programs which can directly run on the processor. JNI is simply a naming and calling convention so that the JVM (Java…
This theoretical tutorial explains exceptions, reasons for exceptions, different categories of exception and exception hierarchy.
How to fix incompatible JVM issue while installing Eclipse While installing Eclipse in windows, got one error like above and unable to proceed with the installation. This video describes how to successfully install Eclipse. How to solve incompa…
Suggested Courses
Course of the Month9 days, 21 hours left to enroll

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question