Fetch a .jsp generated page using LWP & Perl

Posted on 2003-03-03
Medium Priority
Last Modified: 2006-11-17
I would like to fetch a web page using the  LWP module, and have the results parsed and then
email users based on the contents of the page.  I have a perl scrip tthat I can modify  to do that.
My problem is,  the website I need the information from generates the page using  javascript,
 and the results appear on the browser  with the extension of .jsp (example: : landregister_new.jsp) ,
which is difficult  for me to even print the screen on, using my netscape 4.6,  let alone return something
 suitable  for parsing.

Can anyone provide  a perl script or some java code or whatever,  to convert the .jsp to html (preferably)
 or ascii, or .txt or  something that I can handle with a perl script ?
Question by:caciqueman
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
  • 2
LVL 20

Expert Comment

ID: 8061860
A suffix of .jsp means that Java Server Pages is being used to generate the page, but what arrives in response to the request should still be HTML, possibly with embedded Javascript. Unless some fancy games are being played, the data you want should probably be there in the HTML. Perhaps if you could provide a sample or example of what's causing you a problem?

Do you have 'lynx' available? It's a text-only browser. Can it render the data you want on these pages? If so, another approach you can take is to dump the page with lynx and use ordinary regular expressions and such to take the page apart and extract the data you're interested in.

Author Comment

ID: 8066045
I don't have lynx, and I am fetching this for use on a unix server, and then parsing
the html for certain phrases.  I think the problem is arising because the html page
that calls the .jsp page is fine, but from the .jsp page you need to make a further
selection to proceed....so basically that's why I wanted a script to be able to pass
from the .jsp page one to page two, WITH the completed choice from page one. I
know the data that I need passed, so thats not a problem.
LVL 20

Accepted Solution

jmcg earned 300 total points
ID: 8067447
Lynx is available for free for UNIX, so you _could_ have lynx.

Aside from that, you need to examine how the .jsp page is passing the choice made on to the next step. Is it a <form>? A list of different URL links? It should be apparent from the source of the page how it's being done -- your LWP script simply needs to simulate that step.

Author Comment

ID: 8071781
I won't go the lynx route but thank you for your information.

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
In the distant past (last year) I hacked together a little toy that would allow a couple of Manager types to query, preview, and extract data from a number of MongoDB instances, to their tool of choice: Excel (http://dilbert.com/strips/comic/2007-08…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Six Sigma Control Plans

764 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question