Solved

Parse & Download HTML pages

Posted on 2002-06-19
6
253 Views
Last Modified: 2010-03-05
I want to write a Perl script that does two things:

1. Parse a set of HTML pages in different directories recursively and extract all <A HREF=.....> </A> tags into separate text files while maintaining the directory structure. The text files have to reside in the same directories as the html pages. (Note that tags/links are to html pages).

2. Use the extracted tags from text files and download the html pages and save them to the respective directories.

I have managed to extract the tags but into a single file. The directory structure needs to be maintained. I would enter the home directory and the script should do the extraction and download recursively.

Appreciate your help.
0
Comment
Question by:ankhan100599
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 3
  • 2
6 Comments
 
LVL 1

Expert Comment

by:smisk
ID: 7093601
this may be off topic, but have you tried using the tool wget?
0
 

Author Comment

by:ankhan100599
ID: 7095095
I dont think wget can work recursively through a directory structure, downloading html pages against links in the text files. If it can please let me know how.

Remember I need to fire this off from a home directory, all downloaded html pages will be placed in the same folder as the text file containing the links.

0
 
LVL 1

Expert Comment

by:smisk
ID: 7095772
Ok, I thought you just wanted a simple spider...
0
Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 8

Expert Comment

by:jhurst
ID: 7097238
wget -r http://www.yahoo.com

does a recursive, as defined by the -r, get of the entire website and preserves the directory structures completely.  
0
 

Author Comment

by:ankhan100599
ID: 7102984
I have managed to extract the links. Can someone tell me how to extract the actual links (portion between quotes) in the following examples :

<A HREF="htmlsrpl.html#bascomlopt">Basic command-line options:</A>
<A HREF="htmlsrpl.html#old">old="..."</A>
<A HREF="htmlsrpl.html#upcase">upcase=1</A>
<A HREF="htmlsrpl.html#new">new="..."</A>
<A HREF="htmlsrpl.html#old">old="..."</A>
<A HREF="htmlsrpl.html#intags">intags=1</A>
<A HREF="htmlqref.html#syntax">tags</A>
<A HREF="htmlsrpl.html#inclexcl">Element inclusion/exclusion command-line options:</A>
<A HREF="htmlsrpl.html#inside">inside=...</A>
<A HREF="htmlqref.html#syntax">elements</A>
<A HREF="htmlsrpl.html#outside">outside=...</A>
<A HREF="htmlsrpl.html#inmost">inmost=...</A>
<A HREF="htmlsrpl.html#inside">inside=</A>
<A HREF="htmlqref.html#inline">&lt;IMG&gt;</A>
<A HREF="htmlqref.html#br">&lt;BR&gt;</A>
<A HREF="htmlchek.html">htmlchek</A>

I need to pass these as a variable.

Thanks for the responses.
0
 
LVL 1

Accepted Solution

by:
smisk earned 100 total points
ID: 7103744
($link) = $line =~ /HREF="(.*?)"/i;
print "link : $link\n";
0

Featured Post

Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

I've just discovered very important differences between Windows an Unix formats in Perl,at least 5.xx.. MOST IMPORTANT: Use Unix file format while saving Your script. otherwise it will have ^M s or smth likely weird in the EOL, Then DO NOT use m…
A year or so back I was asked to have a play with MongoDB; within half an hour I had downloaded (http://www.mongodb.org/downloads),  installed and started the daemon, and had a console window open. After an hour or two of playing at the command …
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
Six Sigma Control Plans

729 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question