Solved

Parse & Download HTML pages

Posted on 2002-06-19
6
248 Views
Last Modified: 2010-03-05
I want to write a Perl script that does two things:

1. Parse a set of HTML pages in different directories recursively and extract all <A HREF=.....> </A> tags into separate text files while maintaining the directory structure. The text files have to reside in the same directories as the html pages. (Note that tags/links are to html pages).

2. Use the extracted tags from text files and download the html pages and save them to the respective directories.

I have managed to extract the tags but into a single file. The directory structure needs to be maintained. I would enter the home directory and the script should do the extraction and download recursively.

Appreciate your help.
0
Comment
Question by:ankhan100599
  • 3
  • 2
6 Comments
 
LVL 1

Expert Comment

by:smisk
ID: 7093601
this may be off topic, but have you tried using the tool wget?
0
 

Author Comment

by:ankhan100599
ID: 7095095
I dont think wget can work recursively through a directory structure, downloading html pages against links in the text files. If it can please let me know how.

Remember I need to fire this off from a home directory, all downloaded html pages will be placed in the same folder as the text file containing the links.

0
 
LVL 1

Expert Comment

by:smisk
ID: 7095772
Ok, I thought you just wanted a simple spider...
0
Best Practices: Disaster Recovery Testing

Besides backup, any IT division should have a disaster recovery plan. You will find a few tips below relating to the development of such a plan and to what issues one should pay special attention in the course of backup planning.

 
LVL 8

Expert Comment

by:jhurst
ID: 7097238
wget -r http://www.yahoo.com

does a recursive, as defined by the -r, get of the entire website and preserves the directory structures completely.  
0
 

Author Comment

by:ankhan100599
ID: 7102984
I have managed to extract the links. Can someone tell me how to extract the actual links (portion between quotes) in the following examples :

<A HREF="htmlsrpl.html#bascomlopt">Basic command-line options:</A>
<A HREF="htmlsrpl.html#old">old="..."</A>
<A HREF="htmlsrpl.html#upcase">upcase=1</A>
<A HREF="htmlsrpl.html#new">new="..."</A>
<A HREF="htmlsrpl.html#old">old="..."</A>
<A HREF="htmlsrpl.html#intags">intags=1</A>
<A HREF="htmlqref.html#syntax">tags</A>
<A HREF="htmlsrpl.html#inclexcl">Element inclusion/exclusion command-line options:</A>
<A HREF="htmlsrpl.html#inside">inside=...</A>
<A HREF="htmlqref.html#syntax">elements</A>
<A HREF="htmlsrpl.html#outside">outside=...</A>
<A HREF="htmlsrpl.html#inmost">inmost=...</A>
<A HREF="htmlsrpl.html#inside">inside=</A>
<A HREF="htmlqref.html#inline">&lt;IMG&gt;</A>
<A HREF="htmlqref.html#br">&lt;BR&gt;</A>
<A HREF="htmlchek.html">htmlchek</A>

I need to pass these as a variable.

Thanks for the responses.
0
 
LVL 1

Accepted Solution

by:
smisk earned 100 total points
ID: 7103744
($link) = $line =~ /HREF="(.*?)"/i;
print "link : $link\n";
0

Featured Post

PRTG Network Monitor: Intuitive Network Monitoring

Network Monitoring is essential to ensure that computer systems and network devices are running. Use PRTG to monitor LANs, servers, websites, applications and devices, bandwidth, virtual environments, remote systems, IoT, and many more. PRTG is easy to set up & use.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
crawling gofundme 4 126
Perl Script to read file contents, make changes, and build new file 7 114
Edit Odbc.ini using perl module 2 83
PERL - Find newest folder 12 135
Many time we need to work with multiple files all together. If its windows system then we can use some GUI based editor to accomplish our task. But what if you are on putty or have only CLI(Command Line Interface) as an option to  edit your files. I…
There are many situations when we need to display the data in sorted order. For example: Student details by name or by rank or by total marks etc. If you are working on data driven based projects then you will use sorting techniques very frequently.…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
A short tutorial showing how to set up an email signature in Outlook on the Web (previously known as OWA). For free email signatures designs, visit https://www.mail-signatures.com/articles/signature-templates/?sts=6651 If you want to manage em…

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question