Solved

Retrieve data from external websites using perl or application

Posted on 2004-08-09
5
197 Views
Last Modified: 2013-12-25
I need to get data from a very long list of sites (over 8000) and write the resulting html code from each site to seperate text files.  They have similar urls and content.

I am looking for the best way to do this, be it with perl or with a third party application.  Any help would be greatly appreciated.
0
Comment
Question by:Igiwwa
5 Comments
 
LVL 48

Accepted Solution

by:
Tintin earned 84 total points
ID: 11757539
Let's make the following assumptions.

1.  The list of sites (URL's) is in a plain text file.
2.  The output text file will have sequential names (as you haven't specified what format)

then

#!/usr/bin/perl
use strict;
use LWP::Simple;
use File::Basename;

my $list = '/path/to/list/of/sites.txt';
my $outputdir = ' /path/to/outputdir';

open LIST, $list or die "Can not open $list $!\n";

while (<LIST>) {
  chomp;
  my $site=$_;
  my $file=$outputdir . basename($site);
  getstore($site,$file);
}


 
0
 
LVL 4

Assisted Solution

by:alikoank
alikoank earned 83 total points
ID: 11760255
there are already several applications doing this
take a look at XMLTV

http://membled.com/work/apps/xmltv/

or plucker

http://www.plkr.org/
0
 
LVL 51

Assisted Solution

by:ahoffmann
ahoffmann earned 83 total points
ID: 11771778
assuming your URLs in a file, one per line:

wget -i file-withURLs
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

The following is a collection of cases for strange behaviour when using advanced techniques in DOS batch files. You should have some basic experience in batch "programming", as I'm assuming some knowledge and not further explain the basics. For some…
Introduction This tutorial will give you a fast look what you can do with WhizBase. I expect you already know how to work with HTML at least, and that you understand the basics of the internet and how the internet works. WhizBase is a server-s…
Learn the basics of lists in Python. Lists, as their name suggests, are a means for ordering and storing values. : Lists are declared using brackets; for example: t = [1, 2, 3]: Lists may contain a mix of data types; for example: t = ['string', 1, T…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

863 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

22 Experts available now in Live!

Get 1:1 Help Now