August 29, 2008 10:15pm pdt

1. Adam314 558,667
2. ozo 370,137
3. FishMonger 115,555
4. Tintin 91,647
5. hielo 63,970
6. mjcoyne 58,453
7. nedfine 41,965
8. ravenpl 24,800
9. clockwatc... 24,260
10. teraplane 22,380
11. ahoffmann 21,299
12. kawas 19,050
13. oleber 18,450
14. omarfarid 17,068
15. suhasbha... 16,570
16. marchent 13,540
17. nixfreak 12,200
18. nognew 10,600
19. hernst42 10,080
20. agriesser 10,000
21. SmartIntel 9,760
22. sentner 9,648
23. ddrudik 9,100
24. shanikawm 7,996
25. evilrix 7,500

1. ozo 3,641,902
2. Adam314 1,481,086
3. Tintin 739,941
4. kandura 579,965
5. jmcg 517,810
6. mjcoyne 510,591
7. FishMonger 468,189
8. maneshr 385,976
9. teraplane 366,588
10. manav_m... 338,348
11. clockwatc... 240,548
12. rj2 211,481
13. ahoffmann 206,862
14. davorg 153,165
15. Tim_Utsc... 113,107
16. shlomoy 109,110
17. gripe 98,559
18. suhasbha... 91,970
19. Perl_Diver 82,106
20. sstouk 74,470
21. jhurst 72,319
22. rugdog 70,891
23. hielo 69,920
24. nedfine 69,532
25. ZiaTioN 66,995

06.30.2008 at 08:30AM PDT, ID: 23527220

	[x] Attachment Details

download pages according to a criteria

Asked by catalini in Perl Programming Language, Python Scripting Language, Scripting Languages

Tags: perl, python

This question is a follow up on this http://www.experts-exchange.com/Programming/Languages/Scripting/Perl/Q_23476422.html

I need to create a script that downloads a given page
1) http://www.domainname.net/search/

extracts from the page all the links that follow this pattern (where **** can be anything), and downloads them
2) http://www.domainname.net/***********/?sort=alphab

from this second round of pages extracts all the links with this second pattern
3) http://www.domainname.net/pages/**********/

and downloads them too.

The code below, provided by adam341 works perfectly (downloading http://www.domainname.net/search/?start=$start files and doing the incremental passages).

The problem is that when it saves the first set of files it changes all relative links to "file://...." so that the second

m|http://www.domainname.net/.*/\?sort=alphab|;

and

m|http://www.domainname.net/pages/.*/|;

do not find anything to use.

Is there any way to save the pages without changing the content and the relative links? (I've tried to download the same page with firefox and it has all the links correctly saved with http://www.domainname.net/).

Start Free Trial

         
           
             1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:

           
           
             use WWW::Mechanize;
 
my $mech = WWW::Mechanize->new();
 
my $count=1;
for my $start10 (0..900) {
    my $start = $start10 * 10;
    $mech->get("http://www.domainname.net/search/?start=$start");
    $mech->save_content(sprintf("file%04d.txt", $count++));
    
    foreach my $link1 ($mech->links) {
        next unless $link1->url =~ m|http://www.domainname.net/.*/\?sort=alphab|;
        $mech->get($link1->url);
        $mech->save_content(sprintf("file%04d.txt", $count++));
        foreach my $link2 ($mech->links) {
            next unless $link2 =~ m|http://www.domainname.net/pages/.*/|;
            $mech->get($link2->url);
            $mech->save_content(sprintf("file%04d.txt", $count++));
        }
    }
}
           
         

Keywords: download pages according to a criteria

	Title
1	download pages according to a pattern
2	Interesting Passage from the Bible:
3	Perl programming
4	Problem with searching for patterns in …
5	Sorting XML according to attributes in P…
6	Math Pattern Question

Loading Advertisement...

20080716-EE-VQP-32 / EE_QW_2_20070628

download pages according to a criteria

Asked by catalini in Perl Programming Language, Python Scripting Language, Scripting Languages

Tags: perl, python

About this solution