Solved

PHP Crawler Sourceforge

Posted on 2010-09-09
4
1,002 Views
Last Modified: 2012-06-21
Hello,

I am attempting to implement an open source PHP application available from source Forge (http://sourceforge.net/projects/php-crawler).

Unfortunately there is not much documentation however I have managed to get the crawler working to a certain extent-I can target a web page and return all of the content and output to a new html file.

However the MYSQL database table used by the crawler is empty at the end of processing.  I do not need the whole HTML file-I just want one <DIV> section. I thought I could do this using the database contents however I am open to other suggestions.

I have tried using a string function but this returns a blank file so either it is incorrectly written or I cannot use the string function on the HTML file.

I have attached a copy of pro.php which successfully returns the whole page and pro1.php (returns nothing).  I have also attached a copy of index2.php which calls pro.php and pro1.php with crawl address.

The difference between files (where I have attempted to strip out the div) is displayed below-
$data = $usendid;
$string = between('<div id=section c1>', '</div>', $data);

function between($start, $end, $source) {
        $s = strpos($source, $start) + strlen($start);
        return substr($source, $s, strpos($source, $end, $s) - $s);

Can anyone advise me either how I can fix my string function to pull out the required section or how I could use the database table to complete the same thing?  does anyone have any more detailed documentation for PHP crawler?

Thanks
index2.php
pro.php
pro1.php
0
Comment
Question by:javaftper
  • 2
4 Comments
 
LVL 4

Accepted Solution

by:
mpickreign earned 250 total points
ID: 33637022
I would use a preg_match instead of the between function. This should do it...

preg_match('/<div id=section c1>(.+)<\/div>/',$data,$matched_content);
$matched_content[1]     <--- This will be just the content between the div tags.
0
 
LVL 4

Author Comment

by:javaftper
ID: 33648005
thanks.  preg_match works better however i'm having to take and re-create the whole file rather than creating the div in one pass possibly using the MYSQL DB.
Anyone got any docs for php crawler?
0
 
LVL 1

Assisted Solution

by:ahmad_alinat
ahmad_alinat earned 250 total points
ID: 33657465
Instead of using this open source project

you can use the php dom document + xpath interface

suppose you want to get DIV where id = sectionc1

first of all, you must load the page html with DOMDocument

$dom = new DOMDocument();
@$dom->loadHTMLFile(html url);

then use dom xpath to extract the DIV

$xpath = new DOMXPath($dom);
$results  = $xpath->query("//div[@id='sectionc1']");

simple and easy!
0
 
LVL 4

Author Closing Comment

by:javaftper
ID: 33685246
both comments very helpful.
0

Featured Post

Free learning courses: Active Directory Deep Dive

Get a firm grasp on your IT environment when you learn Active Directory best practices with Veeam! Watch all, or choose any amount, of this three-part webinar series to improve your skills. From the basics to virtualization and backup, we got you covered.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Foreword (July, 2015) Since I first wrote this article, years ago, a great many more people have begun using the internet.  They are coming online from every part of the globe, learning, reading, shopping and spending money at an ever-increasing ra…
This article discusses four methods for overlaying images in a container on a web page
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

860 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question