Solved

PHP Crawler Sourceforge

Posted on 2010-09-09
4
993 Views
Last Modified: 2012-06-21
Hello,

I am attempting to implement an open source PHP application available from source Forge (http://sourceforge.net/projects/php-crawler).

Unfortunately there is not much documentation however I have managed to get the crawler working to a certain extent-I can target a web page and return all of the content and output to a new html file.

However the MYSQL database table used by the crawler is empty at the end of processing.  I do not need the whole HTML file-I just want one <DIV> section. I thought I could do this using the database contents however I am open to other suggestions.

I have tried using a string function but this returns a blank file so either it is incorrectly written or I cannot use the string function on the HTML file.

I have attached a copy of pro.php which successfully returns the whole page and pro1.php (returns nothing).  I have also attached a copy of index2.php which calls pro.php and pro1.php with crawl address.

The difference between files (where I have attempted to strip out the div) is displayed below-
$data = $usendid;
$string = between('<div id=section c1>', '</div>', $data);

function between($start, $end, $source) {
        $s = strpos($source, $start) + strlen($start);
        return substr($source, $s, strpos($source, $end, $s) - $s);

Can anyone advise me either how I can fix my string function to pull out the required section or how I could use the database table to complete the same thing?  does anyone have any more detailed documentation for PHP crawler?

Thanks
index2.php
pro.php
pro1.php
0
Comment
Question by:javaftper
  • 2
4 Comments
 
LVL 4

Accepted Solution

by:
mpickreign earned 250 total points
ID: 33637022
I would use a preg_match instead of the between function. This should do it...

preg_match('/<div id=section c1>(.+)<\/div>/',$data,$matched_content);
$matched_content[1]     <--- This will be just the content between the div tags.
0
 
LVL 4

Author Comment

by:javaftper
ID: 33648005
thanks.  preg_match works better however i'm having to take and re-create the whole file rather than creating the div in one pass possibly using the MYSQL DB.
Anyone got any docs for php crawler?
0
 
LVL 1

Assisted Solution

by:ahmad_alinat
ahmad_alinat earned 250 total points
ID: 33657465
Instead of using this open source project

you can use the php dom document + xpath interface

suppose you want to get DIV where id = sectionc1

first of all, you must load the page html with DOMDocument

$dom = new DOMDocument();
@$dom->loadHTMLFile(html url);

then use dom xpath to extract the DIV

$xpath = new DOMXPath($dom);
$results  = $xpath->query("//div[@id='sectionc1']");

simple and easy!
0
 
LVL 4

Author Closing Comment

by:javaftper
ID: 33685246
both comments very helpful.
0

Featured Post

Simplifying Server Workload Migrations

This use case outlines the migration challenges that organizations face and how the Acronis AnyData Engine supports physical-to-physical (P2P), physical-to-virtual (P2V), virtual to physical (V2P), and cross-virtual (V2V) migration scenarios to address these challenges.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Does the idea of dealing with bits scare or confuse you? Does it seem like a waste of time in an age where we all have terabytes of storage? If so, you're missing out on one of the core tools in every professional programmer's toolbox. Learn how to …
This article discusses how to create an extensible mechanism for linked drop downs.
The viewer will learn how to count occurrences of each item in an array.
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

813 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now