Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

PHP Crawler Sourceforge

Posted on 2010-09-09
4
Medium Priority
?
1,020 Views
Last Modified: 2012-06-21
Hello,

I am attempting to implement an open source PHP application available from source Forge (http://sourceforge.net/projects/php-crawler).

Unfortunately there is not much documentation however I have managed to get the crawler working to a certain extent-I can target a web page and return all of the content and output to a new html file.

However the MYSQL database table used by the crawler is empty at the end of processing.  I do not need the whole HTML file-I just want one <DIV> section. I thought I could do this using the database contents however I am open to other suggestions.

I have tried using a string function but this returns a blank file so either it is incorrectly written or I cannot use the string function on the HTML file.

I have attached a copy of pro.php which successfully returns the whole page and pro1.php (returns nothing).  I have also attached a copy of index2.php which calls pro.php and pro1.php with crawl address.

The difference between files (where I have attempted to strip out the div) is displayed below-
$data = $usendid;
$string = between('<div id=section c1>', '</div>', $data);

function between($start, $end, $source) {
        $s = strpos($source, $start) + strlen($start);
        return substr($source, $s, strpos($source, $end, $s) - $s);

Can anyone advise me either how I can fix my string function to pull out the required section or how I could use the database table to complete the same thing?  does anyone have any more detailed documentation for PHP crawler?

Thanks
index2.php
pro.php
pro1.php
0
Comment
Question by:javaftper
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
4 Comments
 
LVL 4

Accepted Solution

by:
mpickreign earned 1000 total points
ID: 33637022
I would use a preg_match instead of the between function. This should do it...

preg_match('/<div id=section c1>(.+)<\/div>/',$data,$matched_content);
$matched_content[1]     <--- This will be just the content between the div tags.
0
 
LVL 4

Author Comment

by:javaftper
ID: 33648005
thanks.  preg_match works better however i'm having to take and re-create the whole file rather than creating the div in one pass possibly using the MYSQL DB.
Anyone got any docs for php crawler?
0
 
LVL 1

Assisted Solution

by:ahmad_alinat
ahmad_alinat earned 1000 total points
ID: 33657465
Instead of using this open source project

you can use the php dom document + xpath interface

suppose you want to get DIV where id = sectionc1

first of all, you must load the page html with DOMDocument

$dom = new DOMDocument();
@$dom->loadHTMLFile(html url);

then use dom xpath to extract the DIV

$xpath = new DOMXPath($dom);
$results  = $xpath->query("//div[@id='sectionc1']");

simple and easy!
0
 
LVL 4

Author Closing Comment

by:javaftper
ID: 33685246
both comments very helpful.
0

Featured Post

Moving data to the cloud? Find out if you’re ready

Before moving to the cloud, it is important to carefully define your db needs, plan for the migration & understand prod. environment. This wp explains how to define what you need from a cloud provider, plan for the migration & what putting a cloud solution into practice entails.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

This post contains step-by-step instructions for setting up alerting in Percona Monitoring and Management (PMM) using Grafana.
In this article, I’ll talk about multi-threaded slave statistics printed in MySQL error log file.
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
The viewer will learn how to dynamically set the form action using jQuery.

704 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question