Solved

PHP FSOCK ERROR

Posted on 2004-08-28
16
746 Views
Last Modified: 2008-02-01
I am getting FSOCK Error - for some reason - Please kindly help me in solving this error.


===
<?php


$path="";

$maxlevel=1;
$pagecount=0;
$pagelevel=0;
$imagecount=0;
$varfiles=array("links.txt","writtenlinks.txt","images.txt","lastredirects.txt","pagelevel.txt","imagecount.txt","pagecount.txt");

function logerror($var) { }

require $path."headerandfooter2.inc";
require $path."vars.inc";
require $path."goback.inc";
require $path."webtools.inc";
require $path."pulltools.inc";
require $path."tools.inc";
require $path."filetools.inc";
require $path."db.inc";
require $path."changegoogle.inc";
require $path."addextrapages.inc";
require $path."stripbody.inc";
require $path."tags.inc";

function pullimages() {
   global $html,$pointer,$nextval,$newdomain,$imagesdir,$imagecount,$ext,$currenturl,$path,$images,$division;
   $pointer=0;
   while (movefwdpast("<img")) {
      testdouble();
      $skip=false;
      $imagecount++;
      $si=$pointer;
      $skip=!movefwdpast(">");
      if (!$skip) {
         $ei=$pointer-1;
         $pointer=$si;
         $skip=!getnextval("src",$ei);
      }
      if (!$skip) {
         $skip=!getext($nextval);
         if ($skip) replacehtml("",strlen($nextval));
      }
      if (!$skip) {
         $imgtrunk="$imagesdir/$division/$imagecount.$ext";
         $newimgurl="http://$newdomain/$imgtrunk";
         $imgurl=makeurl($nextval);
         if (isset($images[$imgurl])) {
            replacehtml("http://$newdomain/$imagesdir/$division/".$images[$imgurl].".$ext",strlen($nextval));
            $skip=true;
         }
      }
      if (!$skip) {
         if (substr($imgurl,0,7)=="http://") {
            $img=readwebpage($imgurl,$currenturl);
            if (stristr($img,"not found")) $img="";
         } else $img="";
         if (!$img) {
            if ($imgurl!=$nextval) replacehtml($imgurl,strlen($nextval));
            $skip=true;
         }
      }
      if (!$skip) {
         wrt($path.$imgtrunk,$img);
         replacehtml($newimgurl,strlen($nextval));
         $images[$imgurl]=$imagecount;
      } else $imagecount--;
   }
}

function getfromkeys($url) {
   global $links,$writtenlinks;
   $newlink=$url;
   $si=stripos($url,"http://",7);
   if (!($si===false)) {
      $shorturl=substr($url,$si);
      if (is_array($links)) {
         reset($links);
         foreach($links as $linkurl=>$val) {
            if (stristr($linkurl,$shorturl)) $newlink=$linkurl;
         }
      }
      if ($newlink==$url && is_array($writtenlinks)) {
         reset($writtenlinks);
         foreach($writtenlinks as $linkurl=>$val) {
            if (stristr($linkurl,$shorturl)) $newlink=$linkurl;
         }
      }
   }
   return $newlink;
}

function getlinks() {
   global $html,$pointer,$nextval,$newdomain,$pagesdir,$pagecount,$currenturl,$path,$links,$pagelevel,$writtenlinks,$maxlevel,$division,$lastredirects,$maxlinks,$maxdownload,$existingpages,$tablename,$sql_result;
   $pointer=0;
   while (movefwdpast("<a")) {
      $skip=false;
      $pagecount++;
      if ($pagecount>$maxlinks) { logerror("break in getlinks, currenturl:$currenturl pagelevel:$pagelevel maxlevel:$maxlevel pointer:$pointer division:$division pagecount:$pagecount"); break; }
      $si=$pointer;
      $skip=!movefwdpast(">");
      if (!$skip) {
         $ei=$pointer-1;
         $pointer=$si;
         $skip=!getnextval("href",$ei);
      }
      if (!$skip) {
         $link=makeurl($nextval);
         $newurl=""; $usedivision=$division;
         if (substr($link,0,7)!="http://") $newurl=$link;
         else {
            if (stristr($link,"url?ntc=")) $link=getfromkeys($link);
            if ($existingpages[$link]) {
               $pagecount--;
               $newurl=$existingpages[$link];
            } elseif (isset($lastredirects[$link])) {
               $usepage=$lastredirects[$link];
               $pagecount--;
            } elseif (isset($links[$link])) {
               $usepage=$links[$link];
               $pagecount--;
            } elseif (isset($writtenlinks[$link])) {
               $usepage=$writtenlinks[$link];
               $pagecount--;
            } elseif ($pagelevel<$maxlevel) {
               # check in db for url and lasturl
               $sql="SELECT pagenum AS usepage,division AS usedivision FROM $tablename WHERE url=\"$link\" OR lasturl=\"$link\" LIMIT 1";
               send_query($sql);
               if ($res=mysql_fetch_assoc($sql_result)) {
                  extract($res);
                  $pagecount--;
               } else {
                  $usepage=$pagecount;
                  if (sizeof($links)>$maxdownload) { logerror("break in getlinks, currenturl:$currenturl pagelevel:$pagelevel maxlevel:$maxlevel pointer:$pointer division:$division pagecount:$pagecount sizeof_links:".sizeof($links)); break; }
                  $links[$link]=$usepage;
               }
            } else {
               $pagecount--;
               $newurl=$link;
            }
            if (!$newurl) $newurl="http://$newdomain/$pagesdir/$usedivision/$usepage.html";
         }
         if ($newurl!=$nextval) replacehtml($newurl,strlen($nextval));
         if (isheadline()) if (getlinktext()) storelinktext();
      } else $pagecount--;
   }
}

function isheadline($maxpagelevel=0) {
   global $nextval,$pagelevel;
   if ($pagelevel>$maxpagelevel) return false; # || http://news.google.com/url?ntc=
   if (($si=strpos($nextval,"url?ntc="))===false) return false;
   if (($ei=strpos($nextval,"&",$si))===false) return false;
   $code=substr($nextval,$si+8,$ei-$si-8);
   if (strlen($code)==5 && $code[4]!="0" && $code[4]!="1" && $code[4]!="2") return false;
   return true;
}

function storelinktext() {
   global $linktext,$tablename,$pagecount,$division,$pagelevel,$maxlevel,$sql_result;
   if ($pagelevel<$maxlevel) {
      $headlinevar="headline=\"".str_replace("\"","\\\"",$linktext)."\"";
      db_pconnect();
      $sql="SELECT pagenum FROM $tablename WHERE $headlinevar LIMIT 1";
      send_query($sql);
      if ($res=mysql_fetch_assoc($sql_result)) $sql="UPDATE $tablename SET pagenum=\"$pagecount\",division=\"$division\",body=\"\",added=CURRENT_TIMESTAMP WHERE $headlinevar LIMIT 1";
      else $sql="INSERT INTO $tablename SET $headlinevar,pagenum=\"$pagecount\",division=\"$division\",added=CURRENT_TIMESTAMP";
      send_query($sql);
   }
}

function updatetags($tagstart,$tagar) {
   global $html,$pointer,$nextval,$newdomain,$pagesdir,$currenturl,$path,$links,$pagelevel,$writtenlinks,$maxlevel,$division;
   extract($tagar);
   if (!$tagurl || !$tagend) goback("missing tagurl:$tagurl or tagend:$tagend tagstart:$tagstart");
   $pointer=0;
   while (movefwdpast($tagstart)) {
      $si=$pointer;
      if (!movefwdpast($tagend)) continue;
      $ei=$pointer-1;
      if ($requiredtag) {
         $pointer=$si;
         if (!getnextval($requiredtag,$ei)) continue;
         if ($requiredtagvalue && $requiredtagvalue!=$nextval) { $pointer=$si; continue; }
      }
      $pointer=$si;
      if (!getnextval($tagurl,$ei)) continue;
      if ($process_nextval) { if (!$process_nextval()) continue; }
      $link=makeurl($nextval);
      $newurl="";
      if (substr($link,0,7)!="http://") $newurl=$link;
      else {
         if (isset($links[$link])) $usepage=$links[$link];
         elseif (isset($writtenlinks[$link])) $usepage=$writtenlinks[$link];
         else $newurl=$link;
         if (!$newurl) $newurl="http://$newdomain/$pagesdir/$division/$usepage.html";
      }
      if ($newurl!=$nextval) replacehtml($newurl,strlen($nextval));
   }
}

function metaurl() { # process_nextval
   global $pointer,$nextval;
   # used for meta http-equiv refresh 0;url=http://www.
   $si2=strpos($nextval,"=");
   if ($si2===false) return false;
   $pointer=$si2+1;
   $nextval=substr($nextval,$pointer);
   $origsize=strlen($nextval);
   $nextval=trim(str_replace(array("\"","'"),"",$nextval));
   $nextval=str_pad($nextval,$origsize-strlen($nextval));
   return true;
}

function setlinks($type) {
   global $html,$pointer,$nextval;
   $pointer=0;
   while (getnextval($type,strlen($html))) {
      $link=makeurl($nextval);
      if ($link!=$nextval) replacehtml($link,strlen($nextval));
   }
}

function writepage() {
   global $path,$pagesdir,$currentpage,$currenturl,$html,$writtenlinks,$links,$origcurrenturl,$images,$pagelevel,$imagecount,$pagecount,$division,$lastredirects;
   testdouble();
   $stopfile=$path."newspider.txt";
   wrt($stopfile,"1");
   $filename=$path."$pagesdir/$division/$currentpage.html";
   wrt($filename,$html);
   $writtenlinks[$origcurrenturl]=$currentpage;
   $links=delete_key($origcurrenturl,$links);
   addtodb();
   erasefiles();
   if (is_array($links)) {
      wrtarray($path."links.txt",$links);
      wrtarray($path."writtenlinks.txt",$writtenlinks);
      wrtarray($path."images.txt",$images);
      wrtarray($path."lastredirects.txt",$lastredirects);
      wrtvar($path."pagelevel.txt",$pagelevel);
      wrtvar($path."imagecount.txt",$imagecount);
      wrtvar($path."pagecount.txt",$pagecount);
   }
   unlink($stopfile);
}

function addtodb() {
   global $currentpage,$tablename,$division,$sql_result,$html,$maxlevel,$nextval,$origcurrenturl,$basedomain,$domainstripto,$domainstripfrom,$currenturl;
   $nextval=$origcurrenturl;
   $shouldstore=(stristr($origcurrenturl,"url?ntc="));
   if ($currenturl!=$origcurrenturl) $addlasturl=",lasturl=\"$currenturl\""; else $addlasturl="";
   if ($currentpage!=0 && isheadline($maxlevel) && $shouldstore) {
      db_pconnect();
      $sql="SELECT headline FROM $tablename WHERE pagenum=\"$currentpage\" AND division=\"$division\" LIMIT 1";
      send_query($sql);
      if ($res=mysql_fetch_assoc($sql_result)) {
         extract($res);
         $text=$html;
#         if (($si=stripos($text,$headline))!==false) $text=substr($text,$si+strlen($headline));
         $text=altremovechunk($text,"<script","/script>");
         $text=stripto($text,array("<body",">"));
         $text=stripfrom($text,"</body");
         $text=stripto($text,$domainstripto[$basedomain]);
         $text=stripfrom($text,$domainstripfrom[$basedomain]);
         $text=str_ireplace("<br>"," ",$text);
         $text=str_ireplace("<br />"," ",$text);
         $text=str_replace(array("\t","\n","\r","  "),array(" "," "," "," "),$text);
         $text=trim(str_replace("\"","\\\"",strip_tags($text)));
         $sql="UPDATE $tablename SET body=\"$text\",base=\"$basedomain\",added=CURRENT_TIMESTAMP,url=\"$origcurrenturl\"$addlasturl WHERE pagenum=\"$currentpage\" AND division=\"$division\"";
         send_query($sql);
      } else {
         $sql="INSERT INTO $tablename SET base=\"$basedomain\",added=CURRENT_TIMESTAMP,url=\"$origcurrenturl\"$addlasturl,pagenum=\"$currentpage\",division=\"$division\"";
         send_query($sql);
      }
   } elseif ($currentpage!=0 && $shouldstore) {
      db_pconnect();
      $sql="INSERT INTO $tablename SET base=\"$basedomain\",added=CURRENT_TIMESTAMP,url=\"$origcurrenturl\"$addlasturl,pagenum=\"$currentpage\",division=\"$division\"";
      send_query($sql);
   }
}

function stripto($text,$ar) {
   if (!$ar) return $text;
   if (!is_array($ar)) $ar=array($ar);
   foreach ($ar AS $stripval) {
      if (($ei=stripos($text,$stripval))!==false) $text=substr($text,$ei+strlen($stripval));
   }
   return $text;
}

function stripfrom($text,$ar) {
   if (!$ar) return $text;
   if (!is_array($ar)) $ar=array($ar);
   foreach ($ar AS $stripval) {
      if (($si=stripos($text,$stripval))!==false) $text=substr($text,0,$si);
   }
   return $text;
}

function erasefiles() {
   global $path,$varfiles;
   reset($varfiles);
   foreach($varfiles AS $filename) {
      if (is_file($path.$filename)) unlink($path.$filename);
   }
}

function setcurrenturl() {
   global $currenturl,$origcurrenturl,$lastredirecturl;
   $origcurrenturl=$currenturl;
   if ($lastredirecturl) {
      if (substr($lastredirecturl,0,7)!="http://" && substr($lastredirecturl,0,8)!="https://") {
         if (($si=stripos($lastredirecturl,"http"))!==false) {
            $currenturl=substr($lastredirecturl,$si);
            setbaseurl();
            $lastredirecturl=makeurl($origcurrenturl);
         } else $lastredirecturl="";
      }
      $currenturl=$lastredirecturl;
   }
}

function setbaseurl() {
   global $currenturl,$baseurl,$basedomain;
   if (substr($currenturl,0,7)=="http://") $start=7;
   elseif (substr($currenturl,0,8)=="https://") $start=8;
   else goback("http:// and https:// not found in currenturl:$currenturl");
   $si=strpos($currenturl,"?",$start);
   if ($si===false) $baseurl=$currenturl;
   else $baseurl=substr($currenturl,0,$si);
   $si=strpos($baseurl,"/",$start);
   if ($si===false) $basedomain=$baseurl;
   else $basedomain=substr($baseurl,0,$si);
   $si=strrpos($baseurl,"/");
   if (!($si===false)) $baseurl=substr($baseurl,0,$si);
}

function preparehtml() {
   global $html,$pointer,$basedomain;
   if (stristr($basedomain,"google.com")) {
      $remove=array(
         " onclick='"=>"'",
         " onclick=\""=>"\"",
"&nbsp;- <a href=\""=>"&nbsp;related&nbsp;&raquo;</b></a>",
" - <a href=\""=>"&nbsp;related&nbsp;&raquo;</a>",
"<table border=0 cellpadding=0 cellspacing=6><tr><td colspan=2 bgcolor=#cccccc><img alt=\"\" width=1 height=1></td></tr><tr><td colspan=2 nowrap><b><font size=-1>In the News</font></b></td></tr>"=>"</td></tr></table>",
      );
      $changevar=true;
   } else $remove=array(
"<live-content>"=>"</live-content>",
);
   reset($remove);
   foreach($remove as $start=>$end) {
      $pointer=0;
      while (removechunk($start,$end)) $pointer=0;
   }
   if ($changevar) changegoogle();
}

function existingpage() {
   global $lastredirecturl,$links,$writtenlinks,$lastredirects,$currentpage,$goto,$html;
   $goto=-1;
   if ($lastredirecturl) {
      if (isset($lastredirects[$lastredirecturl]) && $lastredirects[$lastredirecturl]!=$currentpage) $goto=$lastredirects[$lastredirecturl];
      elseif (isset($links[$lastredirecturl]) && $links[$lastredirecturl]!=$currentpage) $goto=$links[$lastredirecturl];
      elseif (isset($writtenlinks[$lastredirecturl]) && $writtenlinks[$lastredirecturl]!=$currentpage) $goto=$writtenlinks[$lastredirecturl];
      else $lastredirects[$lastredirecturl]=$currentpage;
      if ($goto!=-1) $html=gotohtml($goto);
   }
   return ($goto!=-1);
}

function dopage() {
   global $pagelevel,$maxlevel,$html,$currenturl,$retaincookies,$currentpage,$path,$links,$wrotefirstlinks,$tags,$division;
   testdouble();
   if (!$wrotefirstlinks && is_array($links)) {
      wrtarray($path."firstlinks.txt",$links);
      $wrotefirstlinks=true;
   }
   $retaincookies=false;
   print "reading $currentpage:$currenturl<br>";
   $html=trim(readwebpage($currenturl));
   $retaincookies=true;
   setcurrenturl();
   if (!existingpage()) {
      setbaseurl();
      if (strstr($html,"<blockquote><H1>Not Found</H1>")) $html="";
      if (!$html) {
         $html=gotourlhtml($currenturl);
         append($path."emptypages.txt",date("m d, Y H:i:s")." - /$division/$currentpage.html: $currenturl\n");
      } else {
         preparehtml();
         pullimages();
         getlinks();
         reset($tags);
         foreach($tags AS $tagstart=>$tagar) updatetags($tagstart,$tagar);
         finaledit();
      }
   }
   writepage();
}

function finaledit() {
   global $html,$newdomain;
   $newdomaintail=substr($newdomain,strpos($newdomain,".")+1);
   $replace=array(
   "http://www.google.com"=>"http://$newdomain",
   "google.com"=>"$newdomaintail",
   );
   $html=str_replace(array_keys($replace),array_values($replace),$html);
}

function logtime() {
   global $pagelevel,$path,$division;
   append($path."updatelog.txt","$division - $pagelevel - ".date("m d, Y H:i:s")."\n");
}

function checkfrequency() {
   global $path,$frequency;
   if (is_file($path."starttime.txt")) {
      include $path."starttime.txt";
      if (time()-$starttime<$frequency*60) {
         print "Less than $frequency minutes since last update. Time:".date("m, d Y H:i:s",time())." Starttime:".date("m, d Y H:i:s",$starttime);
         exit;
      }
   }
}

function copydown0() {
   global $path,$pagesdir,$division,$newdomain;
   copy($path."$pagesdir/$division/0.html",$path."$pagesdir/0.html");
   wrt($path."$pagesdir/$division/0.html",gotohtml(0));
}

function gotohtml($goto) {
   global $newdomain,$pagesdir,$division;
   $url="http://$newdomain/$pagesdir/";
   if ($goto!=0) $url.="$division/";
   $url.=$goto.".html";
   $htmlval=gotourlhtml($url);
   return $htmlval;
}

function gotourlhtml($url) {
   $htmlvar="<html>\n<head>\n<META HTTP-EQUIV=\"Refresh\" CONTENT=\"0;URL=$url\">\n</head>\n<body></body>\n</html>\n";
   return $htmlvar;
}

function delete_division() {
   global $tablename,$division;
   db_pconnect();
   $sql="DELETE FROM $tablename WHERE division=\"$division\"";
   send_query($sql);
}

$arefiles=true;
reset($varfiles);
foreach($varfiles AS $filename) {
   if (!is_file($path.$filename)) $arefiles=false;
}
if ($arefiles) {
   stopdouble(15);
   if (is_file($path."division.txt")) include $path."division.txt";
   else { erasefiles(); goback("no division.txt file"); }
   reset($varfiles);
   foreach($varfiles AS $filename) include $path.$filename;
   if (is_file($path."firstlinks.txt")) {
      include $path."firstlinks.txt";
      if (implode("|",$firstlinks)==implode("|",$links)) {
         reset($links);
         $res=each($links);
         $currenturl=$res["key"]; $currentpage=$res["value"];
         setcurrenturl();
         if (strstr($currenturl,"url?ntc=")) if (($si=stripos($currenturl,"http://"))!==false) $currenturl=substr($currenturl,$si);
         setbaseurl();
         $html=gotourlhtml($currenturl);
         writepage();
         append($path."skippedlinks.txt",date("m d, Y H:i:s")." - /$division/$currentpage.html: $currenturl\n");
      }
   }
   $pagelevel--;
#   db_pconnect();
} else {
   checkfrequency();
   stopdouble(15);
   include $path."emailalerts.inc";
   emailalerts();
   logtime();
   erasefiles();
   wrtvar($path."starttime.txt",time());
   if (is_file($path."division.txt")) include $path."division.txt";
   else $division=0;
   $division++;
   if ($division>$divisions) $division=1;
   delete_division();
   $filepaths=array($path."$pagesdir/$division",$path."$imagesdir/$division");
   foreach($filepaths AS $filepath) {
      if (is_dir($filepath)) eraseall($filepath);
      elseif (!makesubdir($filepath)) goback("cannot create $filepath");
   }
   $currentpage=0;
   $currenturl="http://$pulldomain";
   $links[$currenturl]=$currentpage;
#   db_pconnect();
   dopage();
   addextrapages();
   wrtvar($path."division.txt",$division);
   logtime();
}
while ($pagelevel<$maxlevel) {
   $dolinks=$links;
   $pagelevel++;
   if ($pagelevel==0) $pagelevel++;
   reset($dolinks);
   foreach($dolinks AS $currenturl=>$currentpage) dopage();
   logtime();
}
copydown0();
erasefiles();
if (is_file($path."firstlinks.txt")) unlink($path."firstlinks.txt");
?>
=====

Please kindly help me in sort out the problem , I will really appreciate your earliest response in this matter.

Thankin you in advance.
0
Comment
Question by:wcws
  • 8
  • 7
16 Comments
 
LVL 10

Expert Comment

by:frugle
ID: 11921485
Rather than posting reams and reams of irrelevant functions, could you post the exact error and the function containing the problem line?

Cheers

Mike
0
 

Author Comment

by:wcws
ID: 11921919
checking for spiders...
reading 0:http://news.google.com

Warning: fsockopen(): php_network_getaddresses: getaddrinfo failed: Name or service not known in /home/www/Murfmyle/news/snoopy.inc on line 1092

Warning: fsockopen(): unable to connect to images:80 in /home/www/Murfmyle/news/snoopy.inc on line 1092
0
 

Author Comment

by:wcws
ID: 11921924
snoopy.inc is as follows :-
===
<?php

/*************************************************

Snoopy - the PHP net client
Author: Monte Ohrt <monte@ispi.net>
Copyright (c): 1999-2000 ispi, all rights reserved
Version: 1.01

 * This library is free software; you can redistribute it and/or
 * modify it under the terms of the GNU Lesser General Public
 * License as published by the Free Software Foundation; either
 * version 2.1 of the License, or (at your option) any later version.
 *
 * This library is distributed in the hope that it will be useful,
 * but WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 * Lesser General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public
 * License along with this library; if not, write to the Free Software
 * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

You may contact the author of Snoopy by e-mail at:
monte@ispi.net

Or, write to:
Monte Ohrt
CTO, ispi
237 S. 70th suite 220
Lincoln, NE 68510

The latest version of Snoopy can be obtained from:
http://snoopy.sourceforge.com

*************************************************/

class Snoopy
{
      /**** Public variables ****/
      
      /* user definable vars */

      var $host                  =      "www.php.net";            // host name we are connecting to
      var $port                  =      80;                              // port we are connecting to
      var $proxy_host            =      "";                              // proxy host to use
      var $proxy_port            =      "";                              // proxy port to use
      var $agent                  =      "Snoopy v1.01";            // agent we masquerade as
      var      $referer            =      "";                              // referer info to pass
      var $cookies            =      array();                  // array of cookies to pass
                                                                        // $cookies["username"]="joe";
      var      $rawheaders            =      array();                  // array of raw headers to send
                                                                        // $rawheaders["Content-type"]="text/html";

      var $maxredirs            =      5;                              // http redirection depth maximum. 0 = disallow
      var $lastredirectaddr      =      "";                        // contains address of last redirected address
      var      $offsiteok            =      true;                        // allows redirection off-site
      var $maxframes            =      0;                              // frame content depth maximum. 0 = disallow
      var $expandlinks      =      true;                        // expand links to fully qualified URLs.
                                                                        // this only applies to fetchlinks()
                                                                        // or submitlinks()
      var $passcookies      =      true;                        // pass set cookies back through redirects
                                                                        // NOTE: this currently does not respect
                                                                        // dates, domains or paths.
      
      var      $user                  =      "";                              // user for http authentication
      var      $pass                  =      "";                              // password for http authentication
      
      // http accept types
      var $accept                  =      "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */*";
      
      var $results            =      "";                              // where the content is put
            
      var $error                  =      "";                              // error messages sent here
      var      $response_code      =      "";                              // response code returned from server
      var      $headers            =      array();                  // headers returned from server sent here
      var      $maxlength            =      500000;                        // max return data length (body)
      var $read_timeout      =      0;                              // timeout on read operations, in seconds
                                                                        // supported only since PHP 4 Beta 4
                                                                        // set to 0 to disallow timeouts
      var $timed_out            =      false;                        // if a read operation timed out
      var      $status                  =      0;                              // http request status
      
      var      $curl_path            =      "/usr/local/bin/curl";
                                                                        // Snoopy will use cURL for fetching
                                                                        // SSL content if a full system path to
                                                                        // the cURL binary is supplied here.
                                                                        // set to false if you do not have
                                                                        // cURL installed. See http://curl.haxx.se
                                                                        // for details on installing cURL.
                                                                        // Snoopy does *not* use the cURL
                                                                        // library functions built into php,
                                                                        // as these functions are not stable
                                                                        // as of this Snoopy release.
      
      /**** Private variables ****/      
      
      var      $_maxlinelen      =      4096;                        // max line length (headers)
      
      var $_httpmethod      =      "GET";                        // default http request method
      var $_httpversion      =      "HTTP/1.0";                  // default http request version
      var $_submit_method      =      "POST";                        // default submit method
      var $_submit_type      =      "application/x-www-form-urlencoded";      // default submit type
      var $_mime_boundary      =   "";                              // MIME boundary for multipart/form-data submit type
      var $_redirectaddr      =      false;                        // will be set if page fetched is a redirect
      var $_redirectdepth      =      0;                              // increments on an http redirect
      var $_frameurls            =       array();                  // frame src urls
      var $_framedepth      =      0;                              // increments on frame depth
      
      var $_isproxy            =      false;                        // set if using a proxy server
      var $_fp_timeout      =      120;                              // timeout for socket connection

/*======================================================================*\
      Function:      fetch
      Purpose:      fetch the contents of a web page
                        (and possibly other protocols in the
                        future like ftp, nntp, gopher, etc.)
      Input:            $URI      the location of the page to fetch
      Output:            $this->results      the output text from the fetch
\*======================================================================*/

      function fetch($URI)
      {
      
            //preg_match("|^([^:]+)://([^:/]+)(:[\d]+)*(.*)|",$URI,$URI_PARTS);
            $URI_PARTS = parse_url($URI);
            if (!empty($URI_PARTS["user"]))
                  $this->user = $URI_PARTS["user"];
            if (!empty($URI_PARTS["pass"]))
                  $this->pass = $URI_PARTS["pass"];
                        
            switch($URI_PARTS["scheme"])
            {
                  case "http":
                        $this->host = $URI_PARTS["host"];
                        if(!empty($URI_PARTS["port"]))
                              $this->port = $URI_PARTS["port"];
                        if($this->_connect($fp))
                        {
                              if($this->_isproxy)
                              {
                                    // using proxy, send entire URI
                                    $this->_httprequest($URI,$fp,$URI,$this->_httpmethod);
                              }
                              else
                              {
                                    $path = $URI_PARTS["path"].($URI_PARTS["query"] ? "?".$URI_PARTS["query"] : "");
                                    // no proxy, send only the path
                                    $this->_httprequest($path, $fp, $URI, $this->_httpmethod);
                              }
                              
                              $this->_disconnect($fp);

                              if($this->_redirectaddr)
                              {
                                    /* url was redirected, check if we've hit the max depth */
                                    if($this->maxredirs > $this->_redirectdepth)
                                    {
                                          // only follow redirect if it's on this site, or offsiteok is true
                                          if(preg_match("|^http://".preg_quote($this->host)."|i",$this->_redirectaddr) || $this->offsiteok)
                                          {
                                                /* follow the redirect */
                                                $this->_redirectdepth++;
                                                $this->lastredirectaddr=$this->_redirectaddr;
                                                $this->fetch($this->_redirectaddr);
                                          }
                                    }
                              }

                              if($this->_framedepth < $this->maxframes && count($this->_frameurls) > 0)
                              {
                                    $frameurls = $this->_frameurls;
                                    $this->_frameurls = array();
                                    
                                    while(list(,$frameurl) = each($frameurls))
                                    {
                                          if($this->_framedepth < $this->maxframes)
                                          {
                                                $this->fetch($frameurl);
                                                $this->_framedepth++;
                                          }
                                          else
                                                break;
                                    }
                              }                              
                        }
                        else
                        {
                              return false;
                        }
                        return true;                              
                        break;
                  case "https":
                        if(!$this->curl_path || (!is_executable($this->curl_path)))
                              return false;
                        $this->host = $URI_PARTS["host"];
                        if(!empty($URI_PARTS["port"]))
                              $this->port = $URI_PARTS["port"];
                        if($this->_isproxy)
                        {
                              // using proxy, send entire URI
                              $this->_httpsrequest($URI,$URI,$this->_httpmethod);
                        }
                        else
                        {
                              $path = $URI_PARTS["path"].($URI_PARTS["query"] ? "?".$URI_PARTS["query"] : "");
                              // no proxy, send only the path
                              $this->_httpsrequest($path, $URI, $this->_httpmethod);
                        }

                        if($this->_redirectaddr)
                        {
                              /* url was redirected, check if we've hit the max depth */
                              if($this->maxredirs > $this->_redirectdepth)
                              {
                                    // only follow redirect if it's on this site, or offsiteok is true
                                    if(preg_match("|^http://".preg_quote($this->host)."|i",$this->_redirectaddr) || $this->offsiteok)
                                    {
                                          /* follow the redirect */
                                          $this->_redirectdepth++;
                                          $this->lastredirectaddr=$this->_redirectaddr;
                                          $this->fetch($this->_redirectaddr);
                                    }
                              }
                        }

                        if($this->_framedepth < $this->maxframes && count($this->_frameurls) > 0)
                        {
                              $frameurls = $this->_frameurls;
                              $this->_frameurls = array();

                              while(list(,$frameurl) = each($frameurls))
                              {
                                    if($this->_framedepth < $this->maxframes)
                                    {
                                          $this->fetch($frameurl);
                                          $this->_framedepth++;
                                    }
                                    else
                                          break;
                              }
                        }                              
                        return true;                              
                        break;
                  default:
                        // not a valid protocol
                        $this->error      =      'Invalid protocol "'.$URI_PARTS["scheme"].'"\n';
                        return false;
                        break;
            }            
            return true;
      }

/*======================================================================*\
      Function:      submit
      Purpose:      submit an http form
      Input:            $URI      the location to post the data
                        $formvars      the formvars to use.
                              format: $formvars["var"] = "val";
      Output:            $this->results      the text output from the post
\*======================================================================*/

      function submit($URI, $formvars="", $formfiles="")
      {
            unset($postdata);
            
            $postdata = $this->_prepare_post_body($formvars, $formfiles);
                  
            $URI_PARTS = parse_url($URI);
            if (!empty($URI_PARTS["user"]))
                  $this->user = $URI_PARTS["user"];
            if (!empty($URI_PARTS["pass"]))
                  $this->pass = $URI_PARTS["pass"];
                        
            switch($URI_PARTS["scheme"])
            {
                  case "http":
                        $this->host = $URI_PARTS["host"];
                        if(!empty($URI_PARTS["port"]))
                              $this->port = $URI_PARTS["port"];
                        if($this->_connect($fp))
                        {
                              if($this->_isproxy)
                              {
                                    // using proxy, send entire URI
                                    $this->_httprequest($URI,$fp,$URI,$this->_submit_method,$this->_submit_type,$postdata);
                              }
                              else
                              {
                                    $path = $URI_PARTS["path"].($URI_PARTS["query"] ? "?".$URI_PARTS["query"] : "");
                                    // no proxy, send only the path
                                    $this->_httprequest($path, $fp, $URI, $this->_submit_method, $this->_submit_type, $postdata);
                              }
                              
                              $this->_disconnect($fp);

                              if($this->_redirectaddr)
                              {
                                    /* url was redirected, check if we've hit the max depth */
                                    if($this->maxredirs > $this->_redirectdepth)
                                    {                                    
                                          if(!preg_match("|^".$URI_PARTS["scheme"]."://|", $this->_redirectaddr))
                                                $this->_redirectaddr = $this->_expandlinks($this->_redirectaddr,$URI_PARTS["scheme"]."://".$URI_PARTS["host"]);                                    
                                          
                                          // only follow redirect if it's on this site, or offsiteok is true
                                          if(preg_match("|^http://".preg_quote($this->host)."|i",$this->_redirectaddr) || $this->offsiteok)
                                          {
                                                /* follow the redirect */
                                                $this->_redirectdepth++;
                                                $this->lastredirectaddr=$this->_redirectaddr;
                                                $this->submit($this->_redirectaddr,$formvars, $formfiles);
                                          }
                                    }
                              }

                              if($this->_framedepth < $this->maxframes && count($this->_frameurls) > 0)
                              {
                                    $frameurls = $this->_frameurls;
                                    $this->_frameurls = array();
                                    
                                    while(list(,$frameurl) = each($frameurls))
                                    {                                                                                    
                                          if($this->_framedepth < $this->maxframes)
                                          {
                                                $this->fetch($frameurl);
                                                $this->_framedepth++;
                                          }
                                          else
                                                break;
                                    }
                              }                              
                              
                        }
                        else
                        {
                              return false;
                        }
                        return true;                              
                        break;
                  case "https":
                        if(!$this->curl_path || (!is_executable($this->curl_path)))
                              return false;
                        $this->host = $URI_PARTS["host"];
                        if(!empty($URI_PARTS["port"]))
                              $this->port = $URI_PARTS["port"];
                        if($this->_isproxy)
                        {
                              // using proxy, send entire URI
                              $this->_httpsrequest($URI, $URI, $this->_submit_method, $this->_submit_type, $postdata);
                        }
                        else
                        {
                              $path = $URI_PARTS["path"].($URI_PARTS["query"] ? "?".$URI_PARTS["query"] : "");
                              // no proxy, send only the path
                              $this->_httpsrequest($path, $URI, $this->_submit_method, $this->_submit_type, $postdata);
                        }

                        if($this->_redirectaddr)
                        {
                              /* url was redirected, check if we've hit the max depth */
                              if($this->maxredirs > $this->_redirectdepth)
                              {                                    
                                    if(!preg_match("|^".$URI_PARTS["scheme"]."://|", $this->_redirectaddr))
                                          $this->_redirectaddr = $this->_expandlinks($this->_redirectaddr,$URI_PARTS["scheme"]."://".$URI_PARTS["host"]);                                    

                                    // only follow redirect if it's on this site, or offsiteok is true
                                    if(preg_match("|^http://".preg_quote($this->host)."|i",$this->_redirectaddr) || $this->offsiteok)
                                    {
                                          /* follow the redirect */
                                          $this->_redirectdepth++;
                                          $this->lastredirectaddr=$this->_redirectaddr;
                                          $this->submit($this->_redirectaddr,$formvars, $formfiles);
                                    }
                              }
                        }

                        if($this->_framedepth < $this->maxframes && count($this->_frameurls) > 0)
                        {
                              $frameurls = $this->_frameurls;
                              $this->_frameurls = array();

                              while(list(,$frameurl) = each($frameurls))
                              {                                                                                    
                                    if($this->_framedepth < $this->maxframes)
                                    {
                                          $this->fetch($frameurl);
                                          $this->_framedepth++;
                                    }
                                    else
                                          break;
                              }
                        }                              
                        return true;                              
                        break;
                        
                  default:
                        // not a valid protocol
                        $this->error      =      'Invalid protocol "'.$URI_PARTS["scheme"].'"\n';
                        return false;
                        break;
            }            
            return true;
      }

/*======================================================================*\
      Function:      fetchlinks
      Purpose:      fetch the links from a web page
      Input:            $URI      where you are fetching from
      Output:            $this->results      an array of the URLs
\*======================================================================*/

      function fetchlinks($URI)
      {
            if ($this->fetch($URI))
            {                  

                  if(is_array($this->results))
                  {
                        for($x=0;$x<count($this->results);$x++)
                              $this->results[$x] = $this->_striplinks($this->results[$x]);
                  }
                  else
                        $this->results = $this->_striplinks($this->results);

                  if($this->expandlinks)
                        $this->results = $this->_expandlinks($this->results, $URI);
                  return true;
            }
            else
                  return false;
      }

/*======================================================================*\
      Function:      fetchform
      Purpose:      fetch the form elements from a web page
      Input:            $URI      where you are fetching from
      Output:            $this->results      the resulting html form
\*======================================================================*/

      function fetchform($URI)
      {
            
            if ($this->fetch($URI))
            {                  

                  if(is_array($this->results))
                  {
                        for($x=0;$x<count($this->results);$x++)
                              $this->results[$x] = $this->_stripform($this->results[$x]);
                  }
                  else
                        $this->results = $this->_stripform($this->results);
                  
                  return true;
            }
            else
                  return false;
      }
      
      
/*======================================================================*\
      Function:      fetchtext
      Purpose:      fetch the text from a web page, stripping the links
      Input:            $URI      where you are fetching from
      Output:            $this->results      the text from the web page
\*======================================================================*/

      function fetchtext($URI)
      {
            if($this->fetch($URI))
            {                  
                  if(is_array($this->results))
                  {
                        for($x=0;$x<count($this->results);$x++)
                              $this->results[$x] = $this->_striptext($this->results[$x]);
                  }
                  else
                        $this->results = $this->_striptext($this->results);
                  return true;
            }
            else
                  return false;
      }

/*======================================================================*\
      Function:      submitlinks
      Purpose:      grab links from a form submission
      Input:            $URI      where you are submitting from
      Output:            $this->results      an array of the links from the post
\*======================================================================*/

      function submitlinks($URI, $formvars="", $formfiles="")
      {
            if($this->submit($URI,$formvars, $formfiles))
            {                  
                  if(is_array($this->results))
                  {
                        for($x=0;$x<count($this->results);$x++)
                        {
                              $this->results[$x] = $this->_striplinks($this->results[$x]);
                              if($this->expandlinks)
                                    $this->results[$x] = $this->_expandlinks($this->results[$x],$URI);
                        }
                  }
                  else
                  {
                        $this->results = $this->_striplinks($this->results);
                        if($this->expandlinks)
                              $this->results = $this->_expandlinks($this->results,$URI);
                  }
                  return true;
            }
            else
                  return false;
      }

/*======================================================================*\
      Function:      submittext
      Purpose:      grab text from a form submission
      Input:            $URI      where you are submitting from
      Output:            $this->results      the text from the web page
\*======================================================================*/

      function submittext($URI, $formvars = "", $formfiles = "")
      {
            if($this->submit($URI,$formvars, $formfiles))
            {                  
                  if(is_array($this->results))
                  {
                        for($x=0;$x<count($this->results);$x++)
                        {
                              $this->results[$x] = $this->_striptext($this->results[$x]);
                              if($this->expandlinks)
                                    $this->results[$x] = $this->_expandlinks($this->results[$x],$URI);
                        }
                  }
                  else
                  {
                        $this->results = $this->_striptext($this->results);
                        if($this->expandlinks)
                              $this->results = $this->_expandlinks($this->results,$URI);
                  }
                  return true;
            }
            else
                  return false;
      }

      

/*======================================================================*\
      Function:      set_submit_multipart
      Purpose:      Set the form submission content type to
                        multipart/form-data
\*======================================================================*/
      function set_submit_multipart()
      {
            $this->_submit_type = "multipart/form-data";
      }

      
/*======================================================================*\
      Function:      set_submit_normal
      Purpose:      Set the form submission content type to
                        application/x-www-form-urlencoded
\*======================================================================*/
      function set_submit_normal()
      {
            $this->_submit_type = "application/x-www-form-urlencoded";
      }

      
      

/*======================================================================*\
      Private functions
\*======================================================================*/
      
      
/*======================================================================*\
      Function:      _striplinks
      Purpose:      strip the hyperlinks from an html document
      Input:            $document      document to strip.
      Output:            $match            an array of the links
\*======================================================================*/

      function _striplinks($document)
      {      
            preg_match_all("'<\s*a\s.*?href\s*=\s*                  # find <a href=
                                    ([\"\'])?                              # find single or double quote
                                    (?(1) (.*?)\\1 | ([^\s\>]+))            # if quote found, match up to next matching
                                                                              # quote, otherwise match up to next space
                                    'isx",$document,$links);
                                    

            // catenate the non-empty matches from the conditional subpattern

            while(list($key,$val) = each($links[2]))
            {
                  if(!empty($val))
                        $match[] = $val;
            }                        
            
            while(list($key,$val) = each($links[3]))
            {
                  if(!empty($val))
                        $match[] = $val;
            }            
            
            // return the links
            return $match;
      }

/*======================================================================*\
      Function:      _stripform
      Purpose:      strip the form elements from an html document
      Input:            $document      document to strip.
      Output:            $match            an array of the links
\*======================================================================*/

      function _stripform($document)
      {      
            preg_match_all("'<\/?(FORM|INPUT|SELECT|TEXTAREA|(OPTION))[^<>]*>(?(2)(.*(?=<\/?(option|select)[^<>]*>[\r\n]*)|(?=[\r\n]*))|(?=[\r\n]*))'Usi",$document,$elements);
            
            // catenate the matches
            $match = implode("\r\n",$elements[0]);
                        
            // return the links
            return $match;
      }

      
      
/*======================================================================*\
      Function:      _striptext
      Purpose:      strip the text from an html document
      Input:            $document      document to strip.
      Output:            $text            the resulting text
\*======================================================================*/

      function _striptext($document)
      {
            
            // I didn't use preg eval (//e) since that is only available in PHP 4.0.
            // so, list your entities one by one here. I included some of the
            // more common ones.
                                                
            $search = array("'<script[^>]*?>.*?</script>'si",      // strip out javascript
                                    "'<[\/\!]*?[^<>]*?>'si",                  // strip out html tags
                                    "'([\r\n])[\s]+'",                              // strip out white space
                                    "'&(quot|#34|#034|#x22);'i",            // replace html entities
                                    "'&(amp|#38|#038|#x26);'i",                  // added hexadecimal values
                                    "'&(lt|#60|#060|#x3c);'i",
                                    "'&(gt|#62|#062|#x3e);'i",
                                    "'&(nbsp|#160|#xa0);'i",
                                    "'&(iexcl|#161);'i",
                                    "'&(cent|#162);'i",
                                    "'&(pound|#163);'i",
                                    "'&(copy|#169);'i",
                                    "'&(reg|#174);'i",
                                    "'&(deg|#176);'i",
                                    "'&(#39|#039|#x27);'",
                                    "'&(euro|#8364);'i",                        // europe
                                    "'&a(uml|UML);'",                              // german
                                    "'&o(uml|UML);'",
                                    "'&u(uml|UML);'",
                                    "'&A(uml|UML);'",
                                    "'&O(uml|UML);'",
                                    "'&U(uml|UML);'",
                                    "'&szlig;'i",
                                    );
            $replace = array(      "",
                                          "",
                                          "\\1",
                                          "\"",
                                          "&",
                                          "<",
                                          ">",
                                          " ",
                                          chr(161),
                                          chr(162),
                                          chr(163),
                                          chr(169),
                                          chr(174),
                                          chr(176),
                                          chr(39),
                                          chr(128),
                                          "ä",
                                          "ö",
                                          "ü",
                                          "Ä",
                                          "Ö",
                                          "Ü",
                                          "ß",
                                    );
                              
            $text = preg_replace($search,$replace,$document);
                                                
            return $text;
      }

/*======================================================================*\
      Function:      _expandlinks
      Purpose:      expand each link into a fully qualified URL
      Input:            $links                  the links to qualify
                        $URI                  the full URI to get the base from
      Output:            $expandedLinks      the expanded links
\*======================================================================*/

      function _expandlinks($links,$URI)
      {
            
            preg_match("/^[^\?]+/",$URI,$match);

            $match = preg_replace("|/[^\/\.]+\.[^\/\.]+$|","",$match[0]);
                        
            $search = array(       "|^http://".preg_quote($this->host)."|i",
                                          "|^(?!http://)(\/)?(?!mailto:)|i",
                                          "|/\./|",
                                          "|/[^\/]+/\.\./|"
                                    );
                                    
            $replace = array(      "",
                                          $match."/",
                                          "/",
                                          "/"
                                    );                  
                        
            $expandedLinks = preg_replace($search,$replace,$links);

            return $expandedLinks;
      }

/*======================================================================*\
      Function:      _httprequest
      Purpose:      go get the http data from the server
      Input:            $url            the url to fetch
                        $fp                  the current open file pointer
                        $URI            the full URI
                        $body            body contents to send if any (POST)
      Output:            
\*======================================================================*/
      
      function _httprequest($url,$fp,$URI,$http_method,$content_type="",$body="")
      {
            $cookie_headers = '';
            if($this->passcookies && $this->_redirectaddr)
                  $this->setcookies();
                  
            $URI_PARTS = parse_url($URI);
            if(empty($url))
                  $url = "/";
            $headers = $http_method." ".$url." ".$this->_httpversion."\r\n";            
            if(!empty($this->agent))
                  $headers .= "User-Agent: ".$this->agent."\r\n";
            if(!empty($this->host) && !isset($this->rawheaders['Host']))
                  $headers .= "Host: ".$this->host."\r\n";
            if(!empty($this->accept))
                  $headers .= "Accept: ".$this->accept."\r\n";
            if(!empty($this->referer))
                  $headers .= "Referer: ".$this->referer."\r\n";
            if(!empty($this->cookies))
            {                  
                  if(!is_array($this->cookies))
                        $this->cookies = (array)$this->cookies;
      
                  reset($this->cookies);
                  if ( count($this->cookies) > 0 ) {
                        $cookie_headers .= 'Cookie: ';
                        foreach ( $this->cookies as $cookieKey => $cookieVal ) {
                        $cookie_headers .= $cookieKey."=".urlencode($cookieVal)."; ";
                        }
                        $headers .= substr($cookie_headers,0,-2) . "\r\n";
                  }
            }
            if(!empty($this->rawheaders))
            {
                  if(!is_array($this->rawheaders))
                        $this->rawheaders = (array)$this->rawheaders;
                  while(list($headerKey,$headerVal) = each($this->rawheaders))
                        $headers .= $headerKey.": ".$headerVal."\r\n";
            }
            if(!empty($content_type)) {
                  $headers .= "Content-type: $content_type";
                  if ($content_type == "multipart/form-data")
                        $headers .= "; boundary=".$this->_mime_boundary;
                  $headers .= "\r\n";
            }
            if(!empty($body))      
                  $headers .= "Content-length: ".strlen($body)."\r\n";
            if(!empty($this->user) || !empty($this->pass))      
                  $headers .= "Authorization: Basic ".base64_encode($this->user.":".$this->pass)."\r\n";

            $headers .= "\r\n";
            
            // set the read timeout if needed
            if ($this->read_timeout > 0)
                  socket_set_timeout($fp, $this->read_timeout);
            $this->timed_out = false;
            
            fwrite($fp,$headers.$body,strlen($headers.$body));
            
            $this->_redirectaddr = false;
            unset($this->headers);
                                    
            while($currentHeader = fgets($fp,$this->_maxlinelen))
            {
                  if ($this->read_timeout > 0 && $this->_check_timeout($fp))
                  {
                        $this->status=-100;
                        return false;
                  }
                        
                  if($currentHeader == "\r\n")
                        break;
                                    
                  // if a header begins with Location: or URI:, set the redirect
                  if(preg_match("/^(Location:|URI:)/i",$currentHeader))
                  {
                        // get URL portion of the redirect
                        preg_match("/^(Location:|URI:)[ ]+(.*)/",chop($currentHeader),$matches);
                        // look for :// in the Location header to see if hostname is included
                        if(!preg_match("|\:\/\/|",$matches[2]))
                        {
                              // no host in the path, so prepend
                              $this->_redirectaddr = $URI_PARTS["scheme"]."://".$this->host.":".$this->port;
                              // eliminate double slash
                              if(!preg_match("|^/|",$matches[2]))
                                          $this->_redirectaddr .= "/".$matches[2];
                              else
                                          $this->_redirectaddr .= $matches[2];
                        }
                        else
                              $this->_redirectaddr = $matches[2];
                  }
            
                  if(preg_match("|^HTTP/|",$currentHeader))
                  {
                if(preg_match("|^HTTP/[^\s]*\s(.*?)\s|",$currentHeader, $status))
                        {
                              $this->status= $status[1];
                }                        
                        $this->response_code = $currentHeader;
                  }
                        
                  $this->headers[] = $currentHeader;
            }

            $results = '';
            do {
                $_data = fread($fp, $this->maxlength);
                if (strlen($_data) == 0) {
                    break;
                }
                $results .= $_data;
            } while(true);

            if ($this->read_timeout > 0 && $this->_check_timeout($fp))
            {
                  $this->status=-100;
                  return false;
            }
            
            // check if there is a a redirect meta tag
            
            if(preg_match("'<meta[\s]*http-equiv[^>]*?content[\s]*=[\s]*[\"\']?\d+;[\s]+URL[\s]*=[\s]*([^\"\']*?)[\"\']?>'i",$results,$match))
            {
                  $this->_redirectaddr = $this->_expandlinks($match[1],$URI);      
            }

            // have we hit our frame depth and is there frame src to fetch?
            if(($this->_framedepth < $this->maxframes) && preg_match_all("'<frame\s+.*src[\s]*=[\'\"]?([^\'\"\>]+)'i",$results,$match))
            {
                  $this->results[] = $results;
                  for($x=0; $x<count($match[1]); $x++)
                        $this->_frameurls[] = $this->_expandlinks($match[1][$x],$URI_PARTS["scheme"]."://".$this->host);
            }
            // have we already fetched framed content?
            elseif(is_array($this->results))
                  $this->results[] = $results;
            // no framed content
            else
                  $this->results = $results;
            
            return true;
      }

/*======================================================================*\
      Function:      _httpsrequest
      Purpose:      go get the https data from the server using curl
      Input:            $url            the url to fetch
                        $URI            the full URI
                        $body            body contents to send if any (POST)
      Output:            
\*======================================================================*/
      
      function _httpsrequest($url,$URI,$http_method,$content_type="",$body="")
      {
            if($this->passcookies && $this->_redirectaddr)
                  $this->setcookies();

            $headers = array();            
                              
            $URI_PARTS = parse_url($URI);
            if(empty($url))
                  $url = "/";
            // GET ... header not needed for curl
            //$headers[] = $http_method." ".$url." ".$this->_httpversion;            
            if(!empty($this->agent))
                  $headers[] = "User-Agent: ".$this->agent;
            if(!empty($this->host))
                  $headers[] = "Host: ".$this->host;
            if(!empty($this->accept))
                  $headers[] = "Accept: ".$this->accept;
            if(!empty($this->referer))
                  $headers[] = "Referer: ".$this->referer;
            if(!empty($this->cookies))
            {                  
                  if(!is_array($this->cookies))
                        $this->cookies = (array)$this->cookies;
      
                  reset($this->cookies);
                  if ( count($this->cookies) > 0 ) {
                        $cookie_str = 'Cookie: ';
                        foreach ( $this->cookies as $cookieKey => $cookieVal ) {
                        $cookie_str .= $cookieKey."=".urlencode($cookieVal)."; ";
                        }
                        $headers[] = substr($cookie_str,0,-2);
                  }
            }
            if(!empty($this->rawheaders))
            {
                  if(!is_array($this->rawheaders))
                        $this->rawheaders = (array)$this->rawheaders;
                  while(list($headerKey,$headerVal) = each($this->rawheaders))
                        $headers[] = $headerKey.": ".$headerVal;
            }
            if(!empty($content_type)) {
                  if ($content_type == "multipart/form-data")
                        $headers[] = "Content-type: $content_type; boundary=".$this->_mime_boundary;
                  else
                        $headers[] = "Content-type: $content_type";
            }
            if(!empty($body))      
                  $headers[] = "Content-length: ".strlen($body);
            if(!empty($this->user) || !empty($this->pass))      
                  $headers[] = "Authorization: BASIC ".base64_encode($this->user.":".$this->pass);
                  
            for($curr_header = 0; $curr_header < count($headers); $curr_header++)
                  $cmdline_params .= " -H \"".$headers[$curr_header]."\"";
            
            if(!empty($body))
                  $cmdline_params .= " -d \"$body\"";
            
            if($this->read_timeout > 0)
                  $cmdline_params .= " -m ".$this->read_timeout;
            
            $headerfile = uniqid(time());

            exec($this->curl_path." -D \"/tmp/$headerfile\"".$cmdline_params." ".$URI,$results,$return);
            
            if($return)
            {
                  $this->error = "Error: cURL could not retrieve the document, error $return.";
                  return false;
            }
                  
                  
            $results = implode("\r\n",$results);
            
            $result_headers = file("/tmp/$headerfile");
                                    
            $this->_redirectaddr = false;
            unset($this->headers);
                                    
            for($currentHeader = 0; $currentHeader < count($result_headers); $currentHeader++)
            {
                  
                  // if a header begins with Location: or URI:, set the redirect
                  if(preg_match("/^(Location: |URI: )/i",$result_headers[$currentHeader]))
                  {
                        // get URL portion of the redirect
                        preg_match("/^(Location: |URI:)\s+(.*)/",chop($result_headers[$currentHeader]),$matches);
                        // look for :// in the Location header to see if hostname is included
                        if(!preg_match("|\:\/\/|",$matches[2]))
                        {
                              // no host in the path, so prepend
                              $this->_redirectaddr = $URI_PARTS["scheme"]."://".$this->host.":".$this->port;
                              // eliminate double slash
                              if(!preg_match("|^/|",$matches[2]))
                                          $this->_redirectaddr .= "/".$matches[2];
                              else
                                          $this->_redirectaddr .= $matches[2];
                        }
                        else
                              $this->_redirectaddr = $matches[2];
                  }
            
                  if(preg_match("|^HTTP/|",$result_headers[$currentHeader]))
                        $this->response_code = $result_headers[$currentHeader];

                  $this->headers[] = $result_headers[$currentHeader];
            }

            // check if there is a a redirect meta tag
            
            if(preg_match("'<meta[\s]*http-equiv[^>]*?content[\s]*=[\s]*[\"\']?\d+;[\s]+URL[\s]*=[\s]*([^\"\']*?)[\"\']?>'i",$results,$match))
            {
                  $this->_redirectaddr = $this->_expandlinks($match[1],$URI);      
            }

            // have we hit our frame depth and is there frame src to fetch?
            if(($this->_framedepth < $this->maxframes) && preg_match_all("'<frame\s+.*src[\s]*=[\'\"]?([^\'\"\>]+)'i",$results,$match))
            {
                  $this->results[] = $results;
                  for($x=0; $x<count($match[1]); $x++)
                        $this->_frameurls[] = $this->_expandlinks($match[1][$x],$URI_PARTS["scheme"]."://".$this->host);
            }
            // have we already fetched framed content?
            elseif(is_array($this->results))
                  $this->results[] = $results;
            // no framed content
            else
                  $this->results = $results;

            unlink("/tmp/$headerfile");
            
            return true;
      }

/*======================================================================*\
      Function:      setcookies()
      Purpose:      set cookies for a redirection
\*======================================================================*/
      
      function setcookies()
      {
            for($x=0; $x<count($this->headers); $x++)
            {
            if(preg_match('/^set-cookie:[\s]+([^=]+)=([^;]+)/i', $this->headers[$x],$match))
                  $this->cookies[$match[1]] = $match[2];
            }
      }

      
/*======================================================================*\
      Function:      _check_timeout
      Purpose:      checks whether timeout has occurred
      Input:            $fp      file pointer
\*======================================================================*/

      function _check_timeout($fp)
      {
            if ($this->read_timeout > 0) {
                  $fp_status = socket_get_status($fp);
                  if ($fp_status["timed_out"]) {
                        $this->timed_out = true;
                        return true;
                  }
            }
            return false;
      }

/*======================================================================*\
      Function:      _connect
      Purpose:      make a socket connection
      Input:            $fp      file pointer
\*======================================================================*/
      
      function _connect(&$fp)
      {
            if(!empty($this->proxy_host) && !empty($this->proxy_port))
                  {
                        $this->_isproxy = true;
                        $host = $this->proxy_host;
                        $port = $this->proxy_port;
                  }
            else
            {
                  $host = $this->host;
                  $port = $this->port;
            }
      
            $this->status = 0;
            
            if($fp = fsockopen(
                              $host,
                              $port,
                              $errno,
                              $errstr,
                              $this->_fp_timeout
                              ))
            {
                  // socket connection succeeded

                  return true;
            }
            else
            {
                  // socket connection failed
                  $this->status = $errno;
                  switch($errno)
                  {
                        case -3:
                              $this->error="socket creation failed (-3)";
                        case -4:
                              $this->error="dns lookup failure (-4)";
                        case -5:
                              $this->error="connection refused or timed out (-5)";
                        default:
                              $this->error="connection failed (".$errno.")";
                  }
                  return false;
            }
      }
/*======================================================================*\
      Function:      _disconnect
      Purpose:      disconnect a socket connection
      Input:            $fp      file pointer
\*======================================================================*/
      
      function _disconnect($fp)
      {
            return(fclose($fp));
      }

      
/*======================================================================*\
      Function:      _prepare_post_body
      Purpose:      Prepare post body according to encoding type
      Input:            $formvars  - form variables
                        $formfiles - form upload files
      Output:            post body
\*======================================================================*/
      
      function _prepare_post_body($formvars, $formfiles)
      {
            settype($formvars, "array");
            settype($formfiles, "array");

            if (count($formvars) == 0 && count($formfiles) == 0)
                  return;
            
            switch ($this->_submit_type) {
                  case "application/x-www-form-urlencoded":
                        reset($formvars);
                        while(list($key,$val) = each($formvars)) {
                              if (is_array($val) || is_object($val)) {
                                    while (list($cur_key, $cur_val) = each($val)) {
                                          $postdata .= urlencode($key)."[]=".urlencode($cur_val)."&";
                                    }
                              } else
                                    $postdata .= urlencode($key)."=".urlencode($val)."&";
                        }
                        break;

                  case "multipart/form-data":
                        $this->_mime_boundary = "Snoopy".md5(uniqid(microtime()));
                        
                        reset($formvars);
                        while(list($key,$val) = each($formvars)) {
                              if (is_array($val) || is_object($val)) {
                                    while (list($cur_key, $cur_val) = each($val)) {
                                          $postdata .= "--".$this->_mime_boundary."\r\n";
                                          $postdata .= "Content-Disposition: form-data; name=\"$key\[\]\"\r\n\r\n";
                                          $postdata .= "$cur_val\r\n";
                                    }
                              } else {
                                    $postdata .= "--".$this->_mime_boundary."\r\n";
                                    $postdata .= "Content-Disposition: form-data; name=\"$key\"\r\n\r\n";
                                    $postdata .= "$val\r\n";
                              }
                        }
                        
                        reset($formfiles);
                        while (list($field_name, $file_names) = each($formfiles)) {
                              settype($file_names, "array");
                              while (list(, $file_name) = each($file_names)) {
                                    if (!is_readable($file_name)) continue;

                                    $fp = fopen($file_name, "r");
                                    $file_content = fread($fp, filesize($file_name));
                                    fclose($fp);
                                    $base_name = basename($file_name);

                                    $postdata .= "--".$this->_mime_boundary."\r\n";
                                    $postdata .= "Content-Disposition: form-data; name=\"$field_name\"; filename=\"$base_name\"\r\n\r\n";
                                    $postdata .= "$file_content\r\n";
                              }
                        }
                        $postdata .= "--".$this->_mime_boundary."--\r\n";
                        break;
            }

            return $postdata;
      }
}

?>
==
0
 
LVL 10

Expert Comment

by:frugle
ID: 11923169
looks like it's trying to open http://images/ - which doesn't exist.

Check the contents of your text files.

Mike

ps. I don't know what you're trying to do, but it does look a little overcomplicated.
0
 

Author Comment

by:wcws
ID: 11923798
Could please help me , I know this complicated - I will really appreciated if any one here can help me out .

Thanking you all in advance.

0
 
LVL 4

Expert Comment

by:aratani
ID: 11924477
25 points is too low for most people to go and look at a question that big. Just my personal piece of advice; your question should probably be a 400+ point quesiton.

AJ
0
 
LVL 10

Expert Comment

by:frugle
ID: 11924517
In case you missed my previous answer, and suggestion to rectify it - here it is again.

"it looks like the script is trying to open http://images/ - which doesn't exist."

The line that is most probably causing it is $img=readwebpage($imgurl,$currenturl);

"Check the contents of your text files." (links.txt, writtenlinks.txt, images.txt, lastredirects.txt, pagelevel.txt, imagecount.txt, pagecount.txt)

I can't do any more than suggest, as I don't have access to headerandfooter2.inc, vars.inc, goback.inc, webtools.inc, pulltools.inc, tools.inc, filetools.inc, db.inc, changegoogle.inc, addextrapages.inc, stripbody.inc, tags.inc - and that's NOT a request to post them.

Mike
0
 

Author Comment

by:wcws
ID: 11925543
Please kindly visit

www.sohaib.com/files.zip

And you will find all the files you've requested.

Thanking you in advance, and also I've increase the points to 400.

0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 10

Expert Comment

by:frugle
ID: 11929057
Warning: mysql_connect(): Access denied for user: 'root@localhost' (Using password: NO) in /home/virtual/site37/fst/var/www/html/includes/functions/database.php on line 19

erk, you're not using ROOT privs for the webserver user are you? that's just asking for trouble!

for the third, and hopefully last time, "Check the contents of your text files."

it looks like the script is trying to open http://images/ - which doesn't exist.

This will cause the FSOCK error.

Mike
0
 

Author Comment

by:wcws
ID: 11930415
I've fixed that http://images/ images problem - but now what you are saying about root privs ? I didnt quiet understand. I am connecting my database using the user which owns by domain.

In vars.inc I've enter the URL information before images so that problem is fine now , now what problem I am having is

Warning: mkdir(http://www.abc.com/news/pages/1): No such file or directory in /home/virtual/site4/fst/var/www/html/news/filetools.inc on line 55

Although there is already a directory there in the name of pages and 1 /pages/1 so I dont know why its saying warning : mkdir ?
=======
0
 
LVL 10

Accepted Solution

by:
frugle earned 400 total points
ID: 11932029
Don't worry about the root thing, I went up a directory from http://www.sohaib.com/files.zip to http://www.sohaib.com/ and was presented with that error.

If it isn't your site it's not your concern - though if it is, it's a problem with some code.

mkdir requires a local path, you can't mkdir (http:// etc.)

http://www.php.net/manual/en/function.mkdir.php

You should really test for the existance of a directory before trying to make it...

if (!is_dir('/directory'){
    mkdir('/directory') or die('Couldn't make directory "/directory" - please check permissions');
}

This is a separate question though - if your FSOCK question is answered fully you can close it :-)

Cheers

Mike
0
 

Author Comment

by:wcws
ID: 11933450
Just so you know I am not a PHP Expert so please kindly guide me where should I insert this line ??

if (!is_dir('/directory'){
    mkdir('/directory') or die('Couldn't make directory "/directory" - please check permissions');
}

Also if things didnt work at all - I will open a New Question and setup points to 3,000 and will provide you the script to work on and fix the problem in it . ANd make it workable 100% and in All OS.

Thank you,
0
 
LVL 10

Expert Comment

by:frugle
ID: 11933996
Fixing the entire script is a bit out of the scope of ExpertsExchange - I'm here to guide you, not to work for you!

If this is someone else's script you're poking at, they may want to know about the errors you have, and will probably be able to fix it quicker - look at reporting the bugs to the authors.

as for inserting the line, don't do it. it was an example, and would probably break your script if you added it without knowing what you were doing.

Mike
0
 

Author Comment

by:wcws
ID: 11934097
So I think that whole thread doesnt help me .
0
 
LVL 10

Expert Comment

by:frugle
ID: 11934621
The phrase biting off more than you can chew springs immediately to mind.

well, at least we cleared up the FSOCK problems...

Mike
0
 

Author Comment

by:wcws
ID: 11934958
Well, the FSOCk problem didnt go away ! I am still having that problem
0

Featured Post

Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

Join & Write a Comment

Introduction Many web sites contain image galleries; a common design for these galleries includes a page with a collection of thumbnail images.  You can click on each of the thumbnail images to see the larger version of the image.  This is easily i…
Generating table dynamically is the most common issue faced by php developers.... So it seems there is a need of an article that explains the basic concept of generating tables dynamically. It just requires a basic knowledge of html and little maths…
The viewer will learn how to look for a specific file type in a local or remote server directory using PHP.
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.

747 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now