Solved

How do you combine cURL and Regex in PHP

Posted on 2011-02-24
7
571 Views
Last Modified: 2012-05-11
Experts,

How do I corectly structure this cURL and Regex to pull some data from a website then insert it in my database.

Thanks for your help!
<?php


mysql_connect("Host","UN","PW") or die("Unable to connect to SQL server");
mysql_select_db('DB') or die("Unable to SELECT DB");

echo "Connected to DB";
echo "<BR>";

$query = "SELECT * FROM temptable";
$result = mysql_query($query) or die('Error: '.mysql_error());
while($row = mysql_fetch_array($result))
  {
$sku=mysql_real_escape_string($row['sku']);
echo $sku."<br />";


  
    
	$pages = array('home' => 
'http://www.mysite.com/', 
               'login' => 
'https://www.mysite.com/signin.asp', 
               'schedule' => 
'http://www.mysite.com/product.asp?&sku=' . $sku . ''); 
    $ch = curl_init(); 
    //Set options for curl session 
    $options = array(CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1)', 
             CURLOPT_SSL_VERIFYPEER => FALSE, 
             CURLOPT_SSL_VERIFYHOST => 2, 
             CURLOPT_HEADER => TRUE, 
             //CURLOPT_RETURNTRANSFER => TRUE, 
             CURLOPT_COOKIEFILE => '/cookie.txt', 
             CURLOPT_COOKIEJAR => '/cookies.txt'); 
    
    //Hit home page for session cookie 
    $options[CURLOPT_URL] = $pages['home']; 
    curl_setopt_array($ch, $options); 
    curl_exec($ch); 
    
    //Login 
    $options[CURLOPT_URL] = $pages['login']; 
    $options[CURLOPT_POST] = TRUE; 
    $options[CURLOPT_POSTFIELDS] = 'email=MyEmail&password=PW&submitBtn'; 
    $options[CURLOPT_FOLLOWLOCATION] = FALSE; 
    curl_setopt_array($ch, $options); 
    curl_exec($ch); 
    
    //Hit schedule page 
    $options[CURLOPT_URL] = $pages['schedule']; 
    curl_setopt_array($ch, $options); 
    $schedule = curl_exec($ch); 
    
    //Output schedule 
    //echo $schedule;
	
	preg_match_all('%<title>(.*?)</title>%s',$schedule,$matches0);

$title=$matches0[1];
echo $title."<br />";


mysql_query("INSERT INTO temptable2 (title) VALUES('".mysql_real_escape_string($title)."') ");



	}	 
    
    //Close curl session 
    curl_close($ch);
	 
?>

Open in new window

0
Comment
Question by:rlb1
  • 4
  • 3
7 Comments
 
LVL 34

Expert Comment

by:Beverley Portlock
Comment Utility
It looks OK on inspection. What is wrong with it?
0
 

Author Comment

by:rlb1
Comment Utility
The cURL functions are working properly and I am getting data returned from all $pages (home, login, schedule).   The REGEX is not picking up the 'Title' which is the same code that is working perfectly on another script.  The script is not inserting the 'Title' to the database.  

I am not sure what is wrong.  I have spent several hours working on it...  



0
 
LVL 34

Expert Comment

by:Beverley Portlock
Comment Utility
Change

$title=$matches0[1];

to

$title = $matches0[1][0];
0
How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

 

Author Comment

by:rlb1
Comment Utility
bportlock: Thanks.  It seems that the REGEX statement may not be recognizing   $schedule (which is the 3rd page in the array).  The database is receiving some blank inserts with no data.    I have adjusted several things in the script and cannot come up with a solution.   The login is working and the pages are appearing correctly.  Also, echo $title."<br />"; is not returning any data either.

preg_match('%<title>(.*?)</title>%',$schedule,$matches0);

$title=$matches0[1];
echo $title."<br />";

Thanks for your help!!
0
 

Author Comment

by:rlb1
Comment Utility
8 (continuous working on it) Hours Later...  I started completely over, rewrote the script and got it to work!!  


<?php

$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIEJAR, "..cookie.txt");
curl_setopt($ch, CURLOPT_URL,"https://www.mywebsite.com/signin.asp?");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "email=myemail&password=mypassword&submitBtn");

ob_start();      // prevent any output
curl_exec ($ch); // execute the curl command
ob_end_clean();  // stop preventing output

curl_close ($ch);
unset($ch);

$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_COOKIEFILE, "../cookie.txt");
curl_setopt($ch, CURLOPT_URL,"http://www.mywebsite.com/product.asp?sku=03170");

$buf2 = curl_exec ($ch);

curl_close ($ch);

//echo "<PRE>".htmlentities($buf2);

//echo $buf2;


preg_match('%<title>(.*?)<\/title>%',$buf2,$matches0);

$title=$matches0[1];
echo $title."<br />";







?>

Open in new window

0
 
LVL 34

Accepted Solution

by:
Beverley Portlock earned 500 total points
Comment Utility
:-)

0
 

Author Closing Comment

by:rlb1
Comment Utility
Thanks!
0

Featured Post

How to improve team productivity

Quip adds documents, spreadsheets, and tasklists to your Slack experience
- Elevate ideas to Quip docs
- Share Quip docs in Slack
- Get notified of changes to your docs
- Available on iOS/Android/Desktop/Web
- Online/Offline

Join & Write a Comment

Whatever be the reason, if you are working on web development side,  you will need day-today validation codes like email validation, date validation , IP address validation, phone validation on any of the edit page or say at the time of registration…
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

7 Experts available now in Live!

Get 1:1 Help Now