Want to protect your cyber security and still get fast solutions? Ask a secure question today.Go Premium

x
?
Solved

Download images using Curl

Posted on 2011-10-02
14
Medium Priority
?
1,651 Views
Last Modified: 2012-05-12
Hello all,

I have to import images from a webservice.. I found a nice piece of code online using php curl, but it only seems to work with 1 image. As soon as I add another url to the array it stops working.

The main issue:
The images stored on the webservice have a file attachment header which makes the images not show in google chrome. I currently import the path to the images along with other data the image is related to from the webservice into a database and display the images like this on the site. That worked fine till they added the file attachment header and as said now chrome won't display the images... So I figured I would download the images to the server aswell

Question: Will this even be fixed after downloading the image using curl or will the attachment header remain?

Another issue is that since I'm going to try this I would also like to downsize the images when downloading them because the images are quite large sometimes on the webservice...

Since I want the original urls to remain in the database I want to make it as follows
The script below needs to get the urls from another php file in which I pull the paths to the images from the database and display them in a repeat region... This file should replace the array in code below. Next I would like to downsize the image on download and last if necsessary remove the file attachment header from the downloaded image.. Is this possible? ANd if so How?

I am new to curl and fairly new to php overall. Any help would be greatly appreciated...

WKR
 RV

 
<?php
 
$urls=array(
'http://www.xxx.nl/files/filename1.jpg',
'http://www.xxx.nl/files/filename2.jpg');
 
$save_to='path/comes/here/';
 
$mh = curl_multi_init();
foreach ($urls as $i => $url) {
    $g=$save_to.basename($url);
    if(!is_file($g)){
        $conn[$i]=curl_init($url);
        $fp[$i]=fopen ($g, "w");
        curl_setopt ($conn[$i], CURLOPT_FILE, $fp[$i]);
        curl_setopt ($conn[$i], CURLOPT_HEADER ,0);
        curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);
        curl_multi_add_handle ($mh,$conn[$i]);
    }
}
do {
    $n=curl_multi_exec($mh,$active);
}
while ($active);
foreach ($urls as $i => $url) {
    curl_multi_remove_handle($mh,$conn[$i]);
    curl_close($conn[$i]);
    fclose ($fp[$i]);
}
curl_multi_close($mh);
?>

Open in new window

0
Comment
Question by:Witheet
  • 7
  • 6
14 Comments
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 36899626
Is that really the URL of the files you want to download?  If not, please post the actual URL of two or three files you want.  I will see if I can give you a script that will download them.  Bets regards, ~Ray
0
 
LVL 48

Expert Comment

by:hernst42
ID: 36899632
Instead of using the multi handle (parallel downloads) you can just loop over all images
php
 
$urls=array(
'http://www.xxx.nl/files/filename1.jpg',
'http://www.xxx.nl/files/filename2.jpg');
 
$save_to='path/comes/here/';
 
foreach ($urls as $i => $url) {
    $g=$save_to.basename($url);
    if(is_file($g)){
        continue;
    }
    $conn=curl_init($url);
    $fp=fopen ($g, "w");
    curl_setopt ($conn, CURLOPT_FILE, $fp[$i]);
    curl_setopt ($conn, CURLOPT_HEADER ,0);
    curl_setopt($conn,CURLOPT_CONNECTTIMEOUT,60);
    curl_exec($conn);
    curl_close($conn);
    fclose($fp);
}

Open in new window

0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 36899850
Agree with hernst42.  You might need some error visualization.  But tell us the actual URLs and we can give you a better answer.
<?php // RAY_curl_example.php
error_reporting(E_ALL);



// A FUNCTION TO RUN A CURL-GET CLIENT CALL TO A FOREIGN SERVER
function my_curl
( $url
, $get_array=array()
, $timeout=3
, $error_report=TRUE
)
{
    // PREPARE THE ARGUMENT STRING IF NEEDED
    $get_string = NULL;
    foreach ($get_array as $key => $val)
    {
        $get_string
        = $get_string
        . urlencode($key)
        . '='
        . urlencode($val)
        . '&';
    }
    $get_string = rtrim($get_string, '&');
    if (!empty($get_string)) $url .= '?' . $get_string;

    // START CURL
    $curl = curl_init();

    // HEADERS AND OPTIONS APPEAR TO BE A FIREFOX BROWSER REFERRED BY GOOGLE
    $header[] = "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5";
    $header[] = "Cache-Control: max-age=0";
    $header[] = "Connection: keep-alive";
    $header[] = "Keep-Alive: 300";
    $header[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
    $header[] = "Accept-Language: en-us,en;q=0.5";
    $header[] = "Pragma: "; // BROWSERS USUALLY LEAVE BLANK

    // SET THE CURL OPTIONS - SEE http://php.net/manual/en/function.curl-setopt.php
    curl_setopt( $curl, CURLOPT_URL,            $url  );
    curl_setopt( $curl, CURLOPT_USERAGENT,      'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6'  );
    curl_setopt( $curl, CURLOPT_HTTPHEADER,     $header  );
    curl_setopt( $curl, CURLOPT_REFERER,        'http://www.google.com'  );
    curl_setopt( $curl, CURLOPT_ENCODING,       'gzip,deflate'  );
    curl_setopt( $curl, CURLOPT_AUTOREFERER,    TRUE  );
    curl_setopt( $curl, CURLOPT_RETURNTRANSFER, TRUE  );
    curl_setopt( $curl, CURLOPT_FOLLOWLOCATION, TRUE  );
    curl_setopt( $curl, CURLOPT_TIMEOUT,        $timeout  );

    // THIS SEEMS TO LET IT WORK WITH HTTPS SITES
    curl_setopt( $curl, CURLOPT_SSL_VERIFYPEER, FALSE );

    // RUN THE CURL REQUEST AND GET THE RESULTS
    $htm = curl_exec($curl);

    // ON FAILURE HANDLE ERROR MESSAGE
    if ($htm === FALSE)
    {
        if ($error_report)
        {
            $err = curl_errno($curl);
            $inf = curl_getinfo($curl);
            echo "CURL FAIL: $url TIMEOUT=$timeout, CURL_ERRNO=$err";
            var_dump($inf);
        }
        curl_close($curl);
        return FALSE;
    }

    // ON SUCCESS RETURN XML / HTML STRING
    curl_close($curl);
    return $htm;
}




// USAGE EXAMPLE - PUT YOUR FAVORITE URL HERE
$url = "http://finance.yahoo.com/d/quotes.csv";

// PUT YOUR ARRAY OF KEY=>VALUE PAIRS HERE
$arg = array
( 's' => 'lulu'
, 'f' => 'snl1c1ohgvt1'
)
;

// MAKE THE CALL
$htm = my_curl($url, $arg, 2, TRUE);
if (!$htm) die("NO $url");

// SHOW WHAT WE GOT
echo "<pre>";
var_dump($arg);
echo PHP_EOL . $url;
echo PHP_EOL . htmlentities($htm);
echo PHP_EOL;




// TRY ANOTHER URL WITHOUT ARGUMENTS
$url = 'http://twitter.com';
$htm = my_curl($url);
echo PHP_EOL . $url;
echo PHP_EOL . htmlentities($htm);
echo PHP_EOL;

Open in new window

0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:Witheet
ID: 36900274
Thank you kindly for your responses... I will read the posts and let you know asap...

wkr
rv
0
 

Author Comment

by:Witheet
ID: 36900363
Again tyvm for yourfeedback... Looks complicated...
In the meanwhile here are two urls...
The 1 that displays the images was for trying it with another script of Ray Paseur I found on this site:
- http://www.boekbits.nl/uploads/audio/curl_import_array.php
- http://www.boekbits.nl/uploads/audio/curl_import_array2.php

Not sure what works better for you.. So I posted both...

WKR
RV
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 36900541
Actually you do not need to use CURL at all!
http://www.laprbass.com/RAY_temp_witheet.php

Run it on my server and use view source to see the image links, or just click the links to see the images.

Outputs links as follows:
http://www.boekbits.nl/uploads/4de6e88c8d5ac0.89314222.jpg IN WITHEET0 ON LAPRBASS
http://www.boekbits.nl/uploads/4de6e88d18ae52.04396940.jpg IN WITHEET1 ON LAPRBASS
http://www.boekbits.nl/uploads/4de6e88d41af78.34977938.jpg IN WITHEET2 ON LAPRBASS

HTH, ~Ray
<?php // RAY_temp_witheet.php
error_reporting(E_ALL);



// THE URL OF THE WEB PAGE WE WANT TO SCRAPE
$url = "http://www.boekbits.nl/uploads/audio/curl_import_array.php";

// READ THE HTML
$htm = file_get_contents($url);

// CREATE A REGEX TO EXTRACT THE IMAGE URLS
$rgx
= '#'              // REGEX DELIMITER
. '\b'             // WORD BOUNDARY
. '(https?)'       // GROUP OF PROTOCOL
. '(.*?)'          // GROUP OF ANYTHING OR NOTHING
. '(\.jpg)'        // GROUP OF IMAGE FILE SUFFIX
. '\b'             // WORD BOUNDARY
. '#'              // REGEX DELIMITER
. 'i'              // CASE-INSENSITIVE
;

// EXTRACT THE IMAGE URLS
preg_match_all($rgx, $htm, $mat);

// ITERATE OVER THE EXTRACTED URLS
foreach ($mat[0] as $key => $img)
{
    // READ THE IMAGE
    $dat = file_get_contents($img);

    // WRITE THE IMAGE ONTO OUR SERVER
    file_put_contents(getcwd() . "/RAY_junk/witheet$key.jpg", $dat);

    // PRESENT A LINK TO THE IMAGE
    echo "<br/><a target=\"_blank\" href=\"/RAY_junk/witheet$key.jpg\">$img IN WITHEET$key ON LAPRBASS</a>" . PHP_EOL;
}

Open in new window

0
 

Author Comment

by:Witheet
ID: 36900588
I have tried the script of hernst42 this does download more then 1 image, but the images were all 0kb. The script of ray paseur was way to complex for me to grasp at this point thats why I posted the urls...

But I kept trying and came up with the following.
I took the major part of the script I posted at first and some parts from this script provided by ray paseur to another user: http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_26364030.html?sfQueryTermInfo=1+10+30+curl+download+imag
I used the url that displays the images in a repeat region as the source and was able to import the images to server and in to the correct folder. It also appears the attachment header is no longer in place since the images show in chrome...

It resulted in the following script.
If this is properly formatted how hard is it to implement an image resize in this?

Tyvm for your time and patience...

wkr
rv

<?php
 $h = file_get_contents('http://www.xyz.nl/uploads/curl_import_array.php');
// GET THE IMAGE TAGS
preg_match_all('#<img[^>]+src=[\'"]([^\'"]+)#i', $h, $m);
$urls=array_unique($m[1]);
$save_to='path/goes/here/';
$mh = curl_multi_init();
foreach ($urls as $i => $url) {
    $g=$save_to.basename($url);
    if(!is_file($g)){
        $conn[$i]=curl_init($url);
        $fp[$i]=fopen ($g, "w");
        curl_setopt ($conn[$i], CURLOPT_FILE, $fp[$i]);
        curl_setopt ($conn[$i], CURLOPT_HEADER ,0);
        curl_setopt($conn[$i],CURLOPT_CONNECTTIMEOUT,60);
        curl_multi_add_handle ($mh,$conn[$i]);
    }
}
do {
    $n=curl_multi_exec($mh,$active);
}
while ($active);
foreach ($urls as $i => $url) {
    curl_multi_remove_handle($mh,$conn[$i]);
    curl_close($conn[$i]);
    fclose ($fp[$i]);
}
curl_multi_close($mh);
?>

Open in new window

0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 36900612
script of ray paseur was way to complex...

Sorry, it cannot be made simpler.  Did you test it?  It's there on my server waiting for you to click the link.  There are only four functions needed to make it work!  

Get yourself a couple of good books on PHP.  Here are two that I like.
http://www.sitepoint.com/books/phpmysql4/
http://www.amazon.com/PHP-5-Practice-Elliott-White/dp/0672328887

If you read these and internalize them it will not make you a professional, but it will give you a good enough foundation to understand PHP code when you see it.  You might also want to work through the introductory tutorial on the PHP.net site.
http://php.net/tut.php

You can look up any PHP function on the PHP.net web site.  Examples from the last snippet I posted might include these.
http://php.net/manual/en/function.file-get-contents.php
http://php.net/manual/en/function.preg-match-all.php
http://php.net/manual/en/function.getcwd.php
http://php.net/manual/en/function.file-put-contents.php
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 36900617
Sidebar note... Reading a file from a URL is an entirely different activity from resizing images, and you might want to post a separate question about that.  But if you think the script to read and copy the files was complex, wait till you see the image resize script.  The image manipulation functions are considered advanced PHP programming.  Fasten your seat belt!
0
 

Author Comment

by:Witheet
ID: 36900621
Yes I am trying it now...

regarding the complexity... I fully understand...

I have the sitepoint book... I believe you were the one to recommend me this book in another question I asked... I am sure to get the other one aswell...

I will let you know asap how it works out...

tyvm!
0
 
LVL 111

Expert Comment

by:Ray Paseur
ID: 36900648
The SitePoint book will help you get started -- it's PHP 101.  Eli White's book will move you up to intermediate level. more like PHP 201.  After that, you will need to just spend more time programming if you want a deeper understanding of the language.  I completely understand that time is money and some people will not want to spend their time being PHP programmers. But proficiency in PHP (or anything else) takes time and deliberate study.  The chief scientist of Google has this take on it.
http://norvig.com/21-days.html

Best regards, ~Ray
0
 

Author Comment

by:Witheet
ID: 36900691
Ive checked the script and tried it on my own server after having some trouble with the paths it works beautifully.
If I may bother you with one last thing though? How can I make it so it keeps the original filename of the downloaded image - is this possible in this fashion... This way the imagename will correspondent with the filename I have stored in my database...

///

I would really like to get a better grasp of php, but php.net is often very overwhelming and usually I manage allright with the basic understanding of the language... Every now and then I get some requests that go beyond my scope and when it does it goes way beyond... Putting me back in my place... but it is very difficult to take the time needed to enhance skills in php liek the article says... The lack of real life cases also makes it hard to find a starting point to take it to the next level... It's often like it goes from managable, readable scripts to the once like you posted... This all is very overwhelming but a welcome challenge none the less... And as far as it taking ten years... I've given up on those samms teach yourselve like books a long time ago...

TYVM for your time and help!

WKR
RV
0
 
LVL 111

Accepted Solution

by:
Ray Paseur earned 2000 total points
ID: 36900757
See if this makes sense for your needs.  Seems to work OK in my test case.  Cheers, ~Ray
<?php // RAY_temp_witheet.php
error_reporting(E_ALL);



// THE URL OF THE WEB PAGE WE WANT TO SCRAPE
$url = "http://www.boekbits.nl/uploads/audio/curl_import_array.php";

// READ THE HTML
$htm = file_get_contents($url);

// CREATE A REGEX TO EXTRACT THE IMAGE URLS
$rgx
= '#'              // REGEX DELIMITER
. '\b'             // WORD BOUNDARY
. '(https?)'       // GROUP OF PROTOCOL
. '(.*?)'          // GROUP OF ANYTHING OR NOTHING
. '(\.jpg)'        // GROUP OF IMAGE FILE SUFFIX
. '\b'             // WORD BOUNDARY
. '#'              // REGEX DELIMITER
. 'i'              // CASE-INSENSITIVE
;

// EXTRACT THE IMAGE URLS
preg_match_all($rgx, $htm, $mat);

// ITERATE OVER THE EXTRACTED URLS
foreach ($mat[0] as $key => $img)
{
    // READ THE IMAGE
    $dat = file_get_contents($img);

    // GET THE 'BASENAME' OF THE FILE NAME
    $nom = end(explode(DIRECTORY_SEPARATOR, $img));

    // WRITE THE IMAGE ONTO OUR SERVER
    file_put_contents(getcwd() . "/RAY_junk/$nom", $dat);

    // PRESENT A LINK TO THE IMAGE
    echo "<br/><a target=\"_blank\" href=\"/RAY_junk/$nom\">$img IN WITHEET $nom ON LAPRBASS</a>" . PHP_EOL;
}

Open in new window

0
 

Author Comment

by:Witheet
ID: 36900809
Again much appreciated.. I used str_replace to delete the url in front of the image names and after that it works exactly as I need it to...

Very grateful for this... This really helps me a great deal!

WKR
RV

0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

These days socially coordinated efforts have turned into a critical requirement for enterprises.
This holiday season, we’re giving away the gift of knowledge—tech knowledge, that is. Keep reading to see what hacks, tips, and trends we have wrapped and waiting for you under the tree.
The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…
This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
Suggested Courses

580 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question