Grab the search results from a website using curl.

Posted on 2005-04-19
Last Modified: 2012-08-14
Hello everybody:

How would I use curl to grab the search results from a website like I would like a command line script if possible. I want just the returned results (the text), not the html. Also, I would like to get each news item one at a time, not all as one html page. That way, I could write individual news items to a file or post them to a blog. Also is there a way to automatically follow the link to the next page of the search results to continue the download? Thanks.
Question by:jmcnealy1
    LVL 3

    Expert Comment

    try using this to grab a page using curl

    Before using curl, it must be installed.

    $URL = "";
    $string = `/usr/bin/curl $URL`; // note ' not single quote it is tilde ` previous key to 1
    echo ($string);

    $string variable contains the whole webpage.

    You have to write a code parse the required news there after.

    LVL 3

    Accepted Solution

    $URL = "";
    $string = `/usr/bin/curl $URL`;
    //echo ($string);

    // strip only anchor tags
    $data = strip_tags($string,"<a>");

    // do a regular expression search to match the news items. result will be stored in $return
    $data = preg_match_all("/<a[^>].*?item_id=.*?>(.*?)<\/a>/si",$data,$return);

    // here is your item. chioce to store into db or flat file. it has only news title
    for ($i=0; $i<sizeof($return[1]); $i++) {
          echo $i.". ".$return[1][$i]."<br>";

    LVL 25

    Assisted Solution

    You can get results from there using RSS (since that's what the site is all about). RSS is far more reliable easier to handle than attempting to parse a rendered page that could change at any time. All you need to do is URL encode your search term and append it to this URL:

    so a search for 'experts exchange' would be:

    You don't need to use CURL at all - there are plenty of RSS PHP clients that make all this trivial, in particular:

    Once you have the RSS feed contents, you can extract as few or as many as you like, and you can choose to parse their content for links (note that if you simply remove HTML tags, all your links will disappear too). You might like to use an HTML parser to strip HTML formatting more carefully:
    LVL 3

    Expert Comment

    I do agree that HTML parser is always not trustable. Because if there is a change in the page, then we have to code it again.

    But in the case of RSS, we do not need to worry. The structures are standard. We can play with it.

    Author Comment

    Thanks alot for the help. The point about just using rss as it is was a good one. I'll work on that in the future.

    Write Comment

    Please enter a first name

    Please enter a last name

    We will never share this with anyone.

    Featured Post

    How your wiki can always stay up-to-date

    Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
    - Increase transparency
    - Onboard new hires faster
    - Access from mobile/offline

    Foreword (July, 2015) Since I first wrote this article, years ago, a great many more people have begun using the internet.  They are coming online from every part of the globe, learning, reading, shopping and spending money at an ever-increasing ra…
    Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
    Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
    The viewer will learn how to count occurrences of each item in an array.

    761 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    11 Experts available now in Live!

    Get 1:1 Help Now