Web scraping - the best tools for the job

Posted on 2005-04-26
Last Modified: 2013-12-16
Im trying to research what the best web language to use would be to develop a web scraping tool. The tool should scrape employment sites only and lets assume that the developer only has a fair working knowledge of  Coldfusion, and PHP technologies.

Or if anyone has any good resources on the subject then they would also be welcome.

Thanks in advance
Question by:VHSB
    LVL 15

    Expert Comment

    by:VincentPuglia got me...what's "web scraping"?

    Author Comment

    web scraping or screen scraping, its when you automatically parse content from web pages (as long as they are your pages, or you have the permission of the target website). The pages are normally retrieved with http calls etc, are you with me?
    LVL 15

    Expert Comment


    I could be wrong, but I doubt if you will find a ready-made product to suit your needs, primarily because the data you want to 'scrape' is going to be displayed & stored differently from site to site.  Unless, of course, all you want to do is save an image of the screen itself, in which case there are more than a number of screen capture programs out there.

    If you wish to save the data, however, you will need to extract the pertinent data out of the page's source -- not an easy task, given the caveat stated above.

    That having been said, I would assume ASP or PHP would equally usable, even javascript could be used to some extent.

    LVL 29

    Expert Comment

    I use the XMLHTTP object which is becoming quite ubiquitous.

    Dim objH, str

    ' Create an xmlhttp object:
    Set objH = Server.CreateObject("Microsoft.XMLHTTP")
    objH.Open "GET", "", False
    ' Send the request and return the data:

    str = objH.responseBody
    Set objH = Nothing

    str now has the HTML from the site which you can easily parse.


    Author Comment

    Thanks guys, I should have already said that I have already developed a web scraping tool in Coldfusion, and a tedious task it was too. It can be broken easily due to the constant changes in the html and it isnt very flexible i.e it cant scrape many sites effectively.
    However I wanted to know if and how, other languages may be more suitable for the task of web scraping in the future. As .net technology is becoming more popular i thought that would be a good place to start.

    LVL 29

    Accepted Solution

    There are some commercial screen scrapers, although I never use them.

    As far as breakage goes, that is par for the course, unless you can key in on an ID or other tag that remains relatively static when the pages are redesigned.

    That is why they came up with RSS...

    Featured Post

    Maximize Your Threat Intelligence Reporting

    Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

    Join & Write a Comment

    Suggested Solutions

    Title # Comments Views Activity
    Website authentication - 3 58
    Using YubiKey with REST API application 2 54
    buying a domain 9 45
    Pick image, show data 4 26
    Deprecated and Headed for the Dustbin By now, you have probably heard that some PHP features, while convenient, can also cause PHP security problems.  This article discusses one of those, called register_globals.  It is a thing you do not want.  …
    Using SQL Scripts we can save all the SQL queries as files that we use very frequently on our database later point of time. This is one of the feature present under SQL Workshop in Oracle Application Express.
    The purpose of this video is to demonstrate how to add AdSense Ads to a WordPress Website, and how to set up WordPress to automatically place Ads in Sidebars. This will be demonstrated using a Windows 8 PC. Log into your AdSense account. : Cli…
    The viewer will learn how to create and use a small PHP class to apply a watermark to an image. This video shows the viewer the setup for the PHP watermark as well as important coding language. Continue to Part 2 to learn the core code used in creat…

    754 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    20 Experts available now in Live!

    Get 1:1 Help Now