Solved

Best Web scrapping software?

Posted on 2013-01-15
6
538 Views
Last Modified: 2013-01-18
What is the best Web Scrapping software?  I would like to scrap the details  available (in tabular form) on the links (around 40 links) of the following website.

http://www.amfiindia.com/amfimembers.aspx

Could you please help me understand the best software?

Thanks,
--Anand
0
Comment
Question by:FTbridge
  • 3
  • 2
6 Comments
 
LVL 27

Expert Comment

by:Lukasz Chmielewski
ID: 38781893
There is no best, it depend on what you want. If you're into programming, you can write you own scrapper, if not - you can take a look at this links:

http://www.poynter.org/how-tos/digital-strategies/e-media-tidbits/102589/how-to-scrape-websites-for-data-without-programming-skills/

http://blog.outwit.com/?p=55
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 38782236
The information in that web site is © Association of Mutual Funds in India (AMFI) - Copyright 2013.  You need to get their permission to copy, store or repurpose the content.  But that aside, if you can post the exact URLs of one or two of the pages you want to extract and post an example of the data you're trying to get I will be able to show you how to use a PHP script to get this information.
0
 

Author Comment

by:FTbridge
ID: 38782483
Thank you Ray_Paseur and Roads_Roads.

Ray_Paseur,
Here is what I want to do: On the following link there are links for several mutual funds on this page.
http://www.amfiindia.com/amfimembers.aspx

If go to the above URL you will see following fund as the first link:
-  BOI AXA Investment Managers Private Limited
If you click on this text you will see following page
http://www.amfiindia.com/amfiMembers.aspx?mfid=46

I would like to download name of the MF, Address and phone number for all the funds listed on the first page.

I am not going to sell or repurpose the contents. I just want to sort these mutual funds by the address. Thank you for your help in advance.
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 108

Accepted Solution

by:
Ray Paseur earned 500 total points
ID: 38790192
This takes a while to run, so please be patient.  
http://www.laprbass.com/RAY_temp_ftbridge.php

You may want to normalize the address a little bit before you sort (but there are only 40+ data elements so it should be straightforward).

<?php // RAY_temp_ftbridge.php
error_reporting(E_ALL);
echo '<pre>';

// THE URLS AND GETTING THE HTML DOCUMENT
$bas = 'http://www.amfiindia.com/';
$url = $bas . 'amfimembers.aspx';
$htm = file_get_contents($url);

// A SIGNAL STRING TO DECLOP THE HTML
$sig = '<a href="amfiMembers.aspx?mfid=';

// BREAK AND MANIPULATE
$arr = explode($sig, $htm);
unset($arr[0]);
foreach ($arr as $key => $str)
{
    $new = $sig . $str;
    $sub = explode('"', $new);
    $arr[$key] = trim($bas . $sub[1]);
}

// ACTIVATE THIS TO SEE THE LIST OF URLS
// print_r($arr);

// ITERATE OVER THE LINKS
$objs = array();
foreach ($arr as $lnk)
{
    // CREATE AN OBJECT TO HOLD THIS DATA AND READ THE PAGE
    $obj = new StdClass;
    $htm = file_get_contents($lnk);

    // ISOLATE THE NAME
    $sub = explode('Name of the Mutual Fund</td>', $htm);
    $sub = explode('</td>', $sub[1]);
    $sub = strip_tags($sub[0]);
    $obj->nam = $sub;

    // ISOLATE THE ADDRESS
    $sub = explode('Address of AMC</td>', $htm);
    $sub = explode('</td>', $sub[1]);
    $sub = strip_tags($sub[0]);
    $obj->add = $sub;

    // ISOLATE THE PHONE
    $sub = explode('Telephone Number</td>', $htm);
    $sub = explode('</td>', $sub[1]);
    $sub = strip_tags($sub[0]);
    $obj->fon = $sub;

    // ADD THIS OBJECT TO OUR ARRAY
    $objs[] = $obj;
}

// SHOW THE ACQUIRED DATA IN THE ARRAY OF OBJECTS
print_r($objs);

Open in new window

HTH, ~Ray
0
 

Author Closing Comment

by:FTbridge
ID: 38792070
Ray,

This is excellent! Thank you very much for your help on this! You have saved my time!

Best regards,
--Anand
0
 
LVL 108

Expert Comment

by:Ray Paseur
ID: 38793395
Thanks for the points and thanks for using EE, ~Ray
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

The password reset disk is often mentioned as the best solution to deal with the lost Windows password problem. In Windows 2008, 7, Vista and XP, a password reset disk can be easily created. But besides Windows 7/Vista/XP, Windows Server 2008 and ot…
Since pre-biblical times, humans have sought ways to keep secrets, and share the secrets selectively.  This article explores the ways PHP can be used to hide and encrypt information.
Learn how to match and substitute tagged data using PHP regular expressions. Demonstrated on Windows 7, but also applies to other operating systems. Demonstrated technique applies to PHP (all versions) and Firefox, but very similar techniques will w…
With the advent of Windows 10, Microsoft is pushing a Get Windows 10 icon into the notification area (system tray) of qualifying computers. There are many reasons for wanting to remove this icon. This two-part Experts Exchange video Micro Tutorial s…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

17 Experts available now in Live!

Get 1:1 Help Now