Solved

Best Web scrapping software?

Posted on 2013-01-15
6
542 Views
Last Modified: 2013-01-18
What is the best Web Scrapping software?  I would like to scrap the details  available (in tabular form) on the links (around 40 links) of the following website.

http://www.amfiindia.com/amfimembers.aspx

Could you please help me understand the best software?

Thanks,
--Anand
0
Comment
Question by:FTbridge
  • 3
  • 2
6 Comments
 
LVL 27

Expert Comment

by:Lukasz Chmielewski
ID: 38781893
There is no best, it depend on what you want. If you're into programming, you can write you own scrapper, if not - you can take a look at this links:

http://www.poynter.org/how-tos/digital-strategies/e-media-tidbits/102589/how-to-scrape-websites-for-data-without-programming-skills/

http://blog.outwit.com/?p=55
0
 
LVL 109

Expert Comment

by:Ray Paseur
ID: 38782236
The information in that web site is © Association of Mutual Funds in India (AMFI) - Copyright 2013.  You need to get their permission to copy, store or repurpose the content.  But that aside, if you can post the exact URLs of one or two of the pages you want to extract and post an example of the data you're trying to get I will be able to show you how to use a PHP script to get this information.
0
 

Author Comment

by:FTbridge
ID: 38782483
Thank you Ray_Paseur and Roads_Roads.

Ray_Paseur,
Here is what I want to do: On the following link there are links for several mutual funds on this page.
http://www.amfiindia.com/amfimembers.aspx

If go to the above URL you will see following fund as the first link:
-  BOI AXA Investment Managers Private Limited
If you click on this text you will see following page
http://www.amfiindia.com/amfiMembers.aspx?mfid=46

I would like to download name of the MF, Address and phone number for all the funds listed on the first page.

I am not going to sell or repurpose the contents. I just want to sort these mutual funds by the address. Thank you for your help in advance.
0
Free Tool: IP Lookup

Get more info about an IP address or domain name, such as organization, abuse contacts and geolocation.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

 
LVL 109

Accepted Solution

by:
Ray Paseur earned 500 total points
ID: 38790192
This takes a while to run, so please be patient.  
http://www.laprbass.com/RAY_temp_ftbridge.php

You may want to normalize the address a little bit before you sort (but there are only 40+ data elements so it should be straightforward).

<?php // RAY_temp_ftbridge.php
error_reporting(E_ALL);
echo '<pre>';

// THE URLS AND GETTING THE HTML DOCUMENT
$bas = 'http://www.amfiindia.com/';
$url = $bas . 'amfimembers.aspx';
$htm = file_get_contents($url);

// A SIGNAL STRING TO DECLOP THE HTML
$sig = '<a href="amfiMembers.aspx?mfid=';

// BREAK AND MANIPULATE
$arr = explode($sig, $htm);
unset($arr[0]);
foreach ($arr as $key => $str)
{
    $new = $sig . $str;
    $sub = explode('"', $new);
    $arr[$key] = trim($bas . $sub[1]);
}

// ACTIVATE THIS TO SEE THE LIST OF URLS
// print_r($arr);

// ITERATE OVER THE LINKS
$objs = array();
foreach ($arr as $lnk)
{
    // CREATE AN OBJECT TO HOLD THIS DATA AND READ THE PAGE
    $obj = new StdClass;
    $htm = file_get_contents($lnk);

    // ISOLATE THE NAME
    $sub = explode('Name of the Mutual Fund</td>', $htm);
    $sub = explode('</td>', $sub[1]);
    $sub = strip_tags($sub[0]);
    $obj->nam = $sub;

    // ISOLATE THE ADDRESS
    $sub = explode('Address of AMC</td>', $htm);
    $sub = explode('</td>', $sub[1]);
    $sub = strip_tags($sub[0]);
    $obj->add = $sub;

    // ISOLATE THE PHONE
    $sub = explode('Telephone Number</td>', $htm);
    $sub = explode('</td>', $sub[1]);
    $sub = strip_tags($sub[0]);
    $obj->fon = $sub;

    // ADD THIS OBJECT TO OUR ARRAY
    $objs[] = $obj;
}

// SHOW THE ACQUIRED DATA IN THE ARRAY OF OBJECTS
print_r($objs);

Open in new window

HTH, ~Ray
0
 

Author Closing Comment

by:FTbridge
ID: 38792070
Ray,

This is excellent! Thank you very much for your help on this! You have saved my time!

Best regards,
--Anand
0
 
LVL 109

Expert Comment

by:Ray Paseur
ID: 38793395
Thanks for the points and thanks for using EE, ~Ray
0

Featured Post

Free Tool: SSL Checker

Scans your site and returns information about your SSL implementation and certificate. Helpful for debugging and validating your SSL configuration.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

#Citrix #Internet Explorer #Enterprise Mode #IE 11 #IE 8
This article offers some helpful and general tips for safe browsing and online shopping. It offers simple and manageable procedures that help to ensure the safety of one's personal information and the security of any devices.
The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …
This Micro Tutorial will demonstrate how nuggets on the Web are formatted by using Chrome Developer Tools. These tools would not only view the site's CSS but it can also modify it and save the CSS to use on your own site.

791 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question