Solved

Web Scrape Javascript Site

Posted on 2011-09-25
3
421 Views
Last Modified: 2012-09-27
I have never done web scraping before, but have a situation where it would help a ton.  I go to a site that runs a javascript to display data.  I use Chrome and right click the page, and view page source and completely understand the page and can see the data that I want to grab.  I want to do 2 things; play a sound or some notification when a certain number is reached and store data for analysis.  I have done research and found many tools like greasemonkey for firefox, AJAX,  irobotsoft, etc.  I am looking for guidance as to what would be the best way to do this.  I have no problem leaving the site open and logged in while the scraping would be done.
0
Comment
Question by:etechit
3 Comments
 
LVL 82

Expert Comment

by:leakim971
ID: 36601110
0
 
LVL 1

Author Comment

by:etechit
ID: 36602475
Thank you, but the site I am looking at is javascript and these articles don't refer to that at all
0
 
LVL 17

Accepted Solution

by:
selvol earned 500 total points
ID: 38357259
Im going to make it short but hopefully meaningful.....

The 1 thing I do not Understand is the site runs a "JAVASCRIPT" to display.......
That can be a couple of things...


I use and have used Metaproducts Offline Explorer for scraping for 10+ Years,
It can Login, scrape Parts of the page, ajax and many other features.
It is developed by One of the Best developers Code wise and Support wise I've seen.


Please don't think I am just tossing some cheap, lame down loader at you because I used it once.
I am not sure of your exact needs. But from what I read it looks like OE can assist.

Give it a try. If you have any doubts or questions ask me here.

Or go to the Metaproducts forum page.  And ask the actual developer. He usually responds fast.

Selvol



0

Featured Post

What Is Threat Intelligence?

Threat intelligence is often discussed, but rarely understood. Starting with a precise definition, along with clear business goals, is essential.

Join & Write a Comment

Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
Nothing in an HTTP request can be trusted, including HTTP headers and form data.  A form token is a tool that can be used to guard against request forgeries (CSRF).  This article shows an improved approach to form tokens, making it more difficult to…
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

20 Experts available now in Live!

Get 1:1 Help Now