Solved

automate webpage download

Posted on 2011-02-16
3
213 Views
Last Modified: 2012-05-11
Hi,

I am trying to download a few hundred doctor profiles from Vitals.com. When I launch a URL link like the following:
http://www.vitals.com/doctors/Dr_Robert_Laureno.html
I will be asked to first "Click here to vie the full report FREE", and then "I agree" with the terms of use. Then I will see the full profile page. Is there anyway to automate these two clicks in programming?

Your help is greatly appreciated.

Gordon


0
Comment
Question by:gaog66
  • 2
3 Comments
 
LVL 32

Accepted Solution

by:
aleghart earned 500 total points
ID: 34908073
It's called screen scraping.  It's a clear violation of their TOU (Terms Of Use).  You know....that thing you agreed to when you clicked "I agree".

This part in particular:

...You will not copy, reproduce, modify, adapt, translate, distribute, transmit, download, upload, post, sell, rent, license, transfer, publicly perform, publicly display, mirror, frame, scrape, extract, wrap, create derivative works of, reverse engineer, decompile or disassemble any aspect of this Vitals or any of the Content, in whole or in part, in any form or by any means, whether manual or automatic


http://www.vitals.com/termsofuse


Information sites like this are designed for human interaction.  Scraping the data for re-use is unethical and unprofessional.  

Asking for, or providing advice for this activity is not allowed on Experts-Exchange.
0
 
LVL 32

Expert Comment

by:aleghart
ID: 34910734
mod,
recommend delete.  simple TOU, no need for awarding pts.
0

Featured Post

How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

Join & Write a Comment

Suggested Solutions

Title # Comments Views Activity
Java 1603 Error 2 42
How to download latest Internet Explorer? 15 31
Web Browsers Start Page Hijacker 14 76
IPhone using PC internet 17 45
Introduction This article explores the design of a cache system that can improve the performance of a web site or web application.  The assumption is that the web site has many more “read” operations than “write” operations (this is commonly the ca…
When it comes to showing a 404 error page to your visitors, you do not want that generic page to show, and you especially do not want your hosting provider’s ad error page to show either. In this article, I will show you how to enable the custom 40…
This Micro Tutorial will demonstrate how nuggets on the Web are formatted by using Chrome Developer Tools. These tools would not only view the site's CSS but it can also modify it and save the CSS to use on your own site.
How to create a custom search shortcut to site-search Experts Exchange using Google in the Firefox browser. This eliminates the need to type out site:experts-exchange.com whenever you want to search the site. Launch your Bookmark Menu: Press 'Ctrl +…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

18 Experts available now in Live!

Get 1:1 Help Now