Solved

automate webpage download

Posted on 2011-02-16
3
219 Views
Last Modified: 2012-05-11
Hi,

I am trying to download a few hundred doctor profiles from Vitals.com. When I launch a URL link like the following:
http://www.vitals.com/doctors/Dr_Robert_Laureno.html
I will be asked to first "Click here to vie the full report FREE", and then "I agree" with the terms of use. Then I will see the full profile page. Is there anyway to automate these two clicks in programming?

Your help is greatly appreciated.

Gordon


0
Comment
Question by:gaog66
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 32

Accepted Solution

by:
aleghart earned 500 total points
ID: 34908073
It's called screen scraping.  It's a clear violation of their TOU (Terms Of Use).  You know....that thing you agreed to when you clicked "I agree".

This part in particular:

...You will not copy, reproduce, modify, adapt, translate, distribute, transmit, download, upload, post, sell, rent, license, transfer, publicly perform, publicly display, mirror, frame, scrape, extract, wrap, create derivative works of, reverse engineer, decompile or disassemble any aspect of this Vitals or any of the Content, in whole or in part, in any form or by any means, whether manual or automatic


http://www.vitals.com/termsofuse


Information sites like this are designed for human interaction.  Scraping the data for re-use is unethical and unprofessional.  

Asking for, or providing advice for this activity is not allowed on Experts-Exchange.
0
 
LVL 32

Expert Comment

by:aleghart
ID: 34910734
mod,
recommend delete.  simple TOU, no need for awarding pts.
0

Featured Post

Business Impact of IT Communications

What are the business impacts of how well businesses communicate during an IT incident? Targeting, speed, and transparency all matter. Find out more in this infographic.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Periodically we have to update or add SSL certificates for customers. Depending upon your hosting plan you may be responsible for the installation and/or key generation. In the wake of Heartbleed many sites were forced to re-key. We will concen…
Do you come here a lot? Are you lazy like me and don't want to go through the "trouble" of having to click your Dock's Safari icon and then having to click your Experts Exchange Favorites bookmark to get here? Well then this article is for you.
This Micro Tutorial will demonstrate how nuggets on the Web are formatted by using Chrome Developer Tools. These tools would not only view the site's CSS but it can also modify it and save the CSS to use on your own site.
How to create a custom search shortcut to site-search Experts Exchange using Google in the Firefox browser. This eliminates the need to type out site:experts-exchange.com whenever you want to search the site. Launch your Bookmark Menu: Press 'Ctrl +…

751 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question