Solved

automate webpage download

Posted on 2011-02-16
3
220 Views
Last Modified: 2012-05-11
Hi,

I am trying to download a few hundred doctor profiles from Vitals.com. When I launch a URL link like the following:
http://www.vitals.com/doctors/Dr_Robert_Laureno.html
I will be asked to first "Click here to vie the full report FREE", and then "I agree" with the terms of use. Then I will see the full profile page. Is there anyway to automate these two clicks in programming?

Your help is greatly appreciated.

Gordon


0
Comment
Question by:gaog66
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 32

Accepted Solution

by:
aleghart earned 500 total points
ID: 34908073
It's called screen scraping.  It's a clear violation of their TOU (Terms Of Use).  You know....that thing you agreed to when you clicked "I agree".

This part in particular:

...You will not copy, reproduce, modify, adapt, translate, distribute, transmit, download, upload, post, sell, rent, license, transfer, publicly perform, publicly display, mirror, frame, scrape, extract, wrap, create derivative works of, reverse engineer, decompile or disassemble any aspect of this Vitals or any of the Content, in whole or in part, in any form or by any means, whether manual or automatic


http://www.vitals.com/termsofuse


Information sites like this are designed for human interaction.  Scraping the data for re-use is unethical and unprofessional.  

Asking for, or providing advice for this activity is not allowed on Experts-Exchange.
0
 
LVL 32

Expert Comment

by:aleghart
ID: 34910734
mod,
recommend delete.  simple TOU, no need for awarding pts.
0

Featured Post

Optimum High-Definition Video Viewing and Control

The ATEN VM0404HA 4x4 4K HDMI Matrix Switch supports 4K resolutions of UHD (3840 x 2160) and DCI (4096 x 2160) with refresh rates of 30 Hz (4:4:4) and 60 Hz (4:2:0). It is ideal for applications where the routing of 4K digital signals is required.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

SSL stands for “Secure Sockets Layer” and an SSL certificate is a critical component to keeping your website safe, secured, and compliant. Any ecommerce website must have an SSL certificate to ensure the safe handling of sensitive information like…
Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
Google currently has a new report that is in beta and coming soon to Webmaster Tool accounts. This Micro Tutorial will highlight new features for Google Webmaster Tools.
This Micro Tutorial will demonstrate how nuggets on the Web are formatted by using Chrome Developer Tools. These tools would not only view the site's CSS but it can also modify it and save the CSS to use on your own site.

631 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question