Still celebrating National IT Professionals Day with 3 months of free Premium Membership. Use Code ITDAY17

x
?
Solved

automate webpage download

Posted on 2011-02-16
3
Medium Priority
?
221 Views
Last Modified: 2012-05-11
Hi,

I am trying to download a few hundred doctor profiles from Vitals.com. When I launch a URL link like the following:
http://www.vitals.com/doctors/Dr_Robert_Laureno.html
I will be asked to first "Click here to vie the full report FREE", and then "I agree" with the terms of use. Then I will see the full profile page. Is there anyway to automate these two clicks in programming?

Your help is greatly appreciated.

Gordon


0
Comment
Question by:gaog66
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 2
3 Comments
 
LVL 32

Accepted Solution

by:
aleghart earned 2000 total points
ID: 34908073
It's called screen scraping.  It's a clear violation of their TOU (Terms Of Use).  You know....that thing you agreed to when you clicked "I agree".

This part in particular:

...You will not copy, reproduce, modify, adapt, translate, distribute, transmit, download, upload, post, sell, rent, license, transfer, publicly perform, publicly display, mirror, frame, scrape, extract, wrap, create derivative works of, reverse engineer, decompile or disassemble any aspect of this Vitals or any of the Content, in whole or in part, in any form or by any means, whether manual or automatic


http://www.vitals.com/termsofuse


Information sites like this are designed for human interaction.  Scraping the data for re-use is unethical and unprofessional.  

Asking for, or providing advice for this activity is not allowed on Experts-Exchange.
0
 
LVL 32

Expert Comment

by:aleghart
ID: 34910734
mod,
recommend delete.  simple TOU, no need for awarding pts.
0

Featured Post

Moving data to the cloud? Find out if you’re ready

Before moving to the cloud, it is important to carefully define your db needs, plan for the migration & understand prod. environment. This wp explains how to define what you need from a cloud provider, plan for the migration & what putting a cloud solution into practice entails.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

SSL stands for “Secure Sockets Layer” and an SSL certificate is a critical component to keeping your website safe, secured, and compliant. Any ecommerce website must have an SSL certificate to ensure the safe handling of sensitive information like…
Ever visit a website where you spotted a really cool looking Font, yet couldn't figure out which font family it belonged to, or how to get a copy of it for your own use? This article explains the process of doing exactly that, as well as showing how…
Google currently has a new report that is in beta and coming soon to Webmaster Tool accounts. This Micro Tutorial will highlight new features for Google Webmaster Tools.
How to create a custom search shortcut to site-search Experts Exchange using Google in the Firefox browser. This eliminates the need to type out site:experts-exchange.com whenever you want to search the site. Launch your Bookmark Menu: Press 'Ctrl +…
Suggested Courses

715 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question