automate webpage download

Posted on 2011-02-16
Medium Priority
Last Modified: 2012-05-11

I am trying to download a few hundred doctor profiles from Vitals.com. When I launch a URL link like the following:
I will be asked to first "Click here to vie the full report FREE", and then "I agree" with the terms of use. Then I will see the full profile page. Is there anyway to automate these two clicks in programming?

Your help is greatly appreciated.


Question by:gaog66
  • 2
LVL 32

Accepted Solution

aleghart earned 2000 total points
ID: 34908073
It's called screen scraping.  It's a clear violation of their TOU (Terms Of Use).  You know....that thing you agreed to when you clicked "I agree".

This part in particular:

...You will not copy, reproduce, modify, adapt, translate, distribute, transmit, download, upload, post, sell, rent, license, transfer, publicly perform, publicly display, mirror, frame, scrape, extract, wrap, create derivative works of, reverse engineer, decompile or disassemble any aspect of this Vitals or any of the Content, in whole or in part, in any form or by any means, whether manual or automatic


Information sites like this are designed for human interaction.  Scraping the data for re-use is unethical and unprofessional.  

Asking for, or providing advice for this activity is not allowed on Experts-Exchange.
LVL 32

Expert Comment

ID: 34910734
recommend delete.  simple TOU, no need for awarding pts.

Featured Post

Free Tool: Path Explorer

An intuitive utility to help find the CSS path to UI elements on a webpage. These paths are used frequently in a variety of front-end development and QA automation tasks.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

Join & Write a Comment

What You Need to Know when Searching for a Webhost Provider
Media Temple is proud to announce our partnership with the Society of Digital Agencies (SoDA) as their exclusive hosting partner.
This Micro Tutorial will demonstrate how nuggets on the Web are formatted by using Chrome Developer Tools. These tools would not only view the site's CSS but it can also modify it and save the CSS to use on your own site.
Shows how to create a shortcut to site-search Experts Exchange using Google in the Chrome browser. This eliminates the need to type out site:experts-exchange.com whenever you want to search the site. Launch the Search Engine Menu: In chrome, via you…
Suggested Courses

624 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question