Solved

Protecting Web Site Content

Posted on 2007-12-06
9
233 Views
Last Modified: 2013-12-25
Any suggestions on how to protect a web site from scrapers and other site capturing tools. I have reviewed a few products and methods, but wanted to get other thoughts.
0
Comment
Question by:tarrigo
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
9 Comments
 
LVL 7

Accepted Solution

by:
killbrad earned 250 total points
ID: 20421311
Nothing is really gonna work.  At best you will find a program that can rewrite your html into something that looks absolutely terrible, but the site will still look fine to 'scrapers'.  
0
 
LVL 7

Expert Comment

by:mcse2007
ID: 20426495
If you referring to encrypting the traffic when the data travels from the public network to the web server, SSL certificate will do this for you. Also to protect your server from scrapers block all ports in the firewall and only port 80 for http and port 443 for ssl that you need to open, if you are hosting your server.
0
 
LVL 18

Expert Comment

by:Hube02
ID: 20435864
check out this site: http://browsers.garykeith.com/

We have plans to put something similar in place. It requires adding a script to the beginning of every page (more likely an included file) that will check the data file from the above site against the browser that is requesting the page and not not send any content to the browsers listed as blocked in that file.

However, we are planning on doing this in PHP, and not ASP. There are several other tools available there.
0
Threat Trends for MSPs to Watch

See the findings.
Despite its humble beginnings, phishing has come a long way since those first crudely constructed emails. Today, phishing sites can appear and disappear in the length of a coffee break, and it takes more than a little know-how to keep your clients secure.

 
LVL 7

Expert Comment

by:killbrad
ID: 20457822
mcse2007:  Blocking other ports is not gonna stop someone from leeching his website with spider software.  
Hube:  It's a nice idea, but not realistic.  Many webleech programs give you the specific option to spoof the headers sent.  Also, trying to access the page you referenced above in Linux gives me this:

ACCESS DENIED
You do not appear to be using this form in accordance with my Terms of Use.
Continued abuse will eventually result in you losing access to this server!
It's also possible you are using security software that modifies the HTTP_REFERER header.
If I can't confirm the referrer is valid then you can't have access to my forms.

hmmm  :-/
0
 
LVL 18

Expert Comment

by:Hube02
ID: 20462435
killbrad:  In your case I would say that you have a firewall or are going through a proxy server that blocks or modifies the HTTP_REFERER header. (that isn't my site) but it is a security measure that I also implement on forms on sites I build. All forms submit to themselves and if the form gets a post request from somewhere other than itself it will not process the request. This effectively stops 90% of spam bots submitting forms. But it also blocks those that have firewall or proxy servers set up like yours.
0
 
LVL 7

Expert Comment

by:killbrad
ID: 20609078
nope, it just won't let me use elinks to access it.. tried it from several places.
0
 
LVL 1

Expert Comment

by:Computer101
ID: 21447597
Forced accept.

Computer101
EE Admin
0

Featured Post

Resolve Critical IT Incidents Fast

If your data, services or processes become compromised, your organization can suffer damage in just minutes and how fast you communicate during a major IT incident is everything. Learn how to immediately identify incidents & best practices to resolve them quickly and effectively.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When it comes to write a Context Sensitive Help (an online help that is obtained from a specific point in state of software to provide help with that state) ,  first we need to make the file that contains all topics, which are given exclusive IDs. …
If you thought ransomware was bad, think again! Doxware has the potential to be even more damaging.
This tutorial walks through the best practices in adding a local business to Google Maps including how to properly search for duplicates, marker placement, and inputing business details. Login to your Google Account, then search for "Google Mapmaker…
Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…

752 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question