Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people, just like you, are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
Solved

development site is indexed by google even though behind htpasswd

Posted on 2016-11-11
7
69 Views
Last Modified: 2016-11-15
As stated in the title. My development site is protected with an htpasswd, yet somehow google is indexing it. how is that possible?
0
Comment
Question by:jblayney
  • 3
  • 2
  • 2
7 Comments
 
LVL 13

Expert Comment

by:Phil Phillips
ID: 41884313
htpasswd should prevent Google from accessing the site, though you might want to double check that you have the config set up to lock down all URLs. In the web logs, do you still see Google accessing it with successful response codes?

Also, it could have just indexed a previous version. If that's the case, it'll take some time for it to age out.  To speed up the process, you can login to Google webmaster tools and remove the site from the index.
0
 
LVL 26

Expert Comment

by:Dr. Klahn
ID: 41884320
Update robots.txt to exclude all robots on that site:

User-agent: *
Disallow: /

Open in new window


I've noticed that Googlebot occasionally visits my sites with faked browser credentials to avoid complying with robots.txt, often enough that I wrote a mod_rewrite rule to block the behavior. So I'd also throw in an exclusion rule for the Googlebot IP ranges:

# Googlebot's various /24 blocks
66.249.64.0/22
66.249.69.0/24
66.249.73.0/24
66.249.79.0/24

Open in new window

0
 
LVL 1

Author Comment

by:jblayney
ID: 41886558
thanks for responding, you mean this?

Order Allow,Deny
Deny from 66.249.64.0/22
Deny from 66.249.69.0/24
Deny from 66.249.73.0/24
Deny from 66.249.79.0/24
Allow from all

Open in new window

0
Back Up Your Microsoft Windows Server®

Back up all your Microsoft Windows Server – on-premises, in remote locations, in private and hybrid clouds. Your entire Windows Server will be backed up in one easy step with patented, block-level disk imaging. We achieve RTOs (recovery time objectives) as low as 15 seconds.

 
LVL 1

Author Comment

by:jblayney
ID: 41886559
Phil,

where do I do this?
In the web logs, do you still see Google accessing it with successful response codes?
0
 
LVL 13

Assisted Solution

by:Phil Phillips
Phil Phillips earned 250 total points
ID: 41886654
It depends how you have Apache configured to store your logs. If you're on Linux, a common default place is: /var/log/httpd or /var/log/apache2
0
 
LVL 26

Accepted Solution

by:
Dr. Klahn earned 250 total points
ID: 41887051
thanks for responding, you mean this?

(... list of IP blocks)

That's the one.  It can be done in the main config file or in an htaccess file.  If you don't want googlebot in the site at all, it is more efficient to do it in iptables which blocks the request before it gets to Apache.
0
 
LVL 1

Author Closing Comment

by:jblayney
ID: 41888001
thank you
0

Featured Post

Free Tool: Postgres Monitoring System

A PHP and Perl based system to collect and display usage statistics from PostgreSQL databases.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

If you are running a LAMP infrastructure, this little code snippet is very helpful if you are serving lots of HTML, JavaScript and CSS-related information. The mod_deflate module, which is part of the Apache 2.2 application, provides the DEFLATE…
The new Gmail Phishing Scam going around is surprising even the savviest of users with its sophisticated techniques. This attack comes as a nightmare trifecta for email filtering services; sent from a familiar contact, using authentic tone and verbi…
Along with being a a promotional video for my three-day Annielytics Dashboard Seminor, this Micro Tutorial is an intro to Google Analytics API data.
Many of my clients call in with monstrous Gmail overloading issues with Outlook. A quick tip is to turn off the All Mail and Important folders from synching. Here is a quick video I made to show you how to turn off these and other folders in Gmail s…

856 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question