troubleshooting Question

How is Googlebot finding my script?

Avatar of tel2
tel2Flag for New Zealand asked on
Apache Web ServerWeb ServersSecurity
13 Comments1 Solution554 ViewsLast Modified:
Hi Experts,

I've had a Perl script called (which I copied from, sitting in my public_html/XXX directory for years.  If I browse to:    # That's not the real URL
then I get an email telling me the HTTP_REFERER.  Fine so far.

Starting from 1 July 2013, I started getting emails from this script, every 48 hours (approx).  I didn't know anyone could find the script, as my .htaccess file has indexing turned off.  If I browse to I get a "Forbidden" - "404 Not Found" error, as I would expect.  The referer wasn't specified in the emails, so I changed the script to show the script name, the REMOTE_ADDR and the HTTP_USER_AGENT, and here's what came in the next 2 emails I received from that script:
  "The following data was entered at XXX/ (compatible; Googlebot/2.1; +"

A whois on that IP indicates that it is 'Google Inc.', as expected.

Here's my .htaccess file from the ~/public_html directory:
    Options -Indexes
    AddHandler cgi-script .pl .cgi .pm
    SetEnv TZ NZ
    <Files 403.shtml>
    order allow,deny
    allow from all

And here's my .htaccess file from the ~/public_html/XXX directory:
    AddHandler server-parsed .html .htm
    <Files 403.shtml>
    order allow,deny
    allow from all

The about .htaccess files have not changed for a long time.

Q1. How could Google's spider be finding my copy of this script?
Q2. How should I stop such things from happening?

I'm using a webhost which runs Linux, Apache and cPanel.

Join our community to see this answer!
Unlock 1 Answer and 13 Comments.
Start Free Trial
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 13 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros