troubleshooting Question

How is Googlebot finding my script?

Avatar of tel2
tel2Flag for New Zealand asked on
Apache Web ServerSecurityWeb Servers
13 Comments1 Solution554 ViewsLast Modified:
Hi Experts,

I've had a Perl script called mailer.pl (which I copied from www.jmarshall.com/easy/cgi/mailer.pl.txt), sitting in my public_html/XXX directory for years.  If I browse to:
    www.mydomain.com/XXX/mailer.pl    # That's not the real URL
then I get an email telling me the HTTP_REFERER.  Fine so far.

Starting from 1 July 2013, I started getting emails from this script, every 48 hours (approx).  I didn't know anyone could find the script, as my .htaccess file has indexing turned off.  If I browse to www.mydomain.com/XXX I get a "Forbidden" - "404 Not Found" error, as I would expect.  The referer wasn't specified in the emails, so I changed the script to show the script name, the REMOTE_ADDR and the HTTP_USER_AGENT, and here's what came in the next 2 emails I received from that script:
  "The following data was entered at XXX/mailer.pl: 66.249.75.128=Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html):"

A whois on that IP indicates that it is 'Google Inc.', as expected.

Here's my .htaccess file from the ~/public_html directory:
    Options -Indexes
    AddHandler cgi-script .pl .cgi .pm
    SetEnv TZ NZ
    <Files 403.shtml>
    order allow,deny
    allow from all
    </Files>

And here's my .htaccess file from the ~/public_html/XXX directory:
    AddHandler server-parsed .html .htm
    <Files 403.shtml>
    order allow,deny
    allow from all
    </Files>

The about .htaccess files have not changed for a long time.

Q1. How could Google's spider be finding my copy of this mailer.pl script?
Q2. How should I stop such things from happening?

I'm using a webhost which runs Linux, Apache and cPanel.

Thanks.
tel2
Join the community to see this answer!
Join our exclusive community to see this answer & millions of others.
Unlock 1 Answer and 13 Comments.
Join the Community
Learn from the best

Network and collaborate with thousands of CTOs, CISOs, and IT Pros rooting for you and your success.

Andrew Hancock - VMware vExpert
See if this solution works for you by signing up for a 7 day free trial.
Unlock 1 Answer and 13 Comments.
Try for 7 days

”The time we save is the biggest benefit of E-E to our team. What could take multiple guys 2 hours or more each to find is accessed in around 15 minutes on Experts Exchange.

-Mike Kapnisakis, Warner Bros