Solved

robots.txt

Posted on 2001-06-08
6
188 Views
Last Modified: 2006-11-17
I found the following entry in a log file on our server.  The 62.116.37.249 address is completely unknown to me, can anyone please tell me what has happened? (I have changed our address).
Regards
62.116.37.249, -, 6/6/01, 3:03:15, W3SVC1, XXXSERVER, 195.123.123.123, 15, 84, 669, 403, 5, GET, /robots.txt, -,
0
Comment
Question by:Chris_m
6 Comments
 
LVL 19

Expert Comment

by:DreamMaster
ID: 6167365
Probably...and I'm almost certain of this, this means that you have been visited by a search engine's bot...

Cheers,
Max.
0
 

Author Comment

by:Chris_m
ID: 6167469
I suspected that that was the case -- especially when I saw the robots part of the entry, but if that is so, how?  I have shut our server down to all computers except our internal lan.  I know that if I try to access it from outside, I get a message saying it is not accessable.
Regards
0
 
LVL 19

Expert Comment

by:DreamMaster
ID: 6167501
I have no idea...maybe attempts to get there are registered as well??

Max.
0
3 Use Cases for Connected Systems

Our Dev teams are like yours. They’re continually cranking out code for new features/bugs fixes, testing, deploying, testing some more, responding to production monitoring events and more. It’s complex. So, we thought you’d like to see what’s working for us.

 
LVL 8

Expert Comment

by:bebonham
ID: 6168092
do you mean there is no public ip address, or do you mean that port 80 is blocked or inactive...

which?

if there is no public ip address, it is comming from inside your network.  

otherwise, it could feasibly be a robot.

Bob
0
 
LVL 24

Expert Comment

by:SunBow
ID: 6169194
Say, how's about we get these bot's to identify themselves!
0
 
LVL 9

Accepted Solution

by:
TTom earned 50 total points
ID: 6169585
Ah yes, but the 403 in the log indicates that the robot requested a page and was denied.  An error page was sent back indicating that access was denied.

I suspect if you go home and try to access the server yourself, you will generate a similar log entry from your "home" IP address.

In this case, you were visited by a well-behaved bot which was asking permission to visit your server.  That's why it requested robots.txt.

Tom
0

Featured Post

DevOps Toolchain Recommendations

Read this Gartner Research Note and discover how your IT organization can automate and optimize DevOps processes using a toolchain architecture.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
Jump links not quite lining up 18 65
Google search result shows " This site may be hacked" 12 70
Form Processing in PHP 11 32
Standard for Web Development 3 27
Popularity Can Be Measured Sometimes we deal with questions of popularity, and we need a way to collect opinions from our clients.  This article shows a simple teaching example of how we might elect a favorite color by letting our clients vote for …
Password hashing is better than message digests or encryption, and you should be using it instead of message digests or encryption.  Find out why and how in this article, which supplements the original article on PHP Client Registration, Login, Logo…
This tutorial demonstrates how to identify and create boundary or building outlines in Google Maps. In this example, I outline the boundaries of an enclosed skatepark within a community park.  Login to your Google Account, then  Google for "Google M…
This video teaches users how to migrate an existing Wordpress website to a new domain.

910 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now