Solved

robots.txt

Posted on 2001-06-08
6
190 Views
Last Modified: 2006-11-17
I found the following entry in a log file on our server.  The 62.116.37.249 address is completely unknown to me, can anyone please tell me what has happened? (I have changed our address).
Regards
62.116.37.249, -, 6/6/01, 3:03:15, W3SVC1, XXXSERVER, 195.123.123.123, 15, 84, 669, 403, 5, GET, /robots.txt, -,
0
Comment
Question by:Chris_m
6 Comments
 
LVL 19

Expert Comment

by:DreamMaster
ID: 6167365
Probably...and I'm almost certain of this, this means that you have been visited by a search engine's bot...

Cheers,
Max.
0
 

Author Comment

by:Chris_m
ID: 6167469
I suspected that that was the case -- especially when I saw the robots part of the entry, but if that is so, how?  I have shut our server down to all computers except our internal lan.  I know that if I try to access it from outside, I get a message saying it is not accessable.
Regards
0
 
LVL 19

Expert Comment

by:DreamMaster
ID: 6167501
I have no idea...maybe attempts to get there are registered as well??

Max.
0
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

 
LVL 8

Expert Comment

by:bebonham
ID: 6168092
do you mean there is no public ip address, or do you mean that port 80 is blocked or inactive...

which?

if there is no public ip address, it is comming from inside your network.  

otherwise, it could feasibly be a robot.

Bob
0
 
LVL 24

Expert Comment

by:SunBow
ID: 6169194
Say, how's about we get these bot's to identify themselves!
0
 
LVL 9

Accepted Solution

by:
TTom earned 50 total points
ID: 6169585
Ah yes, but the 403 in the log indicates that the robot requested a page and was denied.  An error page was sent back indicating that access was denied.

I suspect if you go home and try to access the server yourself, you will generate a similar log entry from your "home" IP address.

In this case, you were visited by a well-behaved bot which was asking permission to visit your server.  That's why it requested robots.txt.

Tom
0

Featured Post

Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Read about why website design really matters in today's demanding market.
Does your audience prefer people in photos or no people? How can you best highlight what you’re selling? What are your competitors doing, and what can you do that is different and unique from them?  Continue reading to learn how to make your images …
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
This video teaches users how to migrate an existing Wordpress website to a new domain.

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question