Solved

robots.txt

Posted on 2001-06-08
6
191 Views
Last Modified: 2006-11-17
I found the following entry in a log file on our server.  The 62.116.37.249 address is completely unknown to me, can anyone please tell me what has happened? (I have changed our address).
Regards
62.116.37.249, -, 6/6/01, 3:03:15, W3SVC1, XXXSERVER, 195.123.123.123, 15, 84, 669, 403, 5, GET, /robots.txt, -,
0
Comment
Question by:Chris_m
6 Comments
 
LVL 19

Expert Comment

by:DreamMaster
ID: 6167365
Probably...and I'm almost certain of this, this means that you have been visited by a search engine's bot...

Cheers,
Max.
0
 

Author Comment

by:Chris_m
ID: 6167469
I suspected that that was the case -- especially when I saw the robots part of the entry, but if that is so, how?  I have shut our server down to all computers except our internal lan.  I know that if I try to access it from outside, I get a message saying it is not accessable.
Regards
0
 
LVL 19

Expert Comment

by:DreamMaster
ID: 6167501
I have no idea...maybe attempts to get there are registered as well??

Max.
0
Master Your Team's Linux and Cloud Stack

Come see why top tech companies like Mailchimp and Media Temple use Linux Academy to build their employee training programs.

 
LVL 8

Expert Comment

by:bebonham
ID: 6168092
do you mean there is no public ip address, or do you mean that port 80 is blocked or inactive...

which?

if there is no public ip address, it is comming from inside your network.  

otherwise, it could feasibly be a robot.

Bob
0
 
LVL 24

Expert Comment

by:SunBow
ID: 6169194
Say, how's about we get these bot's to identify themselves!
0
 
LVL 9

Accepted Solution

by:
TTom earned 50 total points
ID: 6169585
Ah yes, but the 403 in the log indicates that the robot requested a page and was denied.  An error page was sent back indicating that access was denied.

I suspect if you go home and try to access the server yourself, you will generate a similar log entry from your "home" IP address.

In this case, you were visited by a well-behaved bot which was asking permission to visit your server.  That's why it requested robots.txt.

Tom
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Title # Comments Views Activity
push logos in footer up higher 5 25
how to build a intranet site 8 42
move expression web site to a new server 13 53
Grunt script for Build Process 1 42
Although it can be difficult to imagine, someday your child will have a career of his or her own. He or she will likely start a family, buy a home and start having their own children. So, while being a kid is still extremely important, it’s also …
Any business that wants to seriously grow needs to keep the needs and desires of an international audience of their websites in mind. Making a website friendly to international users isn’t prohibitively expensive and can provide an incredible return…
Viewers will get an overview of the benefits and risks of using Bitcoin to accept payments. What Bitcoin is: Legality: Risks: Benefits: Which businesses are best suited?: Other things you should know: How to get started:
Learn how to create flexible layouts using relative units in CSS.  New relative units added in CSS3 include vw(viewports width), vh(viewports height), vmin(minimum of viewports height and width), and vmax (maximum of viewports height and width).

839 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question