Solved

robots.txt

Posted on 2001-06-08
6
194 Views
Last Modified: 2006-11-17
I found the following entry in a log file on our server.  The 62.116.37.249 address is completely unknown to me, can anyone please tell me what has happened? (I have changed our address).
Regards
62.116.37.249, -, 6/6/01, 3:03:15, W3SVC1, XXXSERVER, 195.123.123.123, 15, 84, 669, 403, 5, GET, /robots.txt, -,
0
Comment
Question by:Chris_m
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 19

Expert Comment

by:DreamMaster
ID: 6167365
Probably...and I'm almost certain of this, this means that you have been visited by a search engine's bot...

Cheers,
Max.
0
 

Author Comment

by:Chris_m
ID: 6167469
I suspected that that was the case -- especially when I saw the robots part of the entry, but if that is so, how?  I have shut our server down to all computers except our internal lan.  I know that if I try to access it from outside, I get a message saying it is not accessable.
Regards
0
 
LVL 19

Expert Comment

by:DreamMaster
ID: 6167501
I have no idea...maybe attempts to get there are registered as well??

Max.
0
Get 15 Days FREE Full-Featured Trial

Benefit from a mission critical IT monitoring with Monitis Premium or get it FREE for your entry level monitoring needs.
-Over 200,000 users
-More than 300,000 websites monitored
-Used in 197 countries
-Recommended by 98% of users

 
LVL 8

Expert Comment

by:bebonham
ID: 6168092
do you mean there is no public ip address, or do you mean that port 80 is blocked or inactive...

which?

if there is no public ip address, it is comming from inside your network.  

otherwise, it could feasibly be a robot.

Bob
0
 
LVL 24

Expert Comment

by:SunBow
ID: 6169194
Say, how's about we get these bot's to identify themselves!
0
 
LVL 9

Accepted Solution

by:
TTom earned 50 total points
ID: 6169585
Ah yes, but the 403 in the log indicates that the robot requested a page and was denied.  An error page was sent back indicating that access was denied.

I suspect if you go home and try to access the server yourself, you will generate a similar log entry from your "home" IP address.

In this case, you were visited by a well-behaved bot which was asking permission to visit your server.  That's why it requested robots.txt.

Tom
0

Featured Post

Don't Cry: How Liquid Web is Ensuring Security

WannaCry is just the start. Read how Liquid Web is protecting itself and its customers against new threats.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Does your audience prefer people in photos or no people? How can you best highlight what you’re selling? What are your competitors doing, and what can you do that is different and unique from them?  Continue reading to learn how to make your images …
Although a lot of people devote their energy toward marketing for specific industries, there are some basic principles that can be applied to any sector imaginable. We’ll look at four steps to take and examine how those steps were put into action fo…
The viewer will learn how to count occurrences of each item in an array.
Any person in technology especially those working for big companies should at least know about the basics of web accessibility. Believe it or not there are even laws in place that require businesses to provide such means for the disabled and aging p…

623 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question