Solved

robots.txt

Posted on 2001-06-08
6
192 Views
Last Modified: 2006-11-17
I found the following entry in a log file on our server.  The 62.116.37.249 address is completely unknown to me, can anyone please tell me what has happened? (I have changed our address).
Regards
62.116.37.249, -, 6/6/01, 3:03:15, W3SVC1, XXXSERVER, 195.123.123.123, 15, 84, 669, 403, 5, GET, /robots.txt, -,
0
Comment
Question by:Chris_m
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
6 Comments
 
LVL 19

Expert Comment

by:DreamMaster
ID: 6167365
Probably...and I'm almost certain of this, this means that you have been visited by a search engine's bot...

Cheers,
Max.
0
 

Author Comment

by:Chris_m
ID: 6167469
I suspected that that was the case -- especially when I saw the robots part of the entry, but if that is so, how?  I have shut our server down to all computers except our internal lan.  I know that if I try to access it from outside, I get a message saying it is not accessable.
Regards
0
 
LVL 19

Expert Comment

by:DreamMaster
ID: 6167501
I have no idea...maybe attempts to get there are registered as well??

Max.
0
Webinar: Aligning, Automating, Winning

Join Dan Russo, Senior Manager of Operations Intelligence, for an in-depth discussion on how Dealertrack, leading provider of integrated digital solutions for the automotive industry, transformed their DevOps processes to increase collaboration and move with greater velocity.

 
LVL 8

Expert Comment

by:bebonham
ID: 6168092
do you mean there is no public ip address, or do you mean that port 80 is blocked or inactive...

which?

if there is no public ip address, it is comming from inside your network.  

otherwise, it could feasibly be a robot.

Bob
0
 
LVL 24

Expert Comment

by:SunBow
ID: 6169194
Say, how's about we get these bot's to identify themselves!
0
 
LVL 9

Accepted Solution

by:
TTom earned 50 total points
ID: 6169585
Ah yes, but the 403 in the log indicates that the robot requested a page and was denied.  An error page was sent back indicating that access was denied.

I suspect if you go home and try to access the server yourself, you will generate a similar log entry from your "home" IP address.

In this case, you were visited by a well-behaved bot which was asking permission to visit your server.  That's why it requested robots.txt.

Tom
0

Featured Post

Resolve Critical IT Incidents Fast

If your data, services or processes become compromised, your organization can suffer damage in just minutes and how fast you communicate during a major IT incident is everything. Learn how to immediately identify incidents & best practices to resolve them quickly and effectively.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

I've been asked to discuss some of the UX activities that I'm using with my team. Here I will share some details about how we approach UX projects.
This article will inform Clients about common and important expectations from the freelancers (Experts) who are looking at your Gig.
Explain concepts important to validation of email addresses with regular expressions. Applies to most languages/tools that uses regular expressions. Consider email address RFCs: Look at HTML5 form input element (with type=email) regex pattern: T…
The viewer will learn how to dynamically set the form action using jQuery.

756 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question