Posted on 2006-05-06
Medium Priority
Last Modified: 2010-04-06
Hello experts,

I'm configuring my robots.txt for upload. My questions is:
In what order do the robots follow the index in the txt? should I place the part regarding all robots first or after the part for specified agents?

User-agent: *
Disallow: /folder/

User-agent: Googlebot*
Disallow: /*.js$

or should it be

User-agent: Googlebot*
Disallow: /*.js$

User-agent: *
Disallow: /folder/

As I remember, there is a order how they "read" the file

Question by:sh2gr
LVL 14

Accepted Solution

nltech earned 2000 total points
ID: 16623824
i put the specific ones above the * wildcard and that's general practice.

older bots may get confused if the wildcard isn't last, but *technically it shouldn't matter what order the rules are in.


check your robots.txt file here


Author Comment

ID: 16624066
Thank you nltech,

that helped.


Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction Since I wrote the original article about Handling Date and Time in PHP and MySQL several years ago, it seemed like now was a good time to update it for object-oriented PHP.  This article does that, replacing as much as possible the pr…
Without even knowing it, most of us are using web applications on a daily basis.  In fact, Gmail and Yahoo email, Twitter, Facebook, and eBay are used by most of us daily—and they are web applications. We generally confuse these web applications to…
Viewers will learn about the regular for loop in Java and how to use it. Definition: Break the for loop down into 3 parts: Syntax when using for loops: Example using a for loop:
Learn how to create flexible layouts using relative units in CSS.  New relative units added in CSS3 include vw(viewports width), vh(viewports height), vmin(minimum of viewports height and width), and vmax (maximum of viewports height and width).
Suggested Courses

862 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question