robots.txt

Hello experts,

I'm configuring my robots.txt for upload. My questions is:
In what order do the robots follow the index in the txt? should I place the part regarding all robots first or after the part for specified agents?

User-agent: *
Disallow: /folder/

User-agent: Googlebot*
Disallow: /*.js$

or should it be

User-agent: Googlebot*
Disallow: /*.js$

User-agent: *
Disallow: /folder/

As I remember, there is a order how they "read" the file

Cheers
Dimi
sh2grAsked:
Who is Participating?
 
nltechConnect With a Mentor Commented:
i put the specific ones above the * wildcard and that's general practice.

older bots may get confused if the wildcard isn't last, but *technically it shouldn't matter what order the rules are in.

http://www.ilovejackdaniels.com/development/robots-txt-file/

check your robots.txt file here

http://tool.motoricerca.info/robots-checker.phtml
0
 
sh2grAuthor Commented:
Thank you nltech,

that helped.

Dimi
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.