sh2gr
asked on
robots.txt
Hello experts,
I'm configuring my robots.txt for upload. My questions is:
In what order do the robots follow the index in the txt? should I place the part regarding all robots first or after the part for specified agents?
User-agent: *
Disallow: /folder/
User-agent: Googlebot*
Disallow: /*.js$
or should it be
User-agent: Googlebot*
Disallow: /*.js$
User-agent: *
Disallow: /folder/
As I remember, there is a order how they "read" the file
Cheers
Dimi
I'm configuring my robots.txt for upload. My questions is:
In what order do the robots follow the index in the txt? should I place the part regarding all robots first or after the part for specified agents?
User-agent: *
Disallow: /folder/
User-agent: Googlebot*
Disallow: /*.js$
or should it be
User-agent: Googlebot*
Disallow: /*.js$
User-agent: *
Disallow: /folder/
As I remember, there is a order how they "read" the file
Cheers
Dimi
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
that helped.
Dimi