Posted on 2006-05-06
Last Modified: 2010-04-06
Hello experts,

I'm configuring my robots.txt for upload. My questions is:
In what order do the robots follow the index in the txt? should I place the part regarding all robots first or after the part for specified agents?

User-agent: *
Disallow: /folder/

User-agent: Googlebot*
Disallow: /*.js$

or should it be

User-agent: Googlebot*
Disallow: /*.js$

User-agent: *
Disallow: /folder/

As I remember, there is a order how they "read" the file

Question by:sh2gr
    LVL 14

    Accepted Solution

    i put the specific ones above the * wildcard and that's general practice.

    older bots may get confused if the wildcard isn't last, but *technically it shouldn't matter what order the rules are in.

    check your robots.txt file here

    Author Comment

    Thank you nltech,

    that helped.


    Featured Post

    6 Surprising Benefits of Threat Intelligence

    All sorts of threat intelligence is available on the web. Intelligence you can learn from, and use to anticipate and prepare for future attacks.

    Join & Write a Comment

    Styling your websites can become very complex. Here I'll show how SASS can help you better organize, maintain and reuse your CSS code.
    Browsers only know CSS so your awesome SASS code needs to be translated into normal CSS. Here I'll try to explain what you should aim for in order to take full advantage of SASS.
    HTML5 has deprecated a few of the older ways of showing media as well as offering up a new way to create games and animations. Audio, video, and canvas are just a few of the adjustments made between XHTML and HTML5. As we learned in our last micr…
    The viewer will learn how to create a basic form using some HTML5 and PHP for later processing. Set up your basic HTML file. Open your form tag and set the method and action attributes.: (CODE) Set up your first few inputs one for the name and …

    754 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    19 Experts available now in Live!

    Get 1:1 Help Now