Wildcard Disallow for robots.txt file ?

Posted on 2012-08-16
Last Modified: 2013-11-19
I have a number of pages of a site that have a paramater that I would like, when a robot finds it, will ignore it.



Can I make one disallow to cover anything that ends in /foobar.raw ??


Specifically, can I put the following in the robots.txt file

disallow /foobar

....and then it would take care of all instances where /foobar.raw is the last item in a url?


Question by:Rowby Goren
    LVL 16

    Accepted Solution

    First you are assuming that all spiders will comply with your robots.txt instructions, and this is not actually the case. Apart from perhaps Google, Inktomi and MS, the majority of spiders that will hit your site will igonore the disallow instructions, and worse still anything "disallowed"  is like a big welcome sign for anyone with malicious intent.

    With this said the robots.txt file is more directory orientated than individual files, so while you can disallow an entire directory (effectively wildcard'ing its contents) you must explicitly state individual files.

    In your post the "disallow /foobar" would be interpreted as disallowing access to the directory "foobar" and all contents in it.

    Thus, you will have to explicitly disallow each and every file you do not want spiders to access.


    disallow /bananas/frog/apple/foobar.raw
    disallow /mary/jack/jill/foobar.raw
    disallow /tony/orlando/dawn/foobar.raw
    LVL 9

    Author Closing Comment

    by:Rowby Goren
    Hi  grahamnonweiler,

    Thanks for that info and observation re behavior of spiders.

    Sorry for the delay in awarding you your points.

    I know now what to do!


    Featured Post

    Enabling OSINT in Activity Based Intelligence

    Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

    Join & Write a Comment

    [Part 5 of a 6 part series called SEO Basics: 5 SEO Secrets for Creating Content that Drives Traffic (…
    Read about why website design really matters in today's demanding market.
    This tutorial will teach you the core code needed to finalize the addition of a watermark to your image. The viewer will use a small PHP class to learn and create a watermark.
    This Micro Tutorial will demonstrate how to add subdomains to your content reports. This can be very importing in having a site with multiple subdomains.

    746 members asked questions and received personalized solutions in the past 7 days.

    Join the community of 500,000 technology professionals and ask your questions.

    Join & Ask a Question

    Need Help in Real-Time?

    Connect with top rated Experts

    20 Experts available now in Live!

    Get 1:1 Help Now