<div>
This can not be done. &nbsp;The Robots Exclusion Standard allows only for the specification of robot names and limited URLs. &nbsp;The <i>hope</i> is that the robot will see it is excluded in robots.txt and then go away quietly.<br />
<br />
In order for that to work, the robot must adhere to the standard -- and most do not. &nbsp;Even some of the halfway respectable robots look at robots.txt, see they are excluded, then ignore it and pull pages anyway.<br />
<br />
Refusing requests according to the agent string doesn't work either. &nbsp;All bad robots fake their user-agent strings so that they look like real browsers run by humans.<br />
<br />
There is no simple solution to this problem and there is not even one that works half the time. &nbsp;Multiple approaches are required and even then 80% recognition should be considered unbelievable success. &nbsp;Have a look at the page below where I detail some travails with bots.<br />
<br />
<a href="http://www.miim.com/thebside/security/dumbbottricks.shtml" rel="ugc">http://www.miim.com/thebside/security/dumbbottricks.shtml</a>
</div>


Fractional CTO

David Favor

IT Tech Support

Chris Kenward

<div>
Hi there and thank you very much for the fast response. This is what I received from my client:<br />
<br />
&quot;&nbsp;A 503 Service Unavailable http status code just pauses the crawls and doesn't impact on existing results. This is therefore what is needed for the moment.&quot;<br />
<br />
Are you saying that this is not possible at all?<br />
<br />
Regards<br />
Chris
</div>


<div>
It cannot be done in robots.txt at all.<br />
<br />
If the robot in question is well-behaved, and identifies itself properly with an unique browser ident string, then it is possible (in Apache) to write a mod_rewrite rule to force a 503 Service Unavailable for any URL requested by that robot. &nbsp;But this handles only that one specific robot.
</div>


<div>
Gosh that's a shame. The site is a Wordpress site and had been hacked. What we wanted to do was try and deter all crawlers from the site until we were sure we had removed all the malware and stopped the creation of any further bad posts.<br />
<br />
Many thanks - I guess it's back to the drawing board. In the meantime I have checked the box in Wordpress for the site that indicates that it will request crawlers to NOT index the site.<br />
<br />
Best wishes<br />
Chris
</div>


<div>
Adding the following comment, for future similar queries...<br />
<br />
1) As Dr. Klahn stated. This cannot be done via robots.txt as this file is only a hint file used by Bots to scrape data out of all blocks listed in the file.<br />
<br />
In other words, if you say block a directory /foo evil Bots will first visit /foo looking for data to scrape, so...<br />
<br />
Using robots.txt is a fools errand.<br />
<br />
2) This said, you can easily accomplish what you're trying to accomplish... if you must...<br />
<br />
By &quot;if you must&quot; I mean if you have sufficient will + budget + time, this can be accomplished.<br />
<br />
This can easily be accomplished by sensing when Bots visit, using a combination of Fail2Ban + other technologies.<br />
<br />
Note: You can only block brain dead, simple Bots using this tech.<br />
<br />
3) You can also block IP block ranges owned by companies like Google. For example, it's trivial to scrape a copy of the GoogleBot IP blocks out of DNS daily, add these to an iptables + ipset firewall setup, then block all incoming requests.<br />
<br />
There is no way to easily produce an HTTPS level 503 using this tech, only a ICMP Reject (disallow any connection).<br />
<br />
If you just return a 503, you'll use #2.<br />
<br />
4) Implementations for these 2x options can be done. The primary consideration is if the money returned for doing this work justifies the time + cost of doing the work.<br />
<br />
If sufficient return exists, then you can hire someone to do this work fairly easy.<br />
<br />
Caveat: To implement #2 or #3 to work quickly + consistently, requires a good starting point. Preferably Kernel version 4.15+ along with iptables/ipset, so you'll be using some Distro like Ubuntu Bionic or CentOS 8 as your starting point.<br />
<br />
Then which ever option you choose will have it's own unique additional requirements.<br />
<br />
Also, be sure your budget contains sufficient funding for managing this tech, as some number of hours/month are required to keep everything running.
</div>


<div>
Thanks to you both! The robots.txt file is history. :)<br />
<br />
Regards<br />
Chris
</div>


<div>
The solution is... generating a 503 is an incorrect solution.<br />
<br />
Better, to me, to use a target like...<br />
<br />

<pre><code class="code-1 nolinks" id="code-20-42998280-1"><span>iptables --wait -A DROP-BOT -j REJECT --reject-with icmp-host-prohibited</span>
</code></pre>
<br />
Then route Bot requests through this target.<br />
<br />
Or better use the TARPIT target, which very rapidly discourages any Bot from visiting your site.
</div>


<div>
<div class="content wysiwyg-content">
I want to create a robots.txt file which will disable crawling on my site by returning a 503 server unavailable error. I cannot find the syntax to this anywhere. Can someone help with this one?<br />
<br />
Many thanks<br />
Chris
</div>
</div>


I want to create a robots.txt file which will disable crawling on my site by returning a 503 server unavailable error. I cannot find the syntax to this anywhere. Can someone help with this one?

Many thanks
Chris

Help with 503 in ROBOTS.TXT

Web development includes all aspects of presenting content on intranets and the Internet, including delivery development, protocols, languages and standards, server software, browser clients, databases and multimedia generation.

Web Development

The Apache HTTP Server is a secure, efficient and extensible server that provides HTTP services in sync with the current HTTP standards. Typically Apache is run on a Unix-like operating system, but it is available for a wide variety of operating systems, including Linux, Novell NetWare, Mac OS-X and Windows. Released under the Apache License, Apache is open-source software.