I do have a question regarding the wikipedia article, something I don't understand.
The article talks about keeping crawlers out of the site or out of specific folders, but what I don't understand is why are the crawlers establishing all those port 80 connections, which is effectively shutting down the site. Is that their intent?
I do have a question regarding the wikipedia article, something I don't understand.
The article talks about keeping crawlers out of the site or out of specific folders, but what I don't understand is why are the crawlers establishing all those port 80 connections, which is effectively shutting down the site. Is that their intent?
Thanks again.