Solved

Increasing 404 error

Posted on 2016-09-29
3
29 Views
Last Modified: 2016-10-19
In my site unwanted 404 pages are indexing daily. Every page contains url like http://example.com/ray-*'*.

some links for the reference

example.com/ray-your-glass/
example.com/rayban-qatar/
example.com/rayban-korea/

Is there any way to block certain word like 'ray,rayban' from indexing in robots.txt

Thanks
0
Comment
Question by:ragav r
  • 2
3 Comments
 
LVL 83

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 250 total points (awarded by participants)
ID: 41822937
No, robots.txt is a voluntary instruction to friendly search bots.  If you're using Apache, you may be able to block URLs that have the letters 'ray' in them in an .htaccess file.  I don't know how to do that but I think it can be done.
0
 
LVL 27

Accepted Solution

by:
Dr. Klahn earned 250 total points (awarded by participants)
ID: 41823191
If you are using Apache 2 on linux,  the mod_nsf module can be useful.  mod_nsf blocks the initial access to a non-existent page and blacklists all succeeding accesses from the offending IPv4 address for a fixed time interval.

If the accesses are coming from just a few IP blocks, use iptables or a mod_rewrite exclusion rule to drop them at the connect stage.

If your site does business only within North America, an iptables rule to drop all connections
(a) on port 80 and
(b) not from the US, Canada or Mexico
gets rid of an enormous amount of hostile accesses.
1
 
LVL 27

Expert Comment

by:Dr. Klahn
ID: 41849722
EE email requested stale question closure.
0

Featured Post

Free Tool: Subnet Calculator

The subnet calculator helps you design networks by taking an IP address and network mask and returning information such as network, broadcast address, and host range.

One of a set of tools we're offering as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
These days, all we hear about hacktivists took down so and so websites and retrieved thousands of user’s data. One of the techniques to get unauthorized access to database is by performing SQL injection. This article is quite lengthy which gives bas…
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to selectively show certain fields based on user input using rules to gather relevant information and data from your forms. The rules feature provides you with an opportunity…
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to choose which pages of your form are visible to your users based on their inputs. The page rules feature provides you with an opportunity to create if:then statements for y…

685 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question