Increasing 404 error

In my site unwanted 404 pages are indexing daily. Every page contains url like http://example.com/ray-*'*.

some links for the reference

example.com/ray-your-glass/
example.com/rayban-qatar/
example.com/rayban-korea/

Is there any way to block certain word like 'ray,rayban' from indexing in robots.txt

Thanks
ragav rAsked:
Who is Participating?
 
Dr. KlahnConnect With a Mentor Principal Software EngineerCommented:
If you are using Apache 2 on linux,  the mod_nsf module can be useful.  mod_nsf blocks the initial access to a non-existent page and blacklists all succeeding accesses from the offending IPv4 address for a fixed time interval.

If the accesses are coming from just a few IP blocks, use iptables or a mod_rewrite exclusion rule to drop them at the connect stage.

If your site does business only within North America, an iptables rule to drop all connections
(a) on port 80 and
(b) not from the US, Canada or Mexico
gets rid of an enormous amount of hostile accesses.
1
 
Dave BaldwinConnect With a Mentor Fixer of ProblemsCommented:
No, robots.txt is a voluntary instruction to friendly search bots.  If you're using Apache, you may be able to block URLs that have the letters 'ray' in them in an .htaccess file.  I don't know how to do that but I think it can be done.
0
 
Dr. KlahnPrincipal Software EngineerCommented:
EE email requested stale question closure.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.