Solved

Increasing 404 error

Posted on 2016-09-29
3
27 Views
Last Modified: 2016-10-19
In my site unwanted 404 pages are indexing daily. Every page contains url like http://example.com/ray-*'*.

some links for the reference

example.com/ray-your-glass/
example.com/rayban-qatar/
example.com/rayban-korea/

Is there any way to block certain word like 'ray,rayban' from indexing in robots.txt

Thanks
0
Comment
Question by:ragav r
  • 2
3 Comments
 
LVL 83

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 250 total points (awarded by participants)
ID: 41822937
No, robots.txt is a voluntary instruction to friendly search bots.  If you're using Apache, you may be able to block URLs that have the letters 'ray' in them in an .htaccess file.  I don't know how to do that but I think it can be done.
0
 
LVL 25

Accepted Solution

by:
Dr. Klahn earned 250 total points (awarded by participants)
ID: 41823191
If you are using Apache 2 on linux,  the mod_nsf module can be useful.  mod_nsf blocks the initial access to a non-existent page and blacklists all succeeding accesses from the offending IPv4 address for a fixed time interval.

If the accesses are coming from just a few IP blocks, use iptables or a mod_rewrite exclusion rule to drop them at the connect stage.

If your site does business only within North America, an iptables rule to drop all connections
(a) on port 80 and
(b) not from the US, Canada or Mexico
gets rid of an enormous amount of hostile accesses.
1
 
LVL 25

Expert Comment

by:Dr. Klahn
ID: 41849722
EE email requested stale question closure.
0

Featured Post

NAS Cloud Backup Strategies

This article explains backup scenarios when using network storage. We review the so-called “3-2-1 strategy” and summarize the methods you can use to send NAS data to the cloud

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Suggested Solutions

Using Quotation Marks in PHP This question (http://www.experts-exchange.com/Web_Development/Web_Languages-Standards/PHP/Q_28217211.html) seems to come up a lot for developers who are new to PHP.  And it got me thinking, "How can we explain the rule…
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
Use Wufoo, an online form creation tool, to make powerful forms. Learn how to choose which pages of your form are visible to your users based on their inputs. The page rules feature provides you with an opportunity to create if:then statements for y…
Learn how to set-up PayPal payment integration in your Wufoo form. Allow your users to remit payment through PayPal upon completion of your online form. This is helpful for collecting membership payments, customer payments, donations, and more.

809 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question