Solved

Increasing 404 error

Posted on 2016-09-29
3
25 Views
Last Modified: 2016-10-19
In my site unwanted 404 pages are indexing daily. Every page contains url like http://example.com/ray-*'*.

some links for the reference

example.com/ray-your-glass/
example.com/rayban-qatar/
example.com/rayban-korea/

Is there any way to block certain word like 'ray,rayban' from indexing in robots.txt

Thanks
0
Comment
Question by:ragav r
  • 2
3 Comments
 
LVL 83

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 250 total points (awarded by participants)
ID: 41822937
No, robots.txt is a voluntary instruction to friendly search bots.  If you're using Apache, you may be able to block URLs that have the letters 'ray' in them in an .htaccess file.  I don't know how to do that but I think it can be done.
0
 
LVL 24

Accepted Solution

by:
Dr. Klahn earned 250 total points (awarded by participants)
ID: 41823191
If you are using Apache 2 on linux,  the mod_nsf module can be useful.  mod_nsf blocks the initial access to a non-existent page and blacklists all succeeding accesses from the offending IPv4 address for a fixed time interval.

If the accesses are coming from just a few IP blocks, use iptables or a mod_rewrite exclusion rule to drop them at the connect stage.

If your site does business only within North America, an iptables rule to drop all connections
(a) on port 80 and
(b) not from the US, Canada or Mexico
gets rid of an enormous amount of hostile accesses.
1
 
LVL 24

Expert Comment

by:Dr. Klahn
ID: 41849722
EE email requested stale question closure.
0

Featured Post

Is Your Active Directory as Secure as You Think?

More than 75% of all records are compromised because of the loss or theft of a privileged credential. Experts have been exploring Active Directory infrastructure to identify key threats and establish best practices for keeping data safe. Attend this month’s webinar to learn more.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

A Change in PHP Behavior with Session Write Short Circuit (http://php.net/manual/en/book.session.php#116217) (Winter 2014)** With the release of PHP 5.6 the session handler changed in a way that many think should be considered a bug.  See the note …
Thoughout my experience working on eCommerce web applications I have seen applications succumbing to increased user demand and throughput. With increased loads the response times started to spike, which leads to user frustration and lost sales. I ha…
This video teaches users how to migrate an existing Wordpress website to a new domain.
Learn how to set-up custom confirmation messages to users who complete your Wufoo form. Include inputs from fields in your form, webpage redirects, and more with Wufoo’s confirmation options.

867 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now