?
Solved

Increasing 404 error

Posted on 2016-09-29
3
Medium Priority
?
54 Views
Last Modified: 2016-10-19
In my site unwanted 404 pages are indexing daily. Every page contains url like http://example.com/ray-*'*.

some links for the reference

example.com/ray-your-glass/
example.com/rayban-qatar/
example.com/rayban-korea/

Is there any way to block certain word like 'ray,rayban' from indexing in robots.txt

Thanks
0
Comment
Question by:ragav r
  • 2
3 Comments
 
LVL 84

Assisted Solution

by:Dave Baldwin
Dave Baldwin earned 1000 total points (awarded by participants)
ID: 41822937
No, robots.txt is a voluntary instruction to friendly search bots.  If you're using Apache, you may be able to block URLs that have the letters 'ray' in them in an .htaccess file.  I don't know how to do that but I think it can be done.
0
 
LVL 30

Accepted Solution

by:
Dr. Klahn earned 1000 total points (awarded by participants)
ID: 41823191
If you are using Apache 2 on linux,  the mod_nsf module can be useful.  mod_nsf blocks the initial access to a non-existent page and blacklists all succeeding accesses from the offending IPv4 address for a fixed time interval.

If the accesses are coming from just a few IP blocks, use iptables or a mod_rewrite exclusion rule to drop them at the connect stage.

If your site does business only within North America, an iptables rule to drop all connections
(a) on port 80 and
(b) not from the US, Canada or Mexico
gets rid of an enormous amount of hostile accesses.
1
 
LVL 30

Expert Comment

by:Dr. Klahn
ID: 41849722
EE email requested stale question closure.
0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Introduction A frequently used term in Object-Oriented design is "SOLID" which is a mnemonic acronym that covers five principles of OO design.  These principles do not stand alone; there is interplay among them.  And they are not laws, merely princ…
A publishing tool, a Version Control System, or a Collaboration Platform! These can be some of the defining words for the two very famous web-hosting Git repositories: Bitbucket and Github. Git is widely used amongst the programmers and developers f…
This video teaches users how to migrate an existing Wordpress website to a new domain.
Wufoo.com provides powerful tools for surveying targeted groups, and utilizing data from completed surveys to find trends, discover areas of demand or customer expectation, and make business decisions on products or services.
Suggested Courses

840 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question