Member_2_6492660_1
asked on
Event Id 1309 Danagerous Request Path on my Site
Windows 2012 R2 Server
IIS 8.5
ASP
I posted this issue some time ago. And I am continually getting this event ID 1309 on my Site.
The error in full is this
Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 7/30/2015 6:36:49 AM
Event ID: 1309
Task Category: Web Event
Level: Warning
Keywords: Classic
User: N/A
Computer: SERV013.FQDN.com
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 7/30/2015 6:36:49 AM
Event time (UTC): 7/30/2015 10:36:49 AM
Event ID: b5cd4bc9c6b443f99a438459d4 686897
Event sequence: 10
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/4/ROOT-1-1308272 4885370518 7
Trust level: Full
Application Virtual Path: /
Application Path: C:\inetpub\wwwroot\Music\
Machine name: SERV013
Process information:
Process ID: 8216
Process name: w3wp.exe
Account name: IIS APPPOOL\Music
Exception information:
Exception type: HttpException
Exception message: A potentially dangerous Request.Path value was detected from the client (&).
at System.Web.HttpRequest.Val idateInput IfRequired ByConfig()
at System.Web.HttpApplication .PipelineS tepManager .ValidateH elper(Http Context context)
Request information:
Request URL: http://www.tomsmp3.com/mp3/mp3musicalbums/&artistname&/&recordingtitle&/&trackfilename&
Request path: /mp3/mp3musicalbums/&a mp;artistn ame&am p;/&am p;recordin gtitle& ;amp;/& ;amp;track filename&a mp;amp
User host address: 207.46.13.113
User:
Is authenticated: False
Authentication Type:
Thread account name: IIS APPPOOL\Music
Thread information:
Thread ID: 23
Thread account name: IIS APPPOOL\Music
Is impersonating: False
Stack trace: at System.Web.HttpRequest.Val idateInput IfRequired ByConfig()
at System.Web.HttpApplication .PipelineS tepManager .ValidateH elper(Http Context context)
Custom event details: Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="ASP.NET 4.0.30319.0" />
<EventID Qualifiers="32768">1309</E ventID>
<Level>3</Level>
<Task>3</Task>
<Keywords>0x80000000000000 </Keywords >
<TimeCreated SystemTime="2015-07-30T10: 36:49.0000 00000Z" />
<EventRecordID>40250</Even tRecordID>
<Channel>Application</Chan nel>
<Computer>SERV013.FQDN.com </Computer >
<Security />
</System>
<EventData>
<Data>3005</Data>
<Data>An unhandled exception has occurred.</Data>
<Data>7/30/2015 6:36:49 AM</Data>
<Data>7/30/2015 10:36:49 AM</Data>
<Data>b5cd4bc9c6b443f99a43 8459d46868 97</Data>
<Data>10</Data>
<Data>1</Data>
<Data>0</Data>
<Data>/LM/W3SVC/4/ROOT-1-1 3082724885 3705187</D ata>
<Data>Full</Data>
<Data>/</Data>
<Data>C:\inetpub\wwwroot\M usic\</Dat a>
<Data>SERV013</Data>
<Data>
</Data>
<Data>8216</Data>
<Data>w3wp.exe</Data>
<Data>IIS APPPOOL\Music</Data>
<Data>HttpException</Data>
<Data>A potentially dangerous Request.Path value was detected from the client (&).
at System.Web.HttpRequest.Val idateInput IfRequired ByConfig()
at System.Web.HttpApplication .PipelineS tepManager .ValidateH elper(Http Context context)
</Data>
<Data>http://www.tomsmp3.com/mp3/mp3musicalbums/&amp;artistname&amp;/&amp;recordingtitle&amp;/&amp;trackfilename&amp</Data>
<Data>/mp3/mp3musicalbums/ &amp;a mp;artistn ame&am p;amp;/&am p;amp;amp; recordingt itle&a mp;amp;/&a mp;amp;amp ;trackfile name&a mp;amp</Da ta>
<Data>207.46.13.113</Data>
<Data>
</Data>
<Data>False</Data>
<Data>
</Data>
<Data>IIS APPPOOL\Music</Data>
<Data>23</Data>
<Data>IIS APPPOOL\Music</Data>
<Data>False</Data>
<Data> at System.Web.HttpRequest.Val idateInput IfRequired ByConfig()
at System.Web.HttpApplication .PipelineS tepManager .ValidateH elper(Http Context context)
</Data>
</EventData>
</Event>
Now I believe it is a certain type of input the end user is trying to put in my search page.
I found a note in my search that read this
"ll he/she has to do is to make sure that the application is not vulnerable to this type of attacks (malformed HTTP requests, SQL injection and so on). "
I have 3 or 4 pages that allows input to search the site
I also Have had to fix the SQL database for this site after several 1309 errors they are injecting search garbage.
So what can I do to stop this from happening ?
I can post my code if needed
Thoughts?
IIS 8.5
ASP
I posted this issue some time ago. And I am continually getting this event ID 1309 on my Site.
The error in full is this
Log Name: Application
Source: ASP.NET 4.0.30319.0
Date: 7/30/2015 6:36:49 AM
Event ID: 1309
Task Category: Web Event
Level: Warning
Keywords: Classic
User: N/A
Computer: SERV013.FQDN.com
Description:
Event code: 3005
Event message: An unhandled exception has occurred.
Event time: 7/30/2015 6:36:49 AM
Event time (UTC): 7/30/2015 10:36:49 AM
Event ID: b5cd4bc9c6b443f99a438459d4
Event sequence: 10
Event occurrence: 1
Event detail code: 0
Application information:
Application domain: /LM/W3SVC/4/ROOT-1-1308272
Trust level: Full
Application Virtual Path: /
Application Path: C:\inetpub\wwwroot\Music\
Machine name: SERV013
Process information:
Process ID: 8216
Process name: w3wp.exe
Account name: IIS APPPOOL\Music
Exception information:
Exception type: HttpException
Exception message: A potentially dangerous Request.Path value was detected from the client (&).
at System.Web.HttpRequest.Val
at System.Web.HttpApplication
Request information:
Request URL: http://www.tomsmp3.com/mp3/mp3musicalbums/&artistname&/&recordingtitle&/&trackfilename&
Request path: /mp3/mp3musicalbums/&a
User host address: 207.46.13.113
User:
Is authenticated: False
Authentication Type:
Thread account name: IIS APPPOOL\Music
Thread information:
Thread ID: 23
Thread account name: IIS APPPOOL\Music
Is impersonating: False
Stack trace: at System.Web.HttpRequest.Val
at System.Web.HttpApplication
Custom event details: Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="ASP.NET 4.0.30319.0" />
<EventID Qualifiers="32768">1309</E
<Level>3</Level>
<Task>3</Task>
<Keywords>0x80000000000000
<TimeCreated SystemTime="2015-07-30T10:
<EventRecordID>40250</Even
<Channel>Application</Chan
<Computer>SERV013.FQDN.com
<Security />
</System>
<EventData>
<Data>3005</Data>
<Data>An unhandled exception has occurred.</Data>
<Data>7/30/2015 6:36:49 AM</Data>
<Data>7/30/2015 10:36:49 AM</Data>
<Data>b5cd4bc9c6b443f99a43
<Data>10</Data>
<Data>1</Data>
<Data>0</Data>
<Data>/LM/W3SVC/4/ROOT-1-1
<Data>Full</Data>
<Data>/</Data>
<Data>C:\inetpub\wwwroot\M
<Data>SERV013</Data>
<Data>
</Data>
<Data>8216</Data>
<Data>w3wp.exe</Data>
<Data>IIS APPPOOL\Music</Data>
<Data>HttpException</Data>
<Data>A potentially dangerous Request.Path value was detected from the client (&).
at System.Web.HttpRequest.Val
at System.Web.HttpApplication
</Data>
<Data>http://www.tomsmp3.com/mp3/mp3musicalbums/&amp;artistname&amp;/&amp;recordingtitle&amp;/&amp;trackfilename&amp</Data>
<Data>/mp3/mp3musicalbums/
<Data>207.46.13.113</Data>
<Data>
</Data>
<Data>False</Data>
<Data>
</Data>
<Data>IIS APPPOOL\Music</Data>
<Data>23</Data>
<Data>IIS APPPOOL\Music</Data>
<Data>False</Data>
<Data> at System.Web.HttpRequest.Val
at System.Web.HttpApplication
</Data>
</EventData>
</Event>
Now I believe it is a certain type of input the end user is trying to put in my search page.
I found a note in my search that read this
"ll he/she has to do is to make sure that the application is not vulnerable to this type of attacks (malformed HTTP requests, SQL injection and so on). "
I have 3 or 4 pages that allows input to search the site
I also Have had to fix the SQL database for this site after several 1309 errors they are injecting search garbage.
So what can I do to stop this from happening ?
I can post my code if needed
Thoughts?
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Dave,
Can you post some of the corresponding entries from the HTTP logs. Last time this occurred, it was due to a search bot, which we blocked with a robots.txt file.
Dan
Can you post some of the corresponding entries from the HTTP logs. Last time this occurred, it was due to a search bot, which we blocked with a robots.txt file.
Dan
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Dan
This is my current robots.txt file which resides in the root of the site.
The IP Address of the Event ID 1309 is not allows the same.
here is the list
157.55.39.196
157.55.39.210 msnbot-157-55-39-196.searc h.msn.com
157.55.39.245
180.76.15.137
180.76.15.154 baiduspider-180-76-15-160. crawl.baid u.com China
180.76.15.160
66.249.75.4 crawl-66-249-75-4.googlebo t.com
66.87.101.169 169-101-87-66.pools.cgn.sp csdns.net
207.46.13.113
207.46.13.134 msnbot-207-46-13-113.searc h.msn.com
What about Dave's suggestions above two?
Do I need to allows update this file ?
Is there a way to stop this from happening all together?
This is all new to me since going to IIS 8.5
This is my current robots.txt file which resides in the root of the site.
User-agent: MJ12bot/v1.4.5
Disallow: /
The IP Address of the Event ID 1309 is not allows the same.
here is the list
157.55.39.196
157.55.39.210 msnbot-157-55-39-196.searc
157.55.39.245
180.76.15.137
180.76.15.154 baiduspider-180-76-15-160.
180.76.15.160
66.249.75.4 crawl-66-249-75-4.googlebo
66.87.101.169 169-101-87-66.pools.cgn.sp
207.46.13.113
207.46.13.134 msnbot-207-46-13-113.searc
What about Dave's suggestions above two?
Do I need to allows update this file ?
Is there a way to stop this from happening all together?
This is all new to me since going to IIS 8.5
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Dan
So like this?
What about the China ip addresses
180.76.15.137
180.76.15.154 baiduspider-180-76-15-160. crawl.baid u.com China
180.76.15.160
And this one
66.87.101.169 169-101-87-66.pools.cgn.sp csdns.net
you mention the HTTP logs
I ran this
I can find the ip addresses but they do not show anything that I can see that is helpful
I must not be looking at the correct thing
Thoughts
Update
Dan
I just found this
Wouldn't that work in place of adding each one that comes along?
Thoughts
So like this?
User-agent: MJ12bot/v1.4.5
Disallow: /mp3/mp3musicalablums
User-agent: msnbot
Disallow: /mp3/mp3musicalablums
User-agent: googlebot
Disallow: /mp3/mp3musicalablums
What about the China ip addresses
180.76.15.137
180.76.15.154 baiduspider-180-76-15-160.
180.76.15.160
And this one
66.87.101.169 169-101-87-66.pools.cgn.sp
you mention the HTTP logs
I ran this
echo off
"C:\Program Files (x86)\Log Parser 2.2\LogParser" "SELECT TO_TIMESTAMP(date, time) AS utc-timestamp, TO_LOCALTIME(utc-timestamp) AS local-timestamp, c-ip, cs-username, EXTRACT_TOKEN(cs-uri-stem,-1,'/') AS filename INTO DATAGRID FROM "C:\inetpub\logs\logfiles\w3svc4\u_ex15*.log"
exit
I can find the ip addresses but they do not show anything that I can see that is helpful
I must not be looking at the correct thing
Thoughts
Update
Dan
I just found this
User-agent: * # any robot
Disallow: /mp3/mp3musicalalbums # disallow this directory
Wouldn't that work in place of adding each one that comes along?
Thoughts
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
The link works from my laptop & my mobile. Not sure what say about the link.
Dan
Dan
66-87-97-4.pools.spcsdns.n et you in Texas? works here as well.
ASKER
Guys
No I am in New Jersey
Dan
I check my firewall whitelist on that url later.
this ip address showed up also
C:\>nslookup 188.165.15.121
Name: boson069.ahrefs.com
Address: 188.165.15.121
what should I do with this ip address and the 66.87.97.4 address???
I already updated my rebots.txt with the other bots
No I am in New Jersey
Dan
I check my firewall whitelist on that url later.
this ip address showed up also
C:\>nslookup 188.165.15.121
Name: boson069.ahrefs.com
Address: 188.165.15.121
what should I do with this ip address and the 66.87.97.4 address???
I already updated my rebots.txt with the other bots
you can block it via your firewall
ASKER
Dave
I do that on my exchange server I wrote a script to scan the event log then update the firewall entries of rogue ip address attempts.
I guess a Firewall Rule to block port 80 on these bad ip addresses
Is that what you are thinking?
I do that on my exchange server I wrote a script to scan the event log then update the firewall entries of rogue ip address attempts.
I guess a Firewall Rule to block port 80 on these bad ip addresses
Is that what you are thinking?
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
I agree with Dave's recommendations.
Dan
Dan
ASKER
Dan here is a bit of the log from the time of the last error
2015-08-03 00:05:45 2015-08-02 20:05:45 66.249.73.131 NULL music4.asp
2015-08-03 00:48:54 2015-08-02 20:48:54 180.76.15.28 NULL
2015-08-03 00:50:01 2015-08-02 20:50:01 180.76.15.10 NULL
2015-08-03 01:07:44 2015-08-02 21:07:44 66.249.73.139 NULL music4.asp
2015-08-03 01:38:45 2015-08-02 21:38:45 66.249.73.139 NULL music4.asp
2015-08-03 01:51:18 2015-08-02 21:51:18 180.76.15.160 NULL &amp;amp;trackfilename &amp;a mp
2015-08-03 02:09:37 2015-08-02 22:09:37 180.76.15.33 NULL
2015-08-03 02:09:45 2015-08-02 22:09:45 66.249.73.131 NULL music4.asp
2015-08-03 02:11:29 2015-08-02 22:11:29 180.76.15.26 NULL
2015-08-03 02:13:06 2015-08-02 22:13:06 141.8.143.204 NULL robots.txt
2015-08-03 02:20:15 2015-08-02 22:20:15 66.249.65.237 NULL robots.txt
2015-08-03 02:20:15 2015-08-02 22:20:15 66.249.65.230 NULL music4.asp
2015-08-03 02:22:04 2015-08-02 22:22:04 141.8.143.204 NULL music4.asp
2015-08-03 02:41:15 2015-08-02 22:41:15 66.249.73.139 NULL music4.asp
You can see that the 180 ip addresses are from China.
Thoughts
2015-08-03 00:05:45 2015-08-02 20:05:45 66.249.73.131 NULL music4.asp
2015-08-03 00:48:54 2015-08-02 20:48:54 180.76.15.28 NULL
2015-08-03 00:50:01 2015-08-02 20:50:01 180.76.15.10 NULL
2015-08-03 01:07:44 2015-08-02 21:07:44 66.249.73.139 NULL music4.asp
2015-08-03 01:38:45 2015-08-02 21:38:45 66.249.73.139 NULL music4.asp
2015-08-03 01:51:18 2015-08-02 21:51:18 180.76.15.160 NULL &amp;amp;trackfilename
2015-08-03 02:09:37 2015-08-02 22:09:37 180.76.15.33 NULL
2015-08-03 02:09:45 2015-08-02 22:09:45 66.249.73.131 NULL music4.asp
2015-08-03 02:11:29 2015-08-02 22:11:29 180.76.15.26 NULL
2015-08-03 02:13:06 2015-08-02 22:13:06 141.8.143.204 NULL robots.txt
2015-08-03 02:20:15 2015-08-02 22:20:15 66.249.65.237 NULL robots.txt
2015-08-03 02:20:15 2015-08-02 22:20:15 66.249.65.230 NULL music4.asp
2015-08-03 02:22:04 2015-08-02 22:22:04 141.8.143.204 NULL music4.asp
2015-08-03 02:41:15 2015-08-02 22:41:15 66.249.73.139 NULL music4.asp
You can see that the 180 ip addresses are from China.
Thoughts
ASKER
Guys
I now have a PowerShell script that extracts the Event Id 1309 ip address adds it to the Firewall block_IP rule I setup to block ports 80 and 443
It gives me a report of the nslookup of that ip address which I then can determine if I need to add another rebot
I attached it to the event Id as a Task
Will monitor this .
Thanks again
I now have a PowerShell script that extracts the Event Id 1309 ip address adds it to the Firewall block_IP rule I setup to block ports 80 and 443
It gives me a report of the nslookup of that ip address which I then can determine if I need to add another rebot
I attached it to the event Id as a Task
Will monitor this .
Thanks again
ASKER
Guys
Update
I am still getting this error 1309
Todays ip address was 157.55.39.198 which is again MSNBOT
This is my robots.txt file
I do have a PowerShell script that takes the ip address from the event error and adds it to my firewall rule that blocks ports 80 and 443 so that ip address will not be able to access my site again
But I thought the ROBOTS.TXT file entries would block them first?
Thoughts
Update
I am still getting this error 1309
Todays ip address was 157.55.39.198 which is again MSNBOT
This is my robots.txt file
User-agent: MJ12bot/v1.4.5
Disallow: /mp3/mp3musicalablums/
User-agent: msnbot
Disallow: /mp3/mp3musicalablums/
User-agent: googlebot
Disallow: /mp3/mp3musicalablums/
User-agent: Baiduspider
Disallow: /mp3/mp3musicalablums/
I do have a PowerShell script that takes the ip address from the event error and adds it to my firewall rule that blocks ports 80 and 443 so that ip address will not be able to access my site again
But I thought the ROBOTS.TXT file entries would block them first?
Thoughts
Thomas,
The msnbot user-agent string is incomplete. There are a few different ones. As with every other bot flying around the web. Here is a list of known bots.
link: http://www.useragentstring.com/pages/Crawlerlist/
The MSNbot has the following user-agent names:
msnbot/2.1
msnbot/2.0b
msnbot/1.1
msnbot/1.0
msnbot/0.9
msnbot/0.11
MSNBOT/0.1
The best way to grab the user-agent is from the http logs. As long as you are logging the user-agent field in the logs.
Dan
The msnbot user-agent string is incomplete. There are a few different ones. As with every other bot flying around the web. Here is a list of known bots.
link: http://www.useragentstring.com/pages/Crawlerlist/
The MSNbot has the following user-agent names:
msnbot/2.1
msnbot/2.0b
msnbot/1.1
msnbot/1.0
msnbot/0.9
msnbot/0.11
MSNBOT/0.1
The best way to grab the user-agent is from the http logs. As long as you are logging the user-agent field in the logs.
Dan
ASKER
Dan
Thanks
This one was bingbot which is BingBot 2.0
Should I add all the MSNbots too?
Here is the IIS Log which I am logging the user-agent
2015-08-07 08:18:26 10.2.8.23 GET /robots.txt - 80 - 157.55.39.198 Mozilla/5.0+(compatible;+b ingbot/2.0 ;++http://www.bing.com/bingbot.htm) - 200 0 0 113
2015-08-07 08:19:01 10.2.8.23 GET /mp3/mp3musicalbums/&a mp;amp;art istname&am p;amp;amp; /&amp; amp;record ingtitle&a mp;amp;amp ;/&amp ;amp;track filename&a mp;amp;amp - 80 - 157.55.39.198 Mozilla/5.0+(compatible;+b ingbot/2.0 ;++http://www.bing.com/bingbot.htm) - 400 0 0 556
I need to work on my logparser to extract the correct info from the IIS logs
Thoughts
Thanks
This one was bingbot which is BingBot 2.0
Should I add all the MSNbots too?
Here is the IIS Log which I am logging the user-agent
2015-08-07 08:18:26 10.2.8.23 GET /robots.txt - 80 - 157.55.39.198 Mozilla/5.0+(compatible;+b
2015-08-07 08:19:01 10.2.8.23 GET /mp3/mp3musicalbums/&a
I need to work on my logparser to extract the correct info from the IIS logs
Thoughts
in your robots.txt you can specify where to search and where not to search. The robots are simply following your links and you haven't given them a sitemap or anything to go by.
ASKER
Dave
Oops I see it now the path is misspelled my fat finders again
Oops I see it now the path is misspelled my fat finders again
Thomas, the big question which David alluded to, is where is this URL coming from? Somehow this URL has now been picked up by several crawlers. So it would be useful to try to find this link.
Oddly enough, when I crawl your site, I see only a single page.
Maybe you could try creating a sitemap.xml, drop it in the root of your site and see if this helps.
Online tool to create a site map: https://www.xml-sitemaps.com/
Dan
Oddly enough, when I crawl your site, I see only a single page.
Maybe you could try creating a sitemap.xml, drop it in the root of your site and see if this helps.
Online tool to create a site map: https://www.xml-sitemaps.com/
Dan
ASKER
Dan
Great just created the sitemap for the site.
Placed it in the root of the site
Do I need to do an IISRESET /noforce?
When I update the robots.,txt do I need to do a IISRESET /noforce ?
Thanks
Great just created the sitemap for the site.
Placed it in the root of the site
Do I need to do an IISRESET /noforce?
When I update the robots.,txt do I need to do a IISRESET /noforce ?
Thanks
No iisreset required.
ASKER
Dan
Thanks
Thanks
ASKER
Guys
This one popped up
2015-08-11 23:44:24 W3SVC4 TGCS013 10.2.8.23 GET /page2.asp - 80 - 157.55.39.227 Mozilla/5.0+(compatible;+b ingbot/2.0 ;++http://www.bing.com/bingbot.htm) - www.tomsmp3.com 404 0 2 175
2015-08-11 23:44:24 W3SVC4 TGCS013 10.2.8.23 GET /music4.asp - 80 - 157.55.39.227 Mozilla/5.0+(compatible;+b ingbot/2.0 ;++http://www.bing.com/bingbot.htm) - www.tomsmp3.com 200 0 0 244
2015-08-11 23:44:24 W3SVC4 TGCS013 10.2.8.23 GET /SelectAlbums.asp |39|80040e14|Incorrect_syn tax_near_' )'. 80 - 157.55.39.227 Mozilla/5.0+(compatible;+b ingbot/2.0 ;++http://www.bing.com/bingbot.htm) - www.tomsmp3.com 500 0 0 176
2015-08-11 23:44:24 W3SVC4 TGCS013 10.2.8.23 GET /&song& - 80 - 157.55.39.227 Mozilla/5.0+(compatible;+b ingbot/2.0 ;++http://www.bing.com/bingbot.htm) - www.tomsmp3.com 400 0 0 423
In my current robots.txt
Thoughts
This one popped up
2015-08-11 23:44:24 W3SVC4 TGCS013 10.2.8.23 GET /page2.asp - 80 - 157.55.39.227 Mozilla/5.0+(compatible;+b
2015-08-11 23:44:24 W3SVC4 TGCS013 10.2.8.23 GET /music4.asp - 80 - 157.55.39.227 Mozilla/5.0+(compatible;+b
2015-08-11 23:44:24 W3SVC4 TGCS013 10.2.8.23 GET /SelectAlbums.asp |39|80040e14|Incorrect_syn
2015-08-11 23:44:24 W3SVC4 TGCS013 10.2.8.23 GET /&song& - 80 - 157.55.39.227 Mozilla/5.0+(compatible;+b
In my current robots.txt
User-agent: MJ12bot/v1.4.5
Disallow: /mp3/mp3musicalbums/
User-agent: Bingbot
Disallow: /mp3/mp3musicalbums/
User-agent: Bingbot/2.0
Disallow: /mp3/mp3musicalbums/
User-agent: msnbot
Disallow: /mp3/mp3musicalbums/
User-agent: msnbot/2.1
Disallow: /mp3/mp3musicalbums/
User-agent: msnbot/2.0b
Disallow: /mp3/mp3musicalbums/
User-agent: msnbot/1.1
Disallow: /mp3/mp3musicalbums/
User-agent: msnbot/1.0
Disallow: /mp3/mp3musicalbums/
User-agent: msnbot/0.9
Disallow: /mp3/mp3musicalbums/
User-agent: msnbot/0.11
Disallow: /mp3/mp3musicalbums/
User-agent: msnbot/0.1
Disallow: /mp3/mp3musicalbums/
User-agent: msrbot
Disallow: /mp3/mp3musicalbums/
User-agent: googlebot
Disallow: /mp3/mp3musicalbums/
User-agent: Baiduspider
Disallow: /mp3/mp3musicalbums/
Thoughts
Thomas, you are going to have to live with undesired hits on your website. Its just life with a website that allows public access or has content that is accessible to the anonymous user.
You will never be able to catch/block 100% of these bad requests.
As for the hits above:
1. a 404 is a file not found. A normal occurrence.
2. a 200 is a valid http request that was successfully fulfilled. Not an error. The webpage "music4.asp" is indexed everywhere in Bing, Google, Duckduckgo...
3. a 500 is a server side error. Probably a code issue on your side.
4. a 400 is a bad request most likely due to a bad URL.
If I were to list all of the 4xx http error codes I see on my sites, on a daily basis, I would get dizzy and probably faint.
What I have implemented is an error handler page that catches all 4xx and 5xx level errors, reports/logs the info and then displays or redirects to a friendly html page saying the content was not found and supplying links back to the main site.
Dan
You will never be able to catch/block 100% of these bad requests.
As for the hits above:
1. a 404 is a file not found. A normal occurrence.
2. a 200 is a valid http request that was successfully fulfilled. Not an error. The webpage "music4.asp" is indexed everywhere in Bing, Google, Duckduckgo...
3. a 500 is a server side error. Probably a code issue on your side.
4. a 400 is a bad request most likely due to a bad URL.
If I were to list all of the 4xx http error codes I see on my sites, on a daily basis, I would get dizzy and probably faint.
What I have implemented is an error handler page that catches all 4xx and 5xx level errors, reports/logs the info and then displays or redirects to a friendly html page saying the content was not found and supplying links back to the main site.
Dan
ASKER
Dan thanks for the insite.
ASKER
First thank you for responding
Where do I place that code? the web.config file? if so how do I add it there?
This is my current web.config in this sites folder
Open in new window
Am I on the right track here?