skeed
asked on
bot net get request attack mitigation?
It seems we have a bot net of roughly 10000 computers sending hudreds of the following request per second to one of our customer's web servers:
[30/Apr/2004:17:16:56 -0700] "GET
/AAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAA-
which comes with an accompanying error_log entry as follows:
Fri Apr 30 17:18:59 2004] [error] [client x.x.x.x] (36)File name too
long: access to /AAAAAAAAAAAAAAAAAAAAAAAAA AAAAA denied
We have tried writing a script to detect these bots on the fly and add them to the server's iptables, but the server quickly becomes overwhelmed with 1500+ iptables entries. We have also tried adding ACLs at our core switches in order to help mitigate the attack, but unfortunately they have a 500 ACL limit. The ips of the bots seem to sufficiently change every hour on the hour such that it makes us believe there are nearly an unlimited amount of bots at this person(s) disposal.
Anyone ever seen something like this before and/or have advice as to how to combat it? Could this be an apache related problem as well relative to apache responding in an inefficient manner to the unusually long urls?
Thanks.
[30/Apr/2004:17:16:56 -0700] "GET
/AAAAAAAAAAAAAAAAAAAAAAAAA
which comes with an accompanying error_log entry as follows:
Fri Apr 30 17:18:59 2004] [error] [client x.x.x.x] (36)File name too
long: access to /AAAAAAAAAAAAAAAAAAAAAAAAA
We have tried writing a script to detect these bots on the fly and add them to the server's iptables, but the server quickly becomes overwhelmed with 1500+ iptables entries. We have also tried adding ACLs at our core switches in order to help mitigate the attack, but unfortunately they have a 500 ACL limit. The ips of the bots seem to sufficiently change every hour on the hour such that it makes us believe there are nearly an unlimited amount of bots at this person(s) disposal.
Anyone ever seen something like this before and/or have advice as to how to combat it? Could this be an apache related problem as well relative to apache responding in an inefficient manner to the unusually long urls?
Thanks.
ASKER
Yeah, lucky us. We have seen similar attacks before, but never this powerful. I made the mistake of not making a good authoritative master list of all the "bots", but am doing so now. Based on what I've seen so far though, it's totally random and there are thousands of them.
I'm definitely wondering if Apache could make a more "efficient" response to such a request. Is there an apache hack that might recognize this request in some way and drop it immediately? I've tried doing a "Files" match for it and then denying it but that didn't help either.
I'm also going to try the iptables "string" match option to see if that helps at all.
Any other suggestions would be immensely appreciated.
I'm definitely wondering if Apache could make a more "efficient" response to such a request. Is there an apache hack that might recognize this request in some way and drop it immediately? I've tried doing a "Files" match for it and then denying it but that didn't help either.
I'm also going to try the iptables "string" match option to see if that helps at all.
Any other suggestions would be immensely appreciated.
in iptables you can block such requests, but it's a performance bootleneck 'cause iptables needs to parse each http packet (needs to terminate the request)
in apache you can use mod_rewrite or mod_security to block the request, I'd suggest to simply drop them
Or you need a Web Application Shield. AFAIK, there are no free products now, think about starting with at least $10000,-
Some traditional network firewalls (iptables still mentioned) also have limited posibilities to block it.
Do you need more detailed informations?
in apache you can use mod_rewrite or mod_security to block the request, I'd suggest to simply drop them
Or you need a Web Application Shield. AFAIK, there are no free products now, think about starting with at least $10000,-
Some traditional network firewalls (iptables still mentioned) also have limited posibilities to block it.
Do you need more detailed informations?
ASKER
I suspect that iptables would crumble, having to inspect each and every http packet.
What do you mean simply dropping them? I have tried using a Deny from IP set of statements, but for some reason these long requests bypass them. I'm not sure how or why. I have also tried using mod_security, but can't seem to find the filter that does the trick. What SecFilter regexp should catch the request above? For some reason mod_security can't catch them either.
I have also thought about hacking the apache source code such that it behaves differently when receiving a long filename but am not sure where to look or if this is the right thing to do.
Web application shield is not an option right now, unfortunately.
What do you mean simply dropping them? I have tried using a Deny from IP set of statements, but for some reason these long requests bypass them. I'm not sure how or why. I have also tried using mod_security, but can't seem to find the filter that does the trick. What SecFilter regexp should catch the request above? For some reason mod_security can't catch them either.
I have also thought about hacking the apache source code such that it behaves differently when receiving a long filename but am not sure where to look or if this is the right thing to do.
Web application shield is not an option right now, unfortunately.
SecFilter "/AAA.*"
> What do you mean simply dropping them?
I mean drop the packets, not reject
> What do you mean simply dropping them?
I mean drop the packets, not reject
ASKER
For some reason, that Filter won't match the packets. Does the request have to be valid for it to work? Remember, the requests are have very large filenames that cause apache to throw errors very early on.
did you enable logging or debugging? It tells you what's going on, usually.
SecFilterDefaultAction "deny,log"
SecFilterDebugLog logs/modsec_debug_log
SecFilterDebugLevel 4
or:
SecAuditEngine On
SecAuditLog logs/audit_log
then a filter:
SecFileter AAAA "deny,log,status:500"
and with the malicious request it should produce log messages
SecFilterDefaultAction "deny,log"
SecFilterDebugLog logs/modsec_debug_log
SecFilterDebugLevel 4
or:
SecAuditEngine On
SecAuditLog logs/audit_log
then a filter:
SecFileter AAAA "deny,log,status:500"
and with the malicious request it should produce log messages
ASKER
ahoffmann,
Here's a mod_security log of the attack:
HTTP/0.9 (null) ========================== ========== ==== Request: 80.119.150.31 - - [Sat May 1 13:13:08 2004] "GET
/AAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAA$
AAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAA.h tml^M^M"
403 0 Handler: (null) Error: access to
/AAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAA$
AAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAA.h tml
failed -------------------------- ---------- ---- GET
/AAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAA$
AAAAAAAAAAAAAAAAAAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAAAA AAAAAAAA.h tml^M^M
I tried various regular expression combinations but nothing worked. Could it be that mod_security cannot block it because the URL is too long?
Thanks.
Here's a mod_security log of the attack:
HTTP/0.9 (null) ==========================
/AAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAA
403 0 Handler: (null) Error: access to
/AAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAA
failed --------------------------
/AAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAA
I tried various regular expression combinations but nothing worked. Could it be that mod_security cannot block it because the URL is too long?
Thanks.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Other than a fast upstream filter there's not a lot you can do about this sort of attack and still keep the web server available for normal use. I don't see any new viruses that mention this sort of attack or and new security advisories about Apache, so this may be something completely new.