Solved

Huge inbound traffic on Dedicated server (WHM/CentOS) DDoS

Posted on 2012-03-25
28
1,259 Views
Last Modified: 2012-06-27
Dear Guys,

You are my last hope.

I have a dedicated server with 100Mbps unmetered bandwidth and somebody is blocking my server for hours in every day with inbound traffic what is more than 100Mbps. If my server is under attack i can't login to whm or ssh i just can reboot my server but if server is rebooted the attack immediately continue so i can't do anything.

The huge inbound traffic using port 80.

I installed and configured ConfigServer Security&Firewall and DDOS Deflate and mod_evasive but these didn't solved the problem.

On my websites on my server have fileupload input form and the maximum filesize in php.ini is 2MB and the max_input_time is 30.

Can You suggest me something to solve this problem?

(Oh, and my host sad i should buy new ip for my server... and nothing else)
 The Bandwidth of the serverThank You very much!
0
Comment
Question by:RexozisT
  • 13
  • 8
  • 4
  • +1
28 Comments
 
LVL 5

Expert Comment

by:1ly4me
Comment Utility
Seems like you have some code leak in your web site, that allows attacker to flood with HTTP requests. I guess attacker is try to upload some file through website. If you have access to SSH, block the port 80 and check httpd log file.
0
 
LVL 38

Expert Comment

by:Rich Rumble
Comment Utility
This could be a number of things, Slowloris (http://ha.ckers.org/slowloris/), LOIC (http://en.wikipedia.org/wiki/Low_Orbit_Ion_Cannon) or many others. It could also be some sort of "legitimate" dos where a large website has misspelled a URL or Img. I've seen this a few times in Ad's, where the banner image gets the wrong URL, you should try to packet capture it to see if you can see what URL they are going after.
Also your ISP would have to deal with a DoS of that size, there is nothing you can do if the pipe is filling up, you can't send RST packets back if the path is already full, sort of like trying to go up stairs when they are full of people coming down. Your ISP should have some Quality of Service terms, or a security team that you can deal with.
-rich
0
 

Author Comment

by:RexozisT
Comment Utility
Dear 1ly4me and richrumble,

Thank you for the reply!

I don't think the attack is through website because if my server is under attack and i reboot the server the attack is continue immediately after reboot.

I'm beginner in dedicated server topci so can you tell me how can i disable port 80 in SSH?

Dear richrumble, can you tell me how can i packet capture?

My host absulutely can't help me. I wrote 5 letters in this topic and the answers were useless.

Otherwise  my server is currently under attack...
Server Traffic
0
 
LVL 38

Expert Comment

by:Rich Rumble
Comment Utility
Are you sure it's DoS or just your own users? http://www.experts-exchange.com/Software/Server_Software/Web_Servers/Apache/Q_27474305.html
Wordpress is a very vulnerable platform, I'd suggest you harden it as much as possible. There have been 9 or more (published) exploits for wordpress in 2012 alone (so far)
http://www.exploit-db.com/search/?action=search&filter_page=1&filter_description=wordpress
I wouldn't use wordpress myself, but if you must I've heard these are good places to start looking for securing it:
http://www.websitedefender.com/secure-wordpress-plugin/
http://www.wpsecuritylock.com/
http://wordpress.org/search/security
Make sure it's an attack (something Cacti/SNMP can't tell you), look at the Apache log's, or whatever webserver is in use. I'd call the ISP (not email) and ask to speak to a legal representative/council/attorney for clarification on their policies involving DoS attacks and quality assurance.
-rich
0
 
LVL 5

Expert Comment

by:1ly4me
Comment Utility
Based on your assumptions and graphs , I think the server is misconfigured and or the recent change in the server configuration causing the trouble.
As you are beginner, this needs lot of knowledge about Linux and services.
I will tell you to check log files of all services.
Generally log files will be on this directory. /var/log/

(In an extreme case, you are under attack by some serious hacker. They are running the app or code to flood your server - You need check you log files before proceed to any change)
0
 

Author Comment

by:RexozisT
Comment Utility
Thank you for replies!

I use wordpress for half year ago and fortunately i didn't have problem before.
I use my current server for half year ago also and before last week i didn't have this traffic problem.
DatatrafficNow i can't login to my server (attack is in progress) but as i can i will check log files.

Thank you very much!
0
 

Author Comment

by:RexozisT
Comment Utility
I'v checked my access_log and i found this:

So my server was working fine but after ~14:40 the incoming traffic was high and before 14:40 there arent anything bad in my access log but at 14:44:

188.6.112.148 - - [26/Mar/2012:14:44:28 +0200] "GET / HTTP/1.1" 200 111
188.6.112.148 - - [26/Mar/2012:14:44:28 +0200] "GET /favicon.ico HTTP/1.1" 404 1993
188.6.112.148 - - [26/Mar/2012:14:44:28 +0200] "GET /cgi-sys/defaultwebpage.cgi HTTP/1.1" 200 3323
188.6.112.148 - - [26/Mar/2012:14:44:28 +0200] "GET /img-sys/bg.jpg HTTP/1.1" 200 508
188.6.112.148 - - [26/Mar/2012:14:44:28 +0200] "GET /img-sys/contentbox.jpg HTTP/1.1" 200 8846
188.6.112.148 - - [26/Mar/2012:14:44:28 +0200] "GET /img-sys/header.jpg HTTP/1.1" 200 19080
127.0.0.1 - - [26/Mar/2012:14:44:31 +0200] "OPTIONS * HTTP/1.0" 200 -
188.6.112.148 - - [26/Mar/2012:14:44:31 +0200] "GET /favicon.ico HTTP/1.1" 404 199

94.44.148.167 - - [26/Mar/2012:14:49:34 +0200] "-" 408 -
89.186.101.135 - - [26/Mar/2012:14:49:34 +0200] "-" 408 
87.229.83.25 - - [26/Mar/2012:14:52:06 +0200] "-" 408 
81.182.20.137 - - [26/Mar/2012:14:55:01 +0200] "\x16\x03\x01" 404 -
81.182.20.137 - - [26/Mar/2012:14:55:01 +0200] "\x16\x03" 404 
81.182.20.137 - - [26/Mar/2012:14:55:29 +0200] "\x16\x03" 404 
81.182.20.137 - - [26/Mar/2012:14:55:31 +0200] "\x16\x03" 404 

212.51.111.137 - - [26/Mar/2012:14:58:02 +0200] "GET / HTTP/1.1" 200 111
212.51.111.137 - - [26/Mar/2012:14:58:02 +0200] "GET /favicon.ico HTTP/1.1" 404 2005
212.51.111.137 - - [26/Mar/2012:14:58:02 +0200] "GET /cgi-sys/defaultwebpage.cgi HTTP/1.1" 200 3323
212.51.111.137 - - [26/Mar/2012:14:58:02 +0200] "GET /img-sys/bg.jpg HTTP/1.1" 200 508
212.51.111.137 - - [26/Mar/2012:14:58:04 +0200] "GET /img-sys/header.jpg HTTP/1.1" 200 19080
212.51.111.137 - - [26/Mar/2012:14:58:04 +0200] "GET /img-sys/contentbox.jpg HTTP/1.1" 200 884

81.182.20.137 - - [26/Mar/2012:14:59:22 +0200] "\x16\x03" 404 -
81.182.20.137 - - [26/Mar/2012:14:59:22 +0200] "\x16\x03" 404 

81.182.20.137 - - [26/Mar/2012:15:00:06 +0200] "\x16\x03" 404 

81.182.20.137 - - [26/Mar/2012:15:00:16 +0200] "\x16\x03" 404 
81.182.20.137 - - [26/Mar/2012:15:03:24 +0200] "\x16\x03" 404 

81.182.20.137 - - [26/Mar/2012:15:35:24 +0200] "\x16\x03" 404 

81.182.20.137 - - [26/Mar/2012:15:39:38 +0200] "\x16\x03" 404 

81.182.20.137 - - [26/Mar/2012:15:39:44 +0200] "\x16\x03" 404 


And in my error_log:

[Mon Mar 26 15:00:06 2012] [error] [client 81.182.20.137] Invalid method in request \x16\x03
[Mon Mar 26 15:00:06 2012] [error] [client 81.182.20.137] File does not exist: /usr/local/apache/htdocs/501.shtml
[Mon Mar 26 15:00:16 2012] [error] [client 81.182.20.137] Invalid method in request \x16\x03
[Mon Mar 26 15:00:16 2012] [error] [client 81.182.20.137] File does not exist: /usr/local/apache/htdocs/501.shtm

...

[Mon Mar 26 15:39:38 2012] [error] [client 81.182.20.137] Invalid method in request \x16\x03
[Mon Mar 26 15:39:38 2012] [error] [client 81.182.20.137] File does not exist: /usr/local/apache/htdocs/501.shtml
[Mon Mar 26 15:39:44 2012] [error] [client 81.182.20.137] Invalid method in request \x16\x03
[Mon Mar 26 15:39:44 2012] [error] [client 81.182.20.137] File does not exist: /usr/local/apache/htdocs/501.shtm

Open in new window


Is it normal?
0
 
LVL 38

Expert Comment

by:Rich Rumble
Comment Utility
81.182.20.137 is an ADSL customer according to a whois: http://whois.arin.net/rest/net/NET-81-0-0-0-1/pft and they probably are trying a DoS attack of some kind: http://forums.freebsd.org/showthread.php?t=10643
If you have PHPMyAdmin or other "GUI" interfaces (like tomcat)installed I'd suggest you secure them. (rpm -qa |grep -i phpmyadmin)
Http 200 is fine, these are very normal. 408 is a timeout, and a 404 is "resource does not exist". Read an Apache hardening guide also, just in case you have a default install.
http://httpd.apache.org/docs/2.0/misc/security_tips.html
-rich
0
 
LVL 5

Expert Comment

by:1ly4me
Comment Utility
No, there is an abnormal activity on your server, which is all clients are trying to access SSL port, but you have no SSL running on port 443 or SSL service is using non-standard port

Also check error_log file
0
 
LVL 13

Expert Comment

by:LinuxGuru
Comment Utility
Hi,

We need to find the number of incoming requests and connections for each ips to determine if there is a ddos attack.

Attach the output of the following commands.

netstat -plan | grep ":80"

Open in new window


Above will output the ips connected to port 80.

If there is a particular ip sending connections we can just block that ip in firewall or route the ip to reject mode and this will hopefully resolve the issue.

Let me know the status.

FYI:

To block the ip in firewall use the following command.
iptables -I INPUT -s IP -j DROP

service iptables save

Open in new window


Sometimes blocking will not work and we need to route the ip. Use the following command to block attackers ip with null route.

route add -host IP-ADDRESS reject

Open in new window

0
 

Author Comment

by:RexozisT
Comment Utility
Dear Guys,

Thank You very much for help!

Now i'm using cloudflare and it is very good, but didn't solve my problem.

Now i'm under icmp attack from only 1 ip address but i can't block.

I'm using csf firewall and i disabled icmp and added some ip table rules for the attacker ip but tcpdump show the attack is on and my incoming data is high.

How can i block this ip?
0
 

Author Comment

by:RexozisT
Comment Utility
Or how can i block all icmp connection?
0
 
LVL 38

Expert Comment

by:Rich Rumble
Comment Utility
Depending on where your sniffer is, you could see the attacks even if the firewall is "blocking", and blocking is more like ignoring. ICMP is exceptionally easy to spoof, and typically not a connection killer, esp from one IP. If you see the ICMP coming into the firewall on the inside, then it's allowing ICMP through, there are also various levels of ICMP that it may or may not allow in, most block all but some firewalls block only certain kinds of icmp packets.
-rich
0
 
LVL 13

Expert Comment

by:LinuxGuru
Comment Utility
Hi,

Did you try a ipaddress route reject for that ip. I have already mentioned about this in my previous reply .

It's not a bad idea to block all ICMP, but it can be difficult to diagnose problems if they come-up.

From your reply, you are facing issue from a particular ip only, so I would recommend you to block using ip address routing.

Let me know the results.
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 

Author Comment

by:RexozisT
Comment Utility
Thank you very much for trying to help me!

I googled for ip route reject and i found this:

http://www.cyberciti.biz/tips/how-do-i-drop-or-block-attackers-ip-with-null-routes.html

and tried route add {BADIP} gw 127.0.0.1 lo

and

route add -host {BADIP} reject

but didn't helped

and i also tried

iptables -I INPUT -s {BADIP} -j DROP

but didnt helped :(

And after the last iptable rule i can see this:

1     928K   60G DROP       all  --  *      *       {BADIP]         0.0.0.0/0      

If i run tcpdump i can see just one ip so im sure there are just one attacker ip.
0
 

Author Comment

by:RexozisT
Comment Utility
My last 24 hours....Traffic
0
 
LVL 13

Expert Comment

by:LinuxGuru
Comment Utility
Ok.

Can you try the following?

route add 10.10.10.1 reject

Open in new window


Replace 10.10.10.1 with the attackers ip.
0
 
LVL 13

Expert Comment

by:LinuxGuru
Comment Utility
After executing the above command run the following and paste the output.

netstat -nr

Open in new window

0
 

Author Comment

by:RexozisT
Comment Utility
Thank you for help, but i already did this (route add 10.10.10.1 reject) and didn't solve the problem. :(

I'm disappointed... I can't solve this?
0
 

Author Comment

by:RexozisT
Comment Utility
Can i disable all icmp connections?
0
 
LVL 13

Expert Comment

by:LinuxGuru
Comment Utility
Yes, you can disable. But whats the output of the following command?

netstat -nr

Open in new window

0
 

Author Comment

by:RexozisT
Comment Utility
The output of netstat -nr:

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
{attackerip}    127.0.0.1       255.255.255.255 UGH       0 0          0 lo
{attackerip}    -                     255.255.255.255 !H        - -          - -
{myserverip}   0.0.0.0         255.255.255.255 UH        0 0          0 eth0
{myserverip}  0.0.0.0         255.255.255.255 UH        0 0          0 eth0
{myserverip}  0.0.0.0         255.255.255.192 U         0 0          0 eth0
{myserverip}    0.0.0.0         255.255.255.0   U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
0.0.0.0           {myserverip}   0.0.0.0         UG        0 0          0 eth0

Open in new window


Can you tell me how can i disable icmp?
0
 
LVL 13

Accepted Solution

by:
LinuxGuru earned 500 total points
Comment Utility
Hi,

You can refer the following url to disable icmp requests.

http://www.lifelinux.com/how-to-disable-ping-responses-in-linux/

Before you disable the ping, kindly try the following.

It seems the attackers ip listed in the 1st line is able to access the server.

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
{attackerip}    127.0.0.1       255.255.255.255 UGH       0 0          0 lo

Open in new window


So please execute the following.

ip route delete {attackerip}

Open in new window


Replace attacker ip with the ip listed in the 1st line.

Then readd the ip.

route add {attackerip} reject

Open in new window


I am recommending routing only because the same ddos attack was blocked some days ago on our servers using the same.

After routing the ip the output should look like below. In my example i have blocked the ip 10.100.10.101.

root@server@~ [07:06:27]> netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.100.10.101   -               255.255.255.255 !H        - -          - -
192.0.2.0       0.0.0.0         255.255.255.0   U         0 0          0 venet0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 venet0
0.0.0.0         0.0.0.0         0.0.0.0         U         0 0          0 venet0

Open in new window


The Flag should be !H. So please try to remove the attackers ip and readd as mentioned above.

Cheers!!!
0
 

Author Comment

by:RexozisT
Comment Utility
Dear testez!

Thank You very much for the help!

Now the netstat -nr output:

Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
{attackerip}       -               255.255.255.255 !H        - -          - -
{myserverip}          0.0.0.0         255.255.255.255 UH        0 0          0 eth0
{myserverip}  0.0.0.0         255.255.255.255 UH        0 0          0 eth0
{myserverip}  0.0.0.0         255.255.255.192 U         0 0          0 eth0
{myserverip}    0.0.0.0         255.255.255.0   U         0 0          0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
0.0.0.0         {myserverip}  0.0.0.0         UG        0 0          0 eth0

Open in new window


I will inform You about the results immediately if the attacker trying again!

Thank You again!
0
 
LVL 13

Expert Comment

by:LinuxGuru
Comment Utility
RexozisT,

No problem..You are always welcome.

If you face an difficulty, just let us know.
0
 

Author Comment

by:RexozisT
Comment Utility
So, today i wasn't realize any problem with my server so maybe you solved the problem! :)

I will waiting for 2-4 days and i will inform you about what happened. :)

Thank you very much!
0
 
LVL 13

Expert Comment

by:LinuxGuru
Comment Utility
RexozisT,

Glad to know that everything works fine now.

:)
0
 

Author Closing Comment

by:RexozisT
Comment Utility
Thank You very much! This solved the problem! :)
0

Featured Post

How to run any project with ease

Manage projects of all sizes how you want. Great for personal to-do lists, project milestones, team priorities and launch plans.
- Combine task lists, docs, spreadsheets, and chat in one
- View and edit from mobile/offline
- Cut down on emails

Join & Write a Comment

This subject  of securing wireless devices conjures up visions of your PC or mobile phone connecting to the Internet through some hotspot at Starbucks. But it is so much more than that. Let’s look at the facts: devices#sthash.eoFY7dic.
Envision that you are chipping away at another e-business site with a team of pundit developers and designers. Everything seems, by all accounts, to be going easily.
Illustrator's Shape Builder tool will let you combine shapes visually and interactively. This video shows the Mac version, but the tool works the same way in Windows. To follow along with this video, you can draw your own shapes or download the file…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

728 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

9 Experts available now in Live!

Get 1:1 Help Now