IRQD Load

We are getting ACK flood from spoofed addresses. And it get the machine's IRQD load 100% for all cores of cpu. But strange thing is TOP command showing cpu usage %0 .
We have closed TSO, GSO ....etc for offloading but nothing change.
Is there any way to solve this better ? Use the hardware cpu if the top is correct  ? or how should i clean the ACK traffic instead of using ACK policier on roouter

PS : Pleas do not write talk to your isp or sth bla bla like that. We need a solution not an advice
FireBallITAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

gheistCommented:
Yes, talk to your ISP to enable state tracking on interface leading to you.
Is this against one machine or all at random? At what rate?
What is your kernel version/distribution (uname -r)
What modules are loaded (lsmod)
What is interrupt rate (cat /proc interrupts ; sleep 10 ; cat /proc interrupts)
fulle dmesg
What the heck is IRQD - i see nothing like that in "top" command on any Linux.
Please post sample of 1000 packets incoming. Without that nobody can guess the attack, and your provider will not even listen to you.
You say to solve it better? But where is your attempt to address the problem if any?
FireBallITAuthor Commented:
Dear gheist thank youo for your answer but please do not write something like talk to your ISP .
We are already an isp and we are getting always same answer if this is a responsibility of an isp we need to solve this

The tcpdump result as given below
tcdump -i p3p1 -Xnvv

00:00:29.571688 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.227.251.112 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 da79 70e3 fb70  E..0.M..u..yp..p
        0x0010:  b214 e592 fbbe 0c38 dcdc ca38 126f 6b16  .......8...8.ok.
        0x0020:  7002 2000 32ce 0000 0204 0578 0101 0402  p...2......x....
00:00:29.571723 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.67.56.11 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 9e7f 7043 380b  E..0.M..u...pC8.
        0x0010:  b214 e592 0fe9 0c38 237d ac1f 0433 1c0c  .......8#}...3..
        0x0020:  7002 2000 1769 0000 0204 0578 0101 0402  p....i.....x....
00:00:29.571758 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.204.30.74 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 b7b7 70cc 1e4a  E..0.M..u...p..J
        0x0010:  b214 e592 fff1 0c38 7027 3b17 b088 2b68  .......8p';...+h
        0x0020:  7002 2000 a944 0000 0204 0578 0101 0402  p....D.....x....
00:00:29.571793 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.222.202.107 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 0b84 70de ca6b  E..0.M..u...p..k
        0x0010:  b214 e592 6153 0c38 6274 557d 64a0 e236  ....aS.8btU}d..6
        0x0020:  7002 2000 2416 0000 0204 0578 0101 0402  p...$......x....
00:00:29.571828 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.120.0.41 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 d62c 7078 0029  E..0.M..u..,px.)
        0x0010:  b214 e592 3221 0c38 e292 d333 1e28 8a21  ....2!.8...3.(.!
        0x0020:  7002 2000 bea9 0000 0204 0578 0101 0402  p..........x....
00:00:29.571868 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.114.55.245 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 9e66 7072 37f5  E..0.M..u..fpr7.
        0x0010:  b214 e592 465b 0c38 3b3b 7d31 a605 cf60  ....F[.8;;}1...`
        0x0020:  7002 2000 a2e6 0000 0204 0578 0101 0402  p..........x....
00:00:29.571908 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.244.181.19 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 20c6 70f4 b513  E..0.M..u...p...
        0x0010:  b214 e592 2873 0c38 c6b5 4179 82fa 4a0f  ....(s.8..Ay..J.
        0x0020:  7002 2000 9bc8 0000 0204 0578 0101 0402  p..........x....
00:00:29.571948 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.11.46.117 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 a84d 700b 2e75  E..0.M..u..Mp..u
        0x0010:  b214 e592 133b 0c38 9d8e 6044 045c f33c  .....;.8..`D.\.<
        0x0020:  7002 2000 1855 0000 0204 0578 0101 0402  p....U.....x....
00:00:29.571972 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.92.201.64 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 0d31 705c c940  E..0.M..u..1p\.@
        0x0010:  b214 e592 8af0 0c38 e7a3 aa0c b935 691c  .......8.....5i.
        0x0020:  7002 2000 46ec 0000 0204 0578 0101 0402  p...F......x....
00:00:29.572012 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.206.150.35 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 3fdc 70ce 9623  E..0.M..u.?.p..#
        0x0010:  b214 e592 67d9 0c38 39e1 561c d425 3a3b  ....g..89.V..%:;
        0x0020:  7002 2000 b252 0000 0204 0578 0101 0402  p....R.....x....
00:00:29.572050 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.201.43.215 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 aa2d 70c9 2bd7  E..0.M..u..-p.+.
        0x0010:  b214 e592 fea0 0c38 c83a af7f f2e4 1e56  .......8.:.....V
        0x0020:  7002 2000 9b45 0000 0204 0578 0101 0402  p....E.....x....
00:00:29.572085 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.8.197.135 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 113e 7008 c587  E..0.M..u..>p...
        0x0010:  b214 e592 0f93 0c38 8442 7b3e b8c0 3a67  .......8.B{>..:g
        0x0020:  7002 2000 87b0 0000 0204 0578 0101 0402  p..........x....
00:00:29.573363 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.97.125.72 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 5924 7061 7d48  E..0.M..u.Y$pa}H
        0x0010:  b214 e592 562d 0c38 ccd7 506a fda0 c263  ....V-.8..Pj...c
        0x0020:  7002 2000 9e5e 0000 0204 0578 0101 0402  p....^.....x....
00:00:29.573414 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.146.80.28 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 861f 7092 501c  E..0.M..u...p.P.
        0x0010:  b214 e592 5355 0c38 1c88 9b3a e5b3 5a21  ....SU.8...:..Z!
        0x0020:  7002 2000 b3e0 0000 0204 0578 0101 0402  p..........x....
00:00:29.573472 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.209.128.143 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 556d 70d1 808f  E..0.M..u.Ump...
        0x0010:  b214 e592 622f 0c38 8fd9 8501 4c09 9f18  ....b/.8....L...
        0x0020:  7002 2000 6bef 0000 0204 0578 0101 0402  p...k......x....
00:00:29.573534 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.170.148.227 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 4140 70aa 94e3  E..0.M..u.A@p...
        0x0010:  b214 e592 afcb 0c38 6ca0 1043 3249 9d2a  .......8l..C2I.*
        0x0020:  7002 2000 bdcb 0000 0204 0578 0101 0402  p..........x....
00:00:29.573590 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.216.241.223 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 e415 70d8 f1df  E..0.M..u...p...
        0x0010:  b214 e592 2ea5 0c38 0d68 ef63 fbb2 5037  .......8.h.c..P7
        0x0020:  7002 2000 e568 0000 0204 0578 0101 0402  p....h.....x....
00:00:29.573660 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.36.214.35 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 0086 7024 d623  E..0.M..u...p$.#
        0x0010:  b214 e592 efed 0c38 3e59 8358 dfcc 213a  .......8>Y.X..!:
        0x0020:  7002 2000 c68d 0000 0204 0578 0101 0402  p..........x....
00:00:29.573706 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.186.223.241 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 f621 70ba dff1  E..0.M..u..!p...
        0x0010:  b214 e592 782e 0c38 08be b23d d2e2 d478  ....x..8...=...x
        0x0020:  7002 2000 944a 0000 0204 0578 0101 0402  p....J.....x....
00:00:29.573772 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.183.98.166 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 7370 70b7 62a6  E..0.M..u.spp.b.
        0x0010:  b214 e592 a964 0c38 3fdc c443 6698 2f2e  .....d.8?..Cf./.
        0x0020:  7002 2000 a8d3 0000 0204 0578 0101 0402  p..........x....
00:00:29.573825 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.151.36.72 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 b1ee 7097 2448  E..0.M..u...p.$H
        0x0010:  b214 e592 a9d6 0c38 1bb5 8c60 2053 df58  .......8...`.S.X
        0x0020:  7002 2000 d904 0000 0204 0578 0101 0402  p..........x....
00:00:29.573882 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.198.94.194 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 7745 70c6 5ec2  E..0.M..u.wEp.^.
        0x0010:  b214 e592 ecc8 0c38 d46f 4a1c 6de4 6e56  .......8.oJ.m.nV
        0x0020:  7002 2000 0864 0000 0204 0578 0101 0402  p....d.....x....
00:00:29.573947 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.177.213.155 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 0081 70b1 d59b  E..0.M..u...p...
        0x0010:  b214 e592 80b0 0c38 90c0 0131 f824 d80a  .......8...1.$..
        0x0020:  7002 2000 965d 0000 0204 0578 0101 0402  p....].....x....
00:00:29.573998 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.243.165.56 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 30a2 70f3 a538  E..0.M..u.0.p..8
        0x0010:  b214 e592 cc1b 0c38 9ced 4754 1708 db02  .......8..GT....
        0x0020:  7002 2000 06e8 0000 0204 0578 0101 0402  p..........x....
00:00:29.574034 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.240.109.125 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 6860 70f0 6d7d  E..0.M..u.h`p.m}
        0x0010:  b214 e592 8b88 0c38 e84a 0479 b9c4 573d  .......8.J.y..W=
        0x0020:  7002 2000 57c0 0000 0204 0578 0101 0402  p...W......x....
00:00:29.574069 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.55.74.48 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 8c66 7037 4a30  E..0.M..u..fp7J0
        0x0010:  b214 e592 ca22 0c38 1c43 8b39 0f1e 7505  .....".8.C.9..u.
        0x0020:  7002 2000 0f52 0000 0204 0578 0101 0402  p....R.....x....
00:00:29.574104 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.201.167.230 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 2e1e 70c9 a7e6  E..0.M..u...p...
        0x0010:  b214 e592 804e 0c38 1126 842e 70b1 ed26  .....N.8.&..p..&
        0x0020:  7002 2000 3351 0000 0204 0578 0101 0402  p...3Q.....x....
00:00:29.574145 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.36.224.158 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 f60a 7024 e09e  E..0.M..u...p$..
        0x0010:  b214 e592 79fe 0c38 6477 5119 f055 c711  ....y..8dwQ..U..
        0x0020:  7002 2000 87c2 0000 0204 0578 0101 0402  p..........x....
00:00:29.574181 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.229.198.243 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 0ef5 70e5 c6f3  E..0.M..u...p...
        0x0010:  b214 e592 6bf6 0c38 e22c 4f08 c564 0936  ....k..8.,O..d.6
        0x0020:  7002 2000 1bdd 0000 0204 0578 0101 0402  p..........x....
00:00:29.574220 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.115.165.31 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 313b 7073 a51f  E..0.M..u.1;ps..
        0x0010:  b214 e592 065c 0c38 226d c83a 2035 8e34  .....\.8"m.:.5.4
        0x0020:  7002 2000 0a7c 0000 0204 0578 0101 0402  p....|.....x....
00:00:29.574256 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.8.184.134 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 1e3f 7008 b886  E..0.M..u..?p...
        0x0010:  b214 e592 c470 0c38 1199 a706 502f 295a  .....p.8....P/)Z
        0x0020:  7002 2000 a053 0000 0204 0578 0101 0402  p....S.....x....
00:00:29.574294 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.127.18.190 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 c390 707f 12be  E..0.M..u...p...
        0x0010:  b214 e592 a150 0c38 c075 d930 be86 505f  .....P.8.u.0..P_
        0x0020:  7002 2000 f261 0000 0204 0578 0101 0402  p....a.....x....
00:00:29.574331 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.201.137.50 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 4cd2 70c9 8932  E..0.M..u.L.p..2
        0x0010:  b214 e592 9bb7 0c38 1610 c976 7708 a654  .......8...vw..T
        0x0020:  7002 2000 2ce5 0000 0204 0578 0101 0402  p...,......x....
00:00:29.574369 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.76.33.234 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 b497 704c 21ea  E..0.M..u...pL!.
        0x0010:  b214 e592 8092 0c38 2727 c828 2162 a73c  .......8''.(!b.<
        0x0020:  7002 2000 f4c4 0000 0204 0578 0101 0402  p..........x....
00:00:29.574403 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.231.89.202 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 7c1c 70e7 59ca  E..0.M..u.|.p.Y.
        0x0010:  b214 e592 8b28 0c38 93d0 7b3f 6dde 9a71  .....(.8..{?m..q
        0x0020:  7002 2000 5242 0000 0204 0578 0101 0402  p...RB.....x....
00:00:29.574446 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.80.16.45 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 c650 7050 102d  E..0.M..u..PpP.-
        0x0010:  b214 e592 cf7f 0c38 7c52 027d cc87 8b2f  .......8|R.}.../
        0x0020:  7002 2000 98f8 0000 0204 0578 0101 0402  p..........x....
00:00:29.574479 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.253.106.220 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 6af4 70fd 6adc  E..0.M..u.j.p.j.
        0x0010:  b214 e592 5c60 0c38 bfc9 3824 8bb0 5371  ....\`.8..8$..Sq
        0x0020:  7002 2000 b032 0000 0204 0578 0101 0402  p....2.....x....
00:00:29.574516 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.83.52.211 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 a1a7 7053 34d3  E..0.M..u...pS4.
        0x0010:  b214 e592 053c 0c38 5401 ce34 4820 c252  .....<.8T..4H..R
        0x0020:  7002 2000 e870 0000 0204 0578 0101 0402  p....p.....x....
00:00:29.574551 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
    112.194.185.192 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 1c4b 70c2 b9c0  E..0.M..u..Kp...
        0x0010:  b214 e592 4b7e 0c38 c8c1 486f 906f f042  ....K~.8..Ho.o.B
        0x0020:  7002 2000 b797 0000 0204 0578 0101 0402  p..........x....
00:00:29.574580 IP (tos 0x0, ttl 117, id 45901, offset 41000, flags [+, rsvd], proto TCP (6), length 48)
^C    112.57.231.244 > 178.20.229.146: ip-proto-6
        0x0000:  4500 0030 b34d b405 7506 ee9f 7039 e7f4  E..0.M..u...p9..
        0x0010:  b214 e592 c57f 0c38 e863 0f4e 7d0d dd2d  .......8.c.N}..-
        0x0020:  7002 2000 4fe1 0000 0204 0578 0101 0402  p...O......x....

Open in new window




and the attached image is top output


Our network topology as this :


EX4500 10G uplink (37.123.100.97)  --------------  Cent os Server p3p1 (37.123.100.98)  -------- Routing options   ------  cent os p3p2 (10.255.255.10  )    -----------  SRX 3600  


This traffic does not passing to the p3p2 but all others ok

[root@244 ~]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         97.100.123.37.s 0.0.0.0         UG    100    0        0 p3p1
default         10.255.255.1    0.0.0.0         UG    101    0        0 p3p2
default         178.20.229.145. 0.0.0.0         UG    102    0        0 em1
10.255.255.0    0.0.0.0         255.255.255.248 U     100    0        0 p3p2
37.123.100.96   0.0.0.0         255.255.255.248 U     100    0        0 p3p1
178.20.229.144  10.255.255.1    255.255.255.248 UG    0      0        0 p3p2
178.20.229.144  0.0.0.0         255.255.255.248 U     0      0        0 em1
You have new mail in /var/spool/mail/root

Open in new window



attack is comming for 178.20.229.146 which is not owned ip on this server and it is not hitting any of iptables rules :


# Generated by iptables-save v1.4.21 on Mon Aug 24 03:44:58 2015
*nat
:PREROUTING ACCEPT [18:1395]
:INPUT ACCEPT [2:95]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [1:35]
#-A PREROUTING -i p3p1 -p tcp -m tcp --dport 80 -j DNAT --to-destination 178.20.229.149:3128
#-A POSTROUTING -o p3p1 -j MASQUERADE
COMMIT
# Completed on Mon Aug 24 03:44:58 2015
# Generated by iptables-save v1.4.21 on Mon Aug 24 03:44:58 2015
*filter
:INPUT ACCEPT [64:4763]
:FORWARD ACCEPT [1:35]
:OUTPUT ACCEPT [42:4215]
-A FORWARD -i p3p1 -p udp -m state --state NEW -m recent --set --name DDOS --mask 255.255.255.255 --rsource
-A FORWARD -i p3p1 -p udp -m state --state NEW -m recent --update --seconds 1 --hitcount 20 --name DDOS --mask 255.255.255.255 --rsource -j DROP
-A FORWARD -i p3p1 -o p3p2 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i p3p2 -o p3p1 -j ACCEPT
COMMIT
# Completed on Mon Aug 24 03:44:58 2015

Open in new window

Screenshot-4.png
Screenshot-1.png
FireBallITAuthor Commented:
Interrups as given below :

[root@244 ~]# cat /proc/interrupts ; sleep 10 ; cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      CPU16      CPU17      CPU18      CPU19      CPU20      CPU21      CPU22      CPU23
   0:        136          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      timer
   1:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      i8042
   8:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      rtc0
   9:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
  12:          4          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      i8042
  17:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
  18:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
  19:         88          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1
  20:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb6
  21:        644    1437082          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb5
  23:       1366       4624         15       3602      11754         26        793       4812       5470       1281       5126      14552        611       7216       8607      17387        130         15        258         38       1331       7077        523       2816   IO-APIC-fasteoi   ata_piix
  72:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  73:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  74:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  75:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  76:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  77:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  78:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  79:       3971        108         38        159        176          0         17        415        625       7030         32       2157         44       3933        361       2888       3927          9         23         16        650       1688      12797         11   PCI-MSI-edge      megasas
  80:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  81:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  82:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  83:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  84:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  85:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  86:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  87:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  88:    4079713    2369130     972808       8533    1619266       1528          0     825198     780466     267611     997395     200884     350727     556367     393738    1661310      10735    1515552        430      68193    2652469    1241106          0     156223   PCI-MSI-edge      p3p1-TxRx-0
  89:    2862428     887534         98      82611     162788          0    4241872      43803          0     285397    1008074          0     441857     333915          0    3681773     359025    3869015     688406       5201    1239821       2894      30017      12076   PCI-MSI-edge      p3p1-TxRx-1
  90:     173714      11720          4     766216     824939      61932          0    1409624     724109      85038     169611    3391795     343952      65938    1818875     153824    3861124          0     164584     540388    1248660      47458     981953    4124939   PCI-MSI-edge      p3p1-TxRx-2
  91:     631998          0    6634278      31702      98621          0    1268714     169989     262689    2310410       1902          0    3743872     261593    1207237       8815      66919    3136802          0          0     355053     383039          3      10657   PCI-MSI-edge      p3p1-TxRx-3
  92:         18       3437          0      21588          0   15741033     270980        518    3787301          0          0      31710      48995          0          0          0     343041          0     119278          0      73316          0    1082534      58819   PCI-MSI-edge      p3p1-TxRx-4
  93:     407483     296975       8837     161498    2007095      68643      93257    4991210      31776    3407023     514238          0      38624      41491     143267     300316     139742     671753     557794          0    3269272    3058618     778140          0   PCI-MSI-edge      p3p1-TxRx-5
  94:         30       1067     626845     168194     465681          0     103700    5732392      16145          0     817333    2097419     480667      45987    1044301     334455      38474          2     748913      17962       1603     135493    3113154    4797024   PCI-MSI-edge      p3p1-TxRx-6
  95:         16     793894      54571      76095     652491          0    3115604       2952     299085     387717    1242213     186680     271140     365910      12094      29573    5647265    3805028    3719141          1          0          0     520144          0   PCI-MSI-edge      p3p1-TxRx-7
  96:         16     683805    4170012    1273875     170995      53872     606119     112682     216735    3139524     432963     158475     238128    1204811     373233    1246219    3340262     352651          0     287580    2235268         77     102084     906031   PCI-MSI-edge      p3p1-TxRx-8
  97:    1273831     247946     196205    5044946      92091      25860    1046837     426770    2669053     127168      57731     315620       1803    1549339      81913          3    1459302     512299      72959     697195    2050722        897    2974441       1914   PCI-MSI-edge      p3p1-TxRx-9
  98:    5409099    1594864     949517    2974366       9510          0      99766     975507     247782     609629       8278    2766487     124107     340929    1015964    1943754     125930     787820     299572     692711          0     161476      32225          4   PCI-MSI-edge      p3p1-TxRx-10
  99:    2604110     506189          0    1293866    1244062    1560882        582      50999     326441    3698682    1418068          0     152949     292443       1123       9372     580259      27799     817220         55     358789     337895    5751235          0   PCI-MSI-edge      p3p1-TxRx-11
 100:       1545     840836         44     497600    2038326     104106       4883          3    1142048    1715525    2325380     557514     545471    6746430          0     781382    1808943     492419          0     202547          0      83522     137443    1548361   PCI-MSI-edge      p3p1-TxRx-12
 101:      98651      58524          0          0      97933          0          0    4167315    1837005     487820          0          0    7191893          0     179338          0          0       7472      51420     187020    5230646      33394    1206900       6909   PCI-MSI-edge      p3p1-TxRx-13
 102:      72778    1373813       8757    1210642     544610          0      40863    1109033     671544         15    1012707        546          0    2601799    1763584    2128546     727940     352127     220592    5677507         28       2791     397273      24726   PCI-MSI-edge      p3p1-TxRx-14
 103:    2831309    3600024          0    1426626       3293          0      17060       3768      29944      63885     382915     147200      98564     531209    7386403      51283          0          0    2476064     259741    2179451      35046        397      40261   PCI-MSI-edge      p3p1-TxRx-15
 104:        593       1181       5015      13013      32889         86       2518         47      41557        589       9742     523322       1325     151123      30241      84997      45085     175725       7871      15567       1465      25363       2082       2716   PCI-MSI-edge      p3p1-TxRx-16
 105:       8877      15249       3440       1714      85113         62      12924     553377        142        172        490        792       8351        740       3282      61912      37231         15      89927      31676      10250         79       3903      17451   PCI-MSI-edge      p3p1-TxRx-17
 106:        420      70663       4725        862        314          4     648434         22      19691      11464       1239        141       3020       4561        462       3816      28201       9579      27584      21078         57      18921       4354         43   PCI-MSI-edge      p3p1-TxRx-18
 107:       2395       3644       1158      11593       8114        159      11420        806        590         15       4943      37234       2809       8639       4541     171054       3244       1468      69682      46734        170      18075        300         37   PCI-MSI-edge      p3p1-TxRx-19
 108:       1042        354       1154       1119     560317          5       3056       2597        894       9000        116       6087       3949       1480       3712     229348       9253      10206       1116        363       8351      11701      17878       1018   PCI-MSI-edge      p3p1-TxRx-20
 109:      15556       8247        136       5676      87414        105       2664       1108         79        513        203         25        584     662487        106       1186       1836         22        325      24662      14088      73768       2816        125   PCI-MSI-edge      p3p1-TxRx-21
 110:       4595       6197         61       1206        511        541         52       8472       5400     178742         11          5      13388       2135        141        540       1367          2       8206     683657        500       1157      17600      18863   PCI-MSI-edge      p3p1-TxRx-22
 111:        773       2556     170734         77       2168          3          8     606427       9419        641      10210         18       9478       2556       2988      40768      33414        138         98       6791       5978     152548      79602      22951   PCI-MSI-edge      p3p1-TxRx-23
 112:          3         86          0         10          0        539          0          0         26          0          0          2          0       1416          0          0         24          0         20          5          1         69          0        844   PCI-MSI-edge      p3p1
 113:     415859    2232488     515719         84    2731739          3    1728233      10998    1113110     541338     215517      41221     218303    2077155      21919     677409    4125968    2529957     918350       2884    1190735        191    1533199          0   PCI-MSI-edge      p3p2-TxRx-0
 114:       5686    2300559     772892     496756         59        651       2009       4630    1457451        700      15323      45114        348      94333     270117      15027     738803     425711     912855    4097832        545      69435     445021      62844   PCI-MSI-edge      p3p2-TxRx-1
 115:     202302      92947       8467     183589      18088     137575      32231     159393    1370498        130      60328       1690      56838    1389191    2778633         17      66600         21         80      83096         69       3784    5172488     618772   PCI-MSI-edge      p3p2-TxRx-2
 116:       1540         70        157      17046        653    3731356         79     253553    1022948       8534     123112     114527      87817     962485        283       4013     900581    1313971    6198626     147599      69301     255267       2671    1073114   PCI-MSI-edge      p3p2-TxRx-3
 117:     283688     500928     128141        150      78847      55930     527396     271370     402877      31908     329336         68    2236501     245753      89539     336870    1158890     142627     249922     118728       6054      43658    1049020      70543   PCI-MSI-edge      p3p2-TxRx-4
 118:     559904         25     567130    2645969      24324      51082    2730977     324948         10      55930         22    3864180     142719     247505        739     129387         39      55178         41       6483       1592     237301      74731    9373705   PCI-MSI-edge      p3p2-TxRx-5
 119:       2569    2297606          1    1055088      94357      46231     862406       9966      49824       1966    2135162     657738      32389     127798    1676836      94602      46962     423702       1209     119040     702913     569886      29149       2500   PCI-MSI-edge      p3p2-TxRx-6
 120:     138178        227         17      50252     228656    3125851    1724227        127         29      60384         18    3214481     123009     248494      12042    9298486     178864     244622       7498       3112    2306221        929        790    2675775   PCI-MSI-edge      p3p2-TxRx-7
 121:         16     875648      80169      77280     253218      37380     108190          0    1108906     129592      98628      14229    1366251      76510    5456458     734689      52136     433873     587059      26556         40       3007      68803       2087   PCI-MSI-edge      p3p2-TxRx-8
 122:       3900       1441       1607     285207         83          5    2992792        998    2513921      40002    2106369     109676     448433     317357    3109898      97824    1150707     495784     298935    4479053      88377        559      71809       1745   PCI-MSI-edge      p3p2-TxRx-9
 123:       1004     455481      39217      18522    2473925     109658      57734      17566     751387       3051      97569         10     369168     648341       3673       6782      15026    1486725     519104    1633260        143    1063446       8468     433014   PCI-MSI-edge      p3p2-TxRx-10
 124:      44746      55470       1749      59125    8454810       1351        496       1649     534728     146571      40337     206490       2922     115443     348903       8078     823064    2923241       8024     427531       1380      56214      89367       1545   PCI-MSI-edge      p3p2-TxRx-11
 125:      39752      10672    5124248        609     680170       1614     930548        546     163193        170    1309561       2516     482399     447457    1301699      61151    2754552     316285      53411    1456578    2849035       3097     124245     919521   PCI-MSI-edge      p3p2-TxRx-12
 126:       7712      59866      21793     600152      79315       4522     233266       3867      60724      16212    7961173          0     865127     510105    1012814       6807     374536     461796      42572    3642818         23       2508    2168135         50   PCI-MSI-edge      p3p2-TxRx-13
 127:     596919    4488055         55       6495     585701       4200      48381     231206          1      22552         93     211775     284928    1806544       9857    1623938      33348      11349         16    1093386      36705      39910     833479    8676416   PCI-MSI-edge      p3p2-TxRx-14
 128:     431933     337326      76685      35683     222600      48636        125      74371       1416    5478214       2529      13810     674641     259856    2267098     685640    1837443     835537        306       2202     333227     365143      42217      17010   PCI-MSI-edge      p3p2-TxRx-15
 129:       1610       2156          5    1391667     132424          0    1383413      46033    6727339          5    4524512    1432960     124806     324434     112676       5717     462281     846876     991861    1548290     112774      63229     723810     211044   PCI-MSI-edge      p3p2-TxRx-16
 130:       8352      32121        904    2270695     985532         18    3052160        163    1274802       1712        997         84     497187    1456692     167631    1192899     158600     700669         11      22304      84705     333402     116696   10072529   PCI-MSI-edge      p3p2-TxRx-17
 131:      38135       1876    1157268        233     406138        168     465558       1055    1609893     377548     488647     123467     167237     340859      40193        297    4683077        407     139600    1769045      17571        633     689473       3701   PCI-MSI-edge      p3p2-TxRx-18
 132:      56970        855       1153      19229          0       2072    8646387     124643       4201     888190         10     142631       7140     151126    1903330      49042        731     266954       2427     272879        105    1341744        434       5242   PCI-MSI-edge      p3p2-TxRx-19
 133:       3468        688     130805         31    4131618       3875     808374        363    3314206     594716     863383    5840754     967008    1443958        821    6776261     523197      10995      65592      18168     256985     732888      84350          3   PCI-MSI-edge      p3p2-TxRx-20
 134:       8770    3997791      36196     846003       6993          3       1390        319       2054       2866        154    1227855     328709         69     695176          9       6443     825777      55558     178017        123       2984      53255     485799   PCI-MSI-edge      p3p2-TxRx-21
 135:    3673195    2027021   10324157        210     190188      78456     243163        131          9      37442     252167    1514961     165762      91009       1182    1509551      89152      98454          9     121840     114544    2111683     134585     262448   PCI-MSI-edge      p3p2-TxRx-22
 136:       8702     572701          5     143834       4038       1270        853     745905     279973    1438980    7719553     146580      59017       2285        486      71089    1747221       1534        169        669       2247     271629       3682     150631   PCI-MSI-edge      p3p2-TxRx-23
 137:        157         68          9          9         22          9         97       2341        556          0        694         33        208        288         29       2231       2355        255         33        545        489       8425       2376        443   PCI-MSI-edge      p3p2
 138:       3069        438      55487       1217        661          0    1100986        521       3170      71796     709786      31340      34236     521811       1058      20927      31688       4024      13705     226934     459490      93533      39241       9197   PCI-MSI-edge      em1-0
 139:         10          4        222        630         55          0          1         71        626         26       2000         61          4       8509         70          4         35        421        474        930         18        975        296         42   PCI-MSI-edge      em1-1
 140:         68        309         16        453        169         19        542         49         53        300       2049       2144       2137        183         16        439       1545          1         26        622       2120        770        978        818   PCI-MSI-edge      em1-2
 141:        558         27        141       1002        100         13       2563        213         39          7        585        110         23        430        326       2585         26         13          2       3985         23         61         69          3   PCI-MSI-edge      em1-3
 142:        838       1247        238         21         11       1019       5156         11       5038       4336        974          1         26       1029       5730        362       2484       4985       9379         75          4        187       1290       1238   PCI-MSI-edge      em1-4
 143:         84        922          0         37         13         41        866        348         12          0        326         11       1439       1658         10          0         21       5464          1         19        249         57          4          2   PCI-MSI-edge      em1-5
 144:       1472        245       1283        202       1940          3       2007       1304       6842        519        170         56         28        624        162          7       9415        773        110          0         82        152          0       6953   PCI-MSI-edge      em1-6
 145:       9016        357       3403         32        827          0        514        262       4488         15         43        125        203       3955       1019       4665        671        776         91      37424         45         72       1718        610   PCI-MSI-edge      em1-7
 147:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-0
 148:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-1
 149:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-2
 150:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-3
 151:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-4
 152:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-5
 153:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-6
 154:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-7
 156:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-0
 157:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-1
 158:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-2
 159:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-3
 160:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-4
 161:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-5
 162:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-6
 163:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-7
 165:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-0
 166:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-1
 167:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-2
 168:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-3
 169:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-4
 170:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-5
 171:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-6
 172:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-7
 NMI:       4376       4040       3655       3841       2557       8255       2087       3536       2371       2609       3557       2920       3380       2787       3231       3980       2952       4209       3142       1868       4092       3002       4809       3247   Non-maskable interrupts
 LOC:    2611446    3261318    3457130    2568007    1587363    5209677    1064619    2538998    1170738    1997930    1911444    2046276    1591660    1286247    1570421    2287817    1240071    2055553    1582708     753177    1875464    1656254    2392998    1508876   Local timer interrupts
 SPU:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Spurious interrupts
 PMI:       4375       4039       3654       3840       2556       8254       2086       3535       2370       2608       3556       2919       3379       2786       3230       3979       2951       4208       3141       1867       4091       3001       4808       3246   Performance monitoring interrupts
 IWI:      75183      54690      35841      72963      49453      89827      53684      74996      48372      75737      35350      63562      75806      56238      57130      67916      41395      58169      38177      80580      47092      23815      48678      17458   IRQ work interrupts
 RTR:         22          0          0          0          0          0          0          0          0          0         23          0          0          0          0          0          0          0          0          0          0          0          0          0   APIC ICR read retries
 RES:     443155     648114     636075     197659     341754     187516     199472     257026     194132     186184     172690     151383      54121      74089     104592      97755      82125     100725      75677      73614      56770      65831      58378      78295   Rescheduling interrupts
 CAL:       6616       7453       9170       6723       6859       7001       6870       7080       6937       7263       6707       7076       5612       5517       4685       4218       5469       5646       5788       5662       5492       5649       5607       5669   Function call interrupts
 TLB:        272        229        194        136        168        953        178        198        196        161        144        237         96        162       1216       1251        171        168         48        125         79         49        112         48   TLB shootdowns
 TRM:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Machine check exceptions
 MCP:        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341   Machine check polls
 ERR:          0
 MIS:          0
            CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       CPU8       CPU9       CPU10      CPU11      CPU12      CPU13      CPU14      CPU15      CPU16      CPU17      CPU18      CPU19      CPU20      CPU21      CPU22      CPU23
   0:        136          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      timer
   1:          3          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      i8042
   8:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      rtc0
   9:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
  12:          4          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      i8042
  17:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
  18:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb4
  19:         88          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1
  20:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   uhci_hcd:usb6
  21:        644    1437284          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb2, uhci_hcd:usb5
  23:       1366       4624         15       3602      11754         26        803       4812       5470       1281       5126      14552        611       7216       8607      17387        130         15        258         38       1331       7077        523       2816   IO-APIC-fasteoi   ata_piix
  72:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  73:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  74:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  75:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  76:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  77:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  78:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      PCIe PME
  79:       3971        108         38        159        176          0         17        415        625       7030         32       2157         44       3933        361       2888       3927          9         23         16        650       1688      12797         11   PCI-MSI-edge      megasas
  80:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  81:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  82:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  83:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  84:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  85:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  86:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  87:          2          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      ioat-msix
  88:    4079713    2369620     972808       8533    1619266       1528          0     825198     780466     267611     997395     200884     350727     556367     393738    1661310      10735    1515552        430      68193    2652469    1241106          0     156223   PCI-MSI-edge      p3p1-TxRx-0
  89:    2862428     887534         98      82611     162788          0    4241872      43803          0     285397    1008074          0     441857     333915          0    3682585     359025    3869015     688406       5201    1239821       2894      30017      12076   PCI-MSI-edge      p3p1-TxRx-1
  90:     173714      11720          4     766216     824939      61932          0    1409624     724109      85038     169611    3391795     343952      65938    1818875     153824    3861124          0     164584     540388    1248660      47745     981953    4124939   PCI-MSI-edge      p3p1-TxRx-2
  91:     631998          0    6634278      31702      98621          0    1268714     169989     262694    2310410       1902          0    3743872     261593    1207237       8815      66919    3136802          0          0     355053     383039          3      10657   PCI-MSI-edge      p3p1-TxRx-3
  92:         18       3437          0      21588          0   15741033     270980        518    3787301          0          0      31710      48995          0          0          0     343041          0     119278          0      73316          0    1082534      59325   PCI-MSI-edge      p3p1-TxRx-4
  93:     407483     296975       8837     161498    2007095      68643      93257    4991210      31776    3407023     514238          0      38624      41491     143267     300316     139742     671753     557794          0    3269556    3058618     778140          0   PCI-MSI-edge      p3p1-TxRx-5
  94:         30       1067     626845     168194     465681          0     103700    5732392      16145          0     817491    2097419     480667      45987    1044301     334455      38474          2     748913      17962       1603     135493    3113154    4797024   PCI-MSI-edge      p3p1-TxRx-6
  95:         16     793894      54571      76095     652491          0    3115604       2952     299085     387717    1242213     187117     271140     365910      12094      29573    5647265    3805028    3719141          1          0          0     520144          0   PCI-MSI-edge      p3p1-TxRx-7
  96:         16     683805    4170012    1273875     170995      54150     606119     112682     216735    3139524     432963     158475     238128    1204811     373233    1246219    3340262     352651          0     287580    2235268         77     102084     906031   PCI-MSI-edge      p3p1-TxRx-8
  97:    1273831     247946     196205    5044946      92091      25860    1046837     426770    2669053     127168      57731     315620       1803    1549339      81913          3    1459302     512299      72959     697763    2050722        897    2974441       1914   PCI-MSI-edge      p3p1-TxRx-9
  98:    5409099    1594864     949517    2974366       9510          0      99766     975507     247782     609629       8278    2766487     124107     340929    1015964    1943754     125930     787820     299572     692711          0     161476      32534          4   PCI-MSI-edge      p3p1-TxRx-10
  99:    2604110     506189          0    1293866    1244062    1560882        582      50999     326441    3698682    1418068          0     152949     292443       1123       9372     580259      27940     817220         55     358789     337895    5751235          0   PCI-MSI-edge      p3p1-TxRx-11
 100:       1545     840836         44     497600    2038326     104106       4883          3    1142048    1715525    2325380     557514     545471    6746430          0     781382    1809216     492419          0     202547          0      83522     137443    1548361   PCI-MSI-edge      p3p1-TxRx-12
 101:      98651      58524          0          0      97933          0          0    4167315    1837005     488147          0          0    7191893          0     179338          0          0       7472      51420     187020    5230646      33394    1206900       6909   PCI-MSI-edge      p3p1-TxRx-13
 102:      73365    1373813       8757    1210642     544610          0      40863    1109033     671544         15    1012707        546          0    2601799    1763584    2128546     727940     352127     220592    5677507         28       2791     397273      24726   PCI-MSI-edge      p3p1-TxRx-14
 103:    2831309    3600024          0    1426626       3293          0      17060       3768      29944      63885     382915     147200      98564     531209    7386403      51283          0          0    2476658     259741    2179451      35046        397      40261   PCI-MSI-edge      p3p1-TxRx-15
 104:        593       1181       5015      13013      32889         86       2541         47      41557        589       9742     523322       1325     151123      30241      84997      45085     175725       7871      15567       1465      25363       2082       2716   PCI-MSI-edge      p3p1-TxRx-16
 105:       8877      15249       3440       1714      85113         62      12924     553377        142        172        490        792       8351        740       3291      61912      37231         15      89927      31676      10250         79       3903      17451   PCI-MSI-edge      p3p1-TxRx-17
 106:        420      70663       4725        862        314          4     648434         22      19691      11464       1239        141       3025       4561        462       3816      28201       9579      27584      21078         57      18921       4354         43   PCI-MSI-edge      p3p1-TxRx-18
 107:       2395       3644       1158      11593       8114        159      11420        806        590         15       4943      37234       2809       8639       4546     171054       3244       1468      69682      46734        170      18075        300         37   PCI-MSI-edge      p3p1-TxRx-19
 108:       1042        354       1161       1119     560317          5       3056       2597        894       9000        116       6087       3949       1480       3712     229348       9253      10206       1116        363       8351      11701      17878       1018   PCI-MSI-edge      p3p1-TxRx-20
 109:      15556       8247        136       5676      87414        105       2664       1108         79        513        203         25        584     662487        106       1186       1841         22        325      24662      14088      73768       2816        125   PCI-MSI-edge      p3p1-TxRx-21
 110:       4595       6197         61       1206        516        541         52       8472       5400     178742         11          5      13388       2135        141        540       1367          2       8206     683657        500       1157      17600      18863   PCI-MSI-edge      p3p1-TxRx-22
 111:        773       2561     170734         77       2168          3          8     606427       9419        641      10210         18       9478       2556       2988      40768      33414        138         98       6791       5978     152548      79602      22951   PCI-MSI-edge      p3p1-TxRx-23
 112:          3         86          0         10          0        539          0          0         26          0          0          2          0       1416          0          0         24          0         20          5          1         69          0        844   PCI-MSI-edge      p3p1
 113:     415859    2232488     515719         84    2731741          3    1728233      10998    1113110     541338     215517      41221     218303    2077155      21919     677409    4125968    2529957     918350       2884    1190740        191    1533199          0   PCI-MSI-edge      p3p2-TxRx-0
 114:       5686    2300559     772892     496756         64        651       2009       4630    1457451        700      15323      45114        348      94333     270118      15027     738803     425711     912855    4097832        545      69435     445021      62844   PCI-MSI-edge      p3p2-TxRx-1
 115:     202302      92947       8467     183589      18088     137575      32231     159393    1370498        130      60328       1690      56838    1389191    2778633         17      66600         21         80      83096         74       3784    5172488     618772   PCI-MSI-edge      p3p2-TxRx-2
 116:       1540         70        157      17046        658    3731356         79     253553    1022948       8534     123112     114527      87817     962485        283       4013     900581    1313971    6198626     147599      69301     255267       2671    1073114   PCI-MSI-edge      p3p2-TxRx-3
 117:     283688     500928     128141        150      78847      55930     527402     271370     402877      31908     329336         68    2236501     245753      89539     336870    1158890     142627     249922     118728       6054      43658    1049020      70543   PCI-MSI-edge      p3p2-TxRx-4
 118:     559904         25     567130    2645969      24324      51082    2730977     324948         10      55930         22    3864180     142719     247505        739     129387         39      55178         41       6483       1599     237301      74731    9373705   PCI-MSI-edge      p3p2-TxRx-5
 119:       2569    2297606          1    1055094      94357      46231     862406       9966      49824       1966    2135162     657738      32389     127798    1676836      94602      46962     423702       1209     119040     702913     569886      29149       2500   PCI-MSI-edge      p3p2-TxRx-6
 120:     138178        227         17      50252     228661    3125851    1724227        127         29      60384         18    3214481     123009     248494      12042    9298486     178864     244622       7498       3112    2306221        929        790    2675775   PCI-MSI-edge      p3p2-TxRx-7
 121:         16     875648      80169      77280     253218      37380     108190          0    1108911     129592      98628      14229    1366251      76510    5456458     734689      52136     433873     587059      26556         40       3007      68803       2087   PCI-MSI-edge      p3p2-TxRx-8
 122:       3900       1441       1607     285207         83          5    2992792        998    2513921      40002    2106369     109676     448433     317357    3109898      97824    1150731     495784     298935    4479053      88377        559      71809       1745   PCI-MSI-edge      p3p2-TxRx-9
 123:       1004     455481      39217      18522    2473925     109658      57734      17566     751387       3051      97569         10     369177     648341       3673       6782      15026    1486725     519104    1633260        143    1063446       8468     433014   PCI-MSI-edge      p3p2-TxRx-10
 124:      44746      55470       1749      59125    8454810       1351        496       1649     534728     146571      40337     206490       2922     115443     348903       8078     823079    2923241       8024     427531       1380      56214      89367       1545   PCI-MSI-edge      p3p2-TxRx-11
 125:      39752      10678    5124248        609     680170       1614     930548        546     163193        170    1309561       2516     482399     447457    1301699      61151    2754552     316285      53411    1456578    2849035       3097     124245     919521   PCI-MSI-edge      p3p2-TxRx-12
 126:       7712      59866      21804     600152      79315       4522     233266       3867      60724      16212    7961173          0     865127     510105    1012814       6807     374536     461796      42572    3642818         23       2508    2168135         50   PCI-MSI-edge      p3p2-TxRx-13
 127:     596919    4488055         55       6501     585701       4200      48381     231206          1      22552         93     211775     284928    1806544       9857    1623938      33348      11349         16    1093386      36705      39910     833479    8676416   PCI-MSI-edge      p3p2-TxRx-14
 128:     431933     337326      76685      35683     222605      48636        125      74371       1416    5478214       2529      13810     674641     259856    2267098     685640    1837443     835537        306       2202     333227     365143      42217      17010   PCI-MSI-edge      p3p2-TxRx-15
 129:       1610       2156          5    1391667     132424          0    1383413      46033    6727339          5    4524512    1432960     124806     324434     112676       5717     462281     846876     991861    1548290     112779      63229     723810     211044   PCI-MSI-edge      p3p2-TxRx-16
 130:       8352      32121        910    2270695     985532         18    3052160        163    1274802       1712        997         84     497187    1456692     167631    1192899     158600     700669         11      22304      84705     333402     116696   10072529   PCI-MSI-edge      p3p2-TxRx-17
 131:      38135       1876    1157268        233     406138        168     465558       1055    1609893     377548     488647     123467     167237     340859      40193        297    4683085        407     139600    1769045      17571        633     689473       3701   PCI-MSI-edge      p3p2-TxRx-18
 132:      56970        855       1160      19229          0       2072    8646387     124643       4201     888190         10     142631       7140     151126    1903330      49042        731     266954       2427     272879        105    1341744        434       5242   PCI-MSI-edge      p3p2-TxRx-19
 133:       3468        688     130805         31    4131618       3875     808374        363    3314206     594716     863383    5840754     967008    1443958        822    6776261     523197      10995      65592      18168     256989     732888      84350          3   PCI-MSI-edge      p3p2-TxRx-20
 134:       8770    3997791      36196     846003       6999          3       1390        319       2054       2866        154    1227855     328709         69     695176          9       6443     825777      55558     178017        123       2984      53255     485799   PCI-MSI-edge      p3p2-TxRx-21
 135:    3673195    2027021   10324157        210     190188      78456     243163        131          9      37442     252167    1514961     165762      91009       1182    1509551      89152      98454          9     121840     114549    2111683     134585     262448   PCI-MSI-edge      p3p2-TxRx-22
 136:       8702     572706          5     143834       4038       1270        853     745905     279973    1438980    7719553     146580      59017       2285        486      71089    1747221       1534        169        669       2247     271629       3682     150631   PCI-MSI-edge      p3p2-TxRx-23
 137:        157         68          9          9         22          9         97       2341        556          0        694         33        208        288         29       2231       2355        255         33        545        489       8425       2376        443   PCI-MSI-edge      p3p2
 138:       3069        438      55487       1217        661          0    1100986        521       3170      71796     709786      31340      34236     521811       1058      20927      31891       4024      13705     226934     459490      93533      39241       9197   PCI-MSI-edge      em1-0
 139:         10          4        222        630         56          0          1         71        626         26       2000         61          4       8509         70          4         35        421        474        930         18        975        296         42   PCI-MSI-edge      em1-1
 140:         68        309         16        466        169         19        542         49         53        300       2049       2144       2137        183         16        439       1545          1         26        622       2120        770        978        818   PCI-MSI-edge      em1-2
 141:        558         27        141       1002        100         13       2563        213         39          7        585        110         35        430        327       2585         26         13          2       3985         23         61         69          3   PCI-MSI-edge      em1-3
 142:        838       1247        238         21         11       1019       5156         11       5038       4336        974          1         26       1029       5730        362       2490       4985       9379         75          4        187       1290       1238   PCI-MSI-edge      em1-4
 143:         84        922          0         37         13         41        866        348         12          0        326         11       1439       1658         10          0         21       5464          1         19        251         57          4          2   PCI-MSI-edge      em1-5
 144:       1472        245       1283        214       1940          3       2007       1304       6842        519        170         56         28        624        162          7       9415        773        110          0         82        152          0       6953   PCI-MSI-edge      em1-6
 145:       9016        357       3404         32        827          0        514        262       4488         15         43        125        203       3955       1019       4665        671        776         91      37424         45         72       1718        610   PCI-MSI-edge      em1-7
 147:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-0
 148:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-1
 149:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-2
 150:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-3
 151:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-4
 152:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-5
 153:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-6
 154:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em2-7
 156:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-0
 157:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-1
 158:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-2
 159:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-3
 160:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-4
 161:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-5
 162:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-6
 163:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em3-7
 165:          1          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-0
 166:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-1
 167:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-2
 168:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-3
 169:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-4
 170:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-5
 171:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-6
 172:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      em4-7
 NMI:       4391       4055       3655       3841       2558       8270       2087       3536       2386       2624       3572       2935       3380       2788       3231       3996       2967       4224       3157       1883       4107       3017       4824       3263   Non-maskable interrupts
 LOC:    2620977    3270902    3458130    2568310    1587678    5219385    1064775    2539378    1180741    2007634    1921280    2055943    1591816    1286500    1572817    2297017    1249807    2065387    1592289     762573    1885184    1666036    2402693    1518409   Local timer interrupts
 SPU:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Spurious interrupts
 PMI:       4390       4054       3654       3840       2557       8269       2086       3535       2385       2623       3571       2934       3379       2787       3230       3995       2966       4223       3156       1882       4106       3016       4823       3262   Performance monitoring interrupts
 IWI:      75183      54690      35841      72963      49453      89827      53684      74996      48372      75737      35350      63562      75806      56238      57130      67916      41395      58169      38177      80580      47092      23815      48678      17458   IRQ work interrupts
 RTR:         22          0          0          0          0          0          0          0          0          0         23          0          0          0          0          0          0          0          0          0          0          0          0          0   APIC ICR read retries
 RES:     443333     648233     641720     197937     341769     187530     199487     257276     194132     186203     172691     151402      54137      74355     104701      97769      82133     100740      75687      73620      56775      65847      58383      78318   Rescheduling interrupts
 CAL:       6616       7453       9170       6723       6859       7001       6870       7080       6937       7263       6707       7076       5612       5517       4685       4218       5469       5646       5788       5662       5492       5649       5607       5669   Function call interrupts
 TLB:        272        229        195        136        168        953        178        198        196        161        144        237         96        162       1216       1251        171        168         48        125         79         49        112         48   TLB shootdowns
 TRM:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Thermal event interrupts
 THR:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Threshold APIC interrupts
 MCE:          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0          0   Machine check exceptions
 MCP:        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341        341   Machine check polls
 ERR:          0
 MIS:          0

Open in new window






Other results :


[root@244 ~]# ethtool -k p3p1
Features for p3p1:
rx-checksumming: off
tx-checksumming: off
        tx-checksum-ipv4: off
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off
        tx-checksum-fcoe-crc: on [fixed]
        tx-checksum-sctp: off
scatter-gather: off
        tx-scatter-gather: off
        tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
        tx-tcp-segmentation: off
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp6-segmentation: off
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: on
tx-vlan-offload: on
ntuple-filters: off
receive-hashing: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: on [fixed]
tx-gre-segmentation: off [fixed]
tx-ipip-segmentation: off [fixed]
tx-sit-segmentation: off [fixed]
tx-udp_tnl-segmentation: on
tx-mpls-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
busy-poll: on [fixed]



[root@244 ~]# lsmod
Module                  Size  Used by
cpufreq_stats          13344  0
nfnetlink_queue        18197  0
nfnetlink_log          17926  0
nfnetlink              14606  2 nfnetlink_log,nfnetlink_queue
bluetooth             372662  0
rfkill                 26536  1 bluetooth
iptable_filter         12810  1
iptable_nat            12875  0
nf_nat_ipv4            14115  1 iptable_nat
ip_tables              27239  2 iptable_filter,iptable_nat
xt_nat                 12681  0
xt_REDIRECT            12757  0
ts_kmp                 12605  22
xt_string              12586  22
ipt_SYNPROXY           12852  0
nf_synproxy_core       14517  1 ipt_SYNPROXY
ipt_MASQUERADE         12678  0
nf_nat_masquerade_ipv4    13412  1 ipt_MASQUERADE
nf_nat                 26146  4 nf_nat_ipv4,xt_nat,nf_nat_masquerade_ipv4,xt_REDIRECT
xt_limit               12711  0
nf_log_ipv4            12767  0
nf_log_common          13317  1 nf_log_ipv4
xt_LOG                 12690  0
xt_recent              18542  2
nf_conntrack_ipv4      14862  4
nf_defrag_ipv4         12729  1 nf_conntrack_ipv4
ip6t_REJECT            12939  2
nf_conntrack_ipv6      18738  2
nf_defrag_ipv6         34651  1 nf_conntrack_ipv6
xt_physdev             12587  0
xt_CT                  12956  0
xt_conntrack           12760  5
nf_conntrack          105702  9 xt_CT,ipt_SYNPROXY,nf_synproxy_core,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_conntrack_ipv6
ip6table_filter        12815  1
ip6_tables             27025  1 ip6table_filter
ixgbe                 301608  0
intel_powerclamp       18764  0
coretemp               13435  0
kvm_intel             148081  0
kvm                   461126  1 kvm_intel
crct10dif_pclmul       14289  0
crc32_pclmul           13113  0
crc32c_intel           22079  0
ghash_clmulni_intel    13259  0
aesni_intel            69884  0
lrw                    13286  1 aesni_intel
gf128mul               14951  1 lrw
glue_helper            13990  1 aesni_intel
ablk_helper            13597  1 aesni_intel
iTCO_wdt               13480  0
cryptd                 20359  3 ghash_clmulni_intel,aesni_intel,ablk_helper
wmi                    19070  0
iTCO_vendor_support    13718  1 iTCO_wdt
vxlan                  37409  1 ixgbe
dcdbas                 14847  0
ip_tunnel              23760  1 vxlan
ipmi_devintf           17572  0
acpi_power_meter       18087  0
ipmi_si                53353  0
ipmi_msghandler        45603  2 ipmi_devintf,ipmi_si
serio_raw              13462  0
pcspkr                 12718  0
i7core_edac            24166  0
ioatdma                67762  0
lpc_ich                21073  0
edac_core              57650  2 i7core_edac
shpchp                 37032  0
mfd_core               13435  1 lpc_ich
xfs                   915019  3
libcrc32c              12644  1 xfs
sr_mod                 22416  0
cdrom                  42556  1 sr_mod
ata_generic            12910  0
pata_acpi              13038  0
usb_storage            66523  0
sd_mod                 45499  3
crc_t10dif             12714  1 sd_mod
crct10dif_common       12595  2 crct10dif_pclmul,crc_t10dif
mgag200                42159  1
syscopyarea            12529  1 mgag200
sysfillrect            12701  1 mgag200
sysimgblt              12640  1 mgag200
i2c_algo_bit           13413  1 mgag200
drm_kms_helper         98226  1 mgag200
ttm                    93488  1 mgag200
drm                   311588  4 ttm,drm_kms_helper,mgag200
ata_piix               35038  0
libata                218854  3 pata_acpi,ata_generic,ata_piix
mdio                   13807  0
i2c_core               40325  4 drm,drm_kms_helper,mgag200,i2c_algo_bit
ptp                    18933  1 ixgbe
pps_core               19106  1 ptp
megaraid_sas          117865  2
dca                    15130  2 ixgbe,ioatdma
bnx2                   89257  0
dm_mirror              22135  0
dm_region_hash         20862  1 dm_mirror
dm_log                 18411  2 dm_region_hash,dm_mirror
dm_mod                104038  11 dm_log,dm_mirror

Open in new window




uname -r is 3.10.0-229.11.1.el7.x86_64
Are You Protected from Q3's Internet Threats?

Every quarter, WatchGuard's Threat Lab releases a security report that analyzes the top threat trends impacting companies around the world. For Q3, we saw that 6.8% of the top 100K websites use insecure SSL protocols. Read the full report to start protecting your business today!

gheistCommented:
You did not provide usable tcpdump (tcpdump -c 1000 -w pcap.pcap -w 10000 -i <interface>) nor model of your network card (ethtool -i <interface>) or system boot messages (dmesg)
No wonder your provider ignores you.
FireBallITAuthor Commented:
[root@244 ~]# ethtool -i p3p1
driver: ixgbe
version: 4.1.2
firmware-version: 0x800006c2, 15.0.27
bus-info: 0000:07:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Open in new window

Screenshot-2.png
FireBallITAuthor Commented:
FireBallITAuthor Commented:
FireBallITAuthor Commented:
dear gheist please if you will write ask to your isp do not respond the questions this time other people do not read them . We need to solve it on our side not on isp side
gheistCommented:
700000 packets in 38s = 20000packets/s or 300Mbps on ethernet wire - not a problem I assume?
Subtracting IRQs we get 600 network hardware interrupts per second - network driver is not a problem?

Dissecting traffic with wireshark shows big part of traffic 90% china 9% turkey rest from the rest - any problem?

IRQs per CPU per 10s:
0      587      708      31      25      34      278      39      0      10      327      158      437      26      0      17      812      528      141      594      568      315      287


There is something wrong with processing on the machine itself.
gheistCommented:
Why dont you enable tx checksum processing on network card? That is something basic that works for 15 years or so:

ethtool -k eth0 tx on rx on
Better?

I see "migration" process between those high CPU users
Limit it to 1 migration of process between CPUs:
sysctl kernel.sched_nr_migrate=2
Better?

24 CPUs - is it hyperthreading? Cut that off.
Better?

Disable iptables dropping packets (essentially completely disable UDP state based dropping). There is lone torrent client in your network using UDP, rest are DNS.
Better?

Where is that dmesg? Doesnt it say something bad about netfilters?
PaulOffordCommented:
One thing you could do to give you temporary relief is to harmlessly dump the incoming ACK flood.  I think you could do this as follows:

1. Identify the Ethernet device that is forwarding the packets to your Centos Server
2. Add a static ARP entry to that device so as to map 178.20.229.146 to, say, 00:00:00:00:00:01

To identify the Ethernet device forwarding the packets:

1.  Run tcpdump again but with the switches -XXnvv
2.  Look up the MAC Address in the MAC tables on your switches to track down which port the device is connected too
3.  Trace the cable from that port back to the device

In the tcpdump output the source MAC address should be the second block of 8 bytes (the first block is the destination MAC address) and will be immediately followed by 0x0800

One thing to bear in mind is that if you do have a device 178.20.229.146 and the rogue ACKs are coming from, say, a router, the device with the address 178.20.229.146 won't be able to send traffic via that router.

Best regards...Paul
PaulOffordCommented:
Ah - just noticed the trace.  So it's not a flood of ACKs, it's a flood of IP Fragments.

The packets are coming from 78:19:f7:99:95:d9 which appears to be a Juniper device.  Remember, it might be a routing interface (layer 3 switching) inside a switch.  In which case you might want to track back further to where the packet is entering your network.  If it is entering the network on a subnetwork other than you the 178.20.229.144/29 network, rather than specify the ARP entry as I described above, add a static route for 178.20.229.146/32 to forward to (next hop) a non-existent IP address.

Again the same caveat regarding any host that has the address 178.20.229.146 - this approach will stop that host being able to communicate across the network with that address.
gheistCommented:
Packet rate is not so tragic.
Problem is dispatch within system with 100% busy process migration between CPUs and rest busy with cleanig up after network IRQ processing.
I think it is about interrupt affinities, numas, irqbalances, hyperthreads and BIOS and the likes
FireBallITAuthor Commented:
Dear paul it is the mac address of our EX4500 traffic comes from our internal network for testing.
But we are getting the same attack from out of the network to 80 port of the server. So web sites getting down. And we do not want to null route or discard the ip address


Dear gheist  we are getting the same attack 2M + pps as far as described on the datasheets X520 intel dual 10G ethernet card should handle 15M+ small pps so i do not understand why it drop packets on ethernet card and blocking other traffic
gheistCommented:
You get 20k packets per second. It is essentially nothing for your network card.
Why you do not have tcp checksum processing enabled on card?
FireBallITAuthor Commented:
actually i have disable it for testing because before disable'in it it was the same nothig changed
gheistCommented:
Get network card to default parameters
You have 0.15% load on your netcard and system chokes. What do you think - is it problem with system software or with the network card?

Still no output like dmesg...

Choose next action - park cores on running system or bind all IRQs from netcard to single CPU ?
FireBallITAuthor Commented:
dmesg attached
log2.txt
gheistCommented:
As I suspected:
nf_conntrack: table full, dropping packet
:)

rise (double, wait a minute, repeat)
net.netfilter.nf_conntrack_max = 65536
until those messages stop in kernel messages
Each conntrack entry is 100 bytes or so 64k of them is 6,5MB
FireBallITAuthor Commented:
what should i do for a optimized resource actually our aim for 2M pps limit
FireBallITAuthor Commented:
I have tryed but still the same probably it is loggng less packet because sflow monitor showing for this test attack 300K pps - 500K pps


echo 256960 > /proc/sys/net/core/rmem_default
echo 256960 > /proc/sys/net/core/rmem_max
echo 256960 > /proc/sys/net/core/wmem_default
echo 256960 > /proc/sys/net/core/wmem_max




echo 0 > /proc/sys/net/ipv4/tcp_timestamps 
echo 1 > /proc/sys/net/ipv4/tcp_sack 
echo 1 > /proc/sys/net/ipv4/tcp_window_scaling


net.netfilter.nf_conntrack_max = 19660800
net.netfilter.nf_conntrack_generic_timeout = 600 
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120 
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60 
net.netfilter.nf_conntrack_tcp_timeout_established = 432000 
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120 
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60 
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30 
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120 
net.netfilter.nf_conntrack_tcp_timeout_close = 10 
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300 
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300 
net.netfilter.nf_conntrack_udp_timeout = 30 
net.netfilter.nf_conntrack_udp_timeout_stream = 180 
net.netfilter.nf_conntrack_icmp_timeout = 30 
net.netfilter.nf_conntrack_events_retry_timeout = 15

Open in new window

gheistCommented:
Data you provided is 20000 packets per second.
That can fill 64k conntracks in 3 seconds, with 5min timeout you will need 350k conntracks or more.

There is /etc/sysctl.conf to preserve sysctl values across reboots.

It is hard to help when you mix and match input from various systems.
in particular dmesg you get ~40 conntrack full mesaages per second. Does it happen with 20k packets/s system or some other system?
FireBallITAuthor Commented:
is there any way to apply the rule without reboot ?

We have SRX 3600 it has the same problem also :) after 800K pps of this packets it is loosing connection.
gheistCommented:
Pardon me - you provided data about 20k packets per second. Out of curiousity I checked my webserver and it has 3x that.

I have no clue what is same problem? If it is about full conntrack rule of the tumb is to double it.

Sysctl documentation (man sysctl ; man sysctl.conf)
echo 123 > /proc/sys/a/b/c
equals
sysctl a.b.c=123
equals
echo a.b.c = 123 >> /etc/sysctl.conf ; sysctl -p

dont set conntrack size on reboot.
it will be 64k
check dmesg - count lines per second generated.
double
reduces? cool, doubble again until conntrack full disappears.
FireBallITAuthor Commented:
i have tryed that but not solved still the same
gheistCommented:
you need to rise conntrack size until it copes with traffic (and keeps memory usage in control) RHEL/CentOS you are using is tuned for things like oracle database or jboss java server, not network router.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
FireBallITAuthor Commented:
thank you
gheistCommented:
FreeBSD or OpenBSD may fit better that place....
FireBallITAuthor Commented:
X520 ethernets has some problems with unsupported sfp s and we can not over come the issue with freebsd :)
gheistCommented:
I think it does not reload recent network card microcodes and takes any SFPs after network card update (like system BIOS)
Your mileage may vary.
FireBallITAuthor Commented:
that is a common issue on the X520 cards you should see on google
x520 intel allow unsupported sfp
search :) ok i will let try my chance on working with contrackts
gheistCommented:
Been there ... with HP official supported modules.... And HP sent replacement next day...
FireBallITAuthor Commented:
we are using dell and i think contract does not for next day but i will try my chance with it also

ths question should be take your interest too
http://www.experts-exchange.com/questions/28708488/TCP-flags.html
gheistCommented:
if you buy competing SFP you may feel 1c guilty it is not certified, but when you pay premium for same SFPs put in handy server box and they come unsupported - you feel sorry for that $$$ you paid in premium.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Linux Networking

From novice to tech pro — start learning today.