SPC Distrubitation problem

We realize that juniper drop packets from SPC which is working on FPC7 and Combo mode
 
as you should see below SRX some times getting the FPC12 or sometimes FPC 11 empty. But fpc7 getting a load more then it's capacity because it is working on combo mode ;
 
We have checked too many things and realize that also as you should see below only SPC on FPC7 has hardware drops.
 
how should we resolve that ?


root> show security monitoring performance spu
fpc  7  pic  0
Last 60 seconds:
 0:  25   1:  26   2:  27   3:  26   4:  26   5:  27
 6:  27   7:  27   8:  27   9:  27  10:  27  11:  27
12:  67  13:  67  14:  67  15:  67  16:  33  17:  25
18:  24  19:  27  20:  22  21:  21  22:  27  23:  27
24:  26  25:  16  26:  19  27:  40  28:  68  29:  63
30:  66  31:  68  32:  67  33:  65  34:  68  35:  65
36:  66  37:  67  38:  63  39:  27  40:  27  41:  27
42:  27  43:  27  44:  27  45:  27  46:  27  47:  27
48:  27  49:  27  50:  27  51:  40  52:  42  53:  42
54:  42  55:  41  56:  41  57:  42  58:  36  59:  27
fpc  11 pic  0
Last 60 seconds:
 0:   0   1:   0   2:   0   3:   0   4:  14   5:  49
 6:  51   7:  51   8:  50   9:  50  10:  51  11:  51
12:  34  13:   0  14:   0  15:   0  16:   0  17:   0
18:   0  19:   0  20:  14  21:  40  22:  44  23:  50
24:  50  25:  38  26:  37  27:  41  28:  30  29:   0
30:   0  31:   0  32:   0  33:   0  34:   0  35:   0
36:   0  37:   0  38:   0  39:   0  40:   0  41:   0
42:   0  43:   0  44:   0  45:   2  46:  27  47:  27
48:  29  49:  29  50:  30  51:  29  52:  20  53:   0
54:   0  55:   0  56:   0  57:   0  58:   0  59:   0
fpc  12 pic  0
Last 60 seconds:
 0:  47   1:  48   2:  50   3:  48   4:   3   5:   0
 6:   0   7:   0   8:   0   9:   0  10:   0  11:   0
12:   0  13:   0  14:   0  15:   0  16:  38  17:  49
18:  45  19:  50  20:   3  21:   0  22:   0  23:   0
24:   0  25:   0  26:   0  27:   0  28:   0  29:   0
30:   0  31:   0  32:   0  33:   0  34:   0  35:   0
36:   0  37:   0  38:   0  39:   0  40:  47  41:  50
42:  50  43:  50  44:  50  45:  32  46:  24  47:  23
48:  22  49:  22  50:  22  51:  22  52:  23  53:  22
54:  21  55:  21  56:  22  57:  21  58:  18  59:  20
root> show chassis hardware
Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                AB4209AA0014      SRX 3600
Midplane         REV 07   710-020310   AAAV0320          SRX 3600 Midplane
PEM 0            rev 08   740-027644   G087FD002R08P     AC Power Supply
PEM 1            rev 08   740-027644   G087FE004B08P     AC Power Supply
CB 0             REV 14   750-021914   AAAV0881          SRX3k RE-12-10
  Routing Engine          BUILTIN      BUILTIN           Routing Engine
  CPP                     BUILTIN      BUILTIN           Central PFE Processor
  Mezz           REV 08   710-021035   AAAN7843          SRX HD Mezzanine Card
FPC 0            REV 16   750-021882   AADE3908          SRX3k SFB 12GE
  PIC 0                   BUILTIN      BUILTIN           8x 1GE-TX 4x 1GE-SFP
FPC 1            REV 20   750-020321   AAFE5669          SRX3k 2x10GE XFP
  PIC 0                   BUILTIN      BUILTIN           2x 10GE-XFP
    Xcvr 0                NON-JNPR     T09L21440         XFP-10G-SR
    Xcvr 1                NON-JNPR     T09L21452         XFP-10G-SR
FPC 4            REV 14   750-020321   AAAV0984          SRX3k 2x10GE XFP
  PIC 0                   BUILTIN      BUILTIN           2x 10GE-XFP
    Xcvr 0                NON-JNPR     T09L21443         XFP-10G-SR
    Xcvr 1                NON-JNPR     T09L21436         XFP-10G-SR
FPC 7            REV 13   750-016077   AADC9162          SRX3k SPC
  PIC 0                   BUILTIN      BUILTIN           SPU Cp-Flow
FPC 10           REV 19   750-017866   AABZ0103          SRX3k NPC
  PIC 0                   BUILTIN      BUILTIN           NPC PIC
FPC 11           REV 16   750-016077   AAEA6880          SRX3k SPC
  PIC 0                   BUILTIN      BUILTIN           SPU Flow
FPC 12           REV 13   750-016077   AADC9166          SRX3k SPC
  PIC 0                   BUILTIN      BUILTIN           SPU Flow
Fan Tray 0       REV 06   750-021599   AAAM4505          SRX 3600 Fan Tray
 
GOT: === spc7, swanhill7, Ingress, QOS, per queue counters:
GOT: Q# P      PktCnt     ByteCnt   ErrPktCnt     DropCnt Buf RateKbps
GOT: -- - ----------- ----------- ----------- ----------- --- --------
Before :   GOT: 0  0   689433376  3198313481           0   173262666   0        0
After  :   GOT: 0  0   716151200  1322793729           0   187301004   0        0


GOT: === spc11, swanhill11, Ingress, QOS, per queue counters:
GOT: Q# P      PktCnt     ByteCnt   ErrPktCnt     DropCnt Buf RateKbps
GOT: -- - ----------- ----------- ----------- ----------- --- --------
Before :  GOT: 0  0   250772733  1150162904           0     1086140   0        0
After  :  GOT: 0  0   260742314  2506553813           0     1086140   0        0


GOT: === spc12, swanhill12, Ingress, QOS, per queue counters:
GOT: Q# P      PktCnt     ByteCnt   ErrPktCnt     DropCnt Buf RateKbps
GOT: -- - ----------- ----------- ----------- ----------- --- --------
Before :  GOT: 0  0   237042682  3273566470           0     4507285   0        0
After  :  GOT: 0  0   241128473  3830250900           0     4507285   0        0

Open in new window



note red bordered column is dropcnt

router
FireBallITAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

giltjrCommented:
Can you post your current config?  Have you tried:

set security forwarding-process application-services session-distribution-mode hash-based

Did you ever get your support contract issues worked out?
0
FireBallITAuthor Commented:
It works on 5k series we have 3k
0
FireBallITAuthor Commented:
Also i really get bored from talking to the juniper they are really bad company
Their customer services not working as expected
0
Ultimate Tool Kit for Technology Solution Provider

Broken down into practical pointers and step-by-step instructions, the IT Service Excellence Tool Kit delivers expert advice for technology solution providers. Get your free copy now.

giltjrCommented:
Unless I am misreading something according to this it is valid on SRX3600 also:

http://www.juniper.net/techpubs/en_US/junos12.3x48/topics/reference/statement-hierarchy/security-hierarchy-forwarding-process.html
Did the command get rejected, or just not do anything once you implemented it?
0
FireBallITAuthor Commented:
Not worked atrack is creating only one session
0
giltjrCommented:
I will have to read a little more, but I don't think you can distribute one session.  I'm fairly sure that a single session has to go to a single SPC in order for it to know the state of the current session.  The distribution process is for multiple sessions.  I also fairly sure that it does not matter what hardware you have 5K or 3K the session state information is not replicated across SPC's.

But let me make sure I understand this, you have a single session that is flooding a SPC with so much traffic it can't handle it?  I would assume that either this attack is coming from inside your own network, or you have a 10Gbps or higher Internet connection.

If I remember correctly you were generating your own attacks internally to do testing.  If this is one of your test cases, just remember unless you plan on getting attacked internally you should limit your "attack" speed to a maximum of your fastest Internet link's speed.
0
FireBallITAuthor Commented:
Yes attack comes from an ip address with 29 byte packets. And it creates only one session. With huge pps .
It seems it distribute connections to SPCs but not well ass expected. because there is no more traffic on this device that is a test lab. and it shares the usage with other spcs.

Our uplink speed is 40Gbit. I am testing it internally with 1Gbit from out of the srx protected network. It causes to loose all connection behind the srx , I have attached the test script for testing purposes .
We can get 5G+ this type of attacks.
We have an ip transit connection with first colo which server Arbor service , and we get all outbound traffic of Turkey from firstcolo. So we just need to stop max 5-6G this type of attacks.
0
FireBallITAuthor Commented:
you have a single session that is flooding a SPC with so much traffic it can't handle it?

Yes but actually there as i said there is no more traffic on the device , and it is distrubuting one session's usage.
1jova
0
giltjrCommented:
The uplink maybe 40Gbps, but do you really have a 40Gbps Internet connection?  That would imply that you in a data center that is a major Internet hub.

What language/format is the file so I can read it?

The smallest UDP packet is 52 bytes and the smallest TCP packet is 64 bytes.  So when you say 29 bytes, do you mean that the packet has 29 bytes of data and then the headers?  Which means that if you have 29 bytes of payload your packets are 81 (UDP) or 93 (TCP) bytes.

In one performance test I saw a fully loaded SRX 3600 (7 SPC's)  handled 6.8 Mpps of  64 byte TCP packets.

In theory 1 Gbps link can do about 3 Mpps of 64 byte TCP packets, which means you are pushing your  SRX (with 3 SPC's)  to its max, maybe a little beyond.
0
FireBallITAuthor Commented:
It was giving the same result when we had 1 spc
 500k pps makes 400mbps and spc cpus not seem under heavy load
Attack script attached is a builded c script which works like ./1jova

We have active 40 gbps connection with 4 different isp ans active 500+ server .
Our major traffic is about udp streams and that attacks killing us
0
giltjrCommented:
"Our major traffic is about udp streams and that attacks killing us"

That explains why you keep testing with UDP. UDP is a pain. Since it is connectionless it is way more difficult to detect attacks than TCP.

It is especially hard if your traffic requires a LOT of upd packets per second.  Most firewalls, including SRX limit UDP attacks by limiting the number of packets that can come from a single source IP address in a second (or some other time period).

Based on what you must service how many UDP pps would you expect from a single IP address for a valid "session".
0
FireBallITAuthor Commented:
on srx there is a limit source session control but there is no limit for source connection unfortunately.
So we decide to create custom rules with filters for each attack and idp custom attacks but it is serving a performance under the expectation
0
FireBallITAuthor Commented:
normally we have no problem our network has 2 L3 device. One of them getting downloads and anounce as number other device sending outbound traffic as spoof something like amplification attack.
Second device stream out so no boddy can effect outputs but when inbound traffic locked (normally it is about %2 - 5 of outbound traffic) all protected network getting down.

so the inbound traffic's device is not under heavy traffic while in normal conditions
0
giltjrCommented:
Actually the SRX does have a limit for source connection, the problem is UDP has no connection, only TCP does.  This is why UDP traffic is a pain to deal with.

SRX does allow you to prevent UDP sweeps, a single source sending to multiple destinations, and it allows to prevent UDP flooding, a single or multiple sources going to a single destination.  This limits the number of packets to 10 per second where the source is the same, but the destination is different:

     set security screen ids-option untrusted-zone udp udp-sweep threshold 1000000

The "1000000" is micro seconds, so if you wanted to limit to 10 packets in 0.5 seconds, you would specify 500000.

This will limit the number of UDP packets to a single destination to 50,000 packets per second.

     set security screen ids-option untrust-zone  udp flood threshold 50000

You would need to know your environment to set these two correctly.  Allowing too much traffic won't help, allowing to little traffic through would cause valid traffic to get dropped/blocked.
0
FireBallITAuthor Commented:
yes but tht attacks comes to same destination port so udp sweep not solved it
and we need to clear the traffic not block it so there must be an answr why it can not pass 500k+ pps on udp


threshold works perfect but it is not acceptible because it  blocks the real traffic too

at the other side screen works on layer 3  / 4 so how the firewall filter does not serve the same performance ?
0
FireBallITAuthor Commented:
And i miss this :



The smallest UDP packet is 52 bytes and the smallest TCP packet is 64 bytes.  So when you say 29 bytes, do you mean that the packet has 29 bytes of data and then the headers?  Which means that if you have 29 bytes of payload your packets are 81 (UDP) or 93 (TCP) bytes.




http://stackoverflow.com/questions/4218553/what-is-the-size-of-udp-packets-if-i-send-0-payload-data-in-c




This means an IP packet with an empty UDP datagram as payload takes at least 28 (IPv4) or 48 (IPv6) bytes, but may take more bytes.


28Byte packet headers and etc & 1 byte data
0
giltjrCommented:
We are both right, but looking at it different ways.  True a empty UDP packet is 28 bytes.  However you are getting this data over a 1 Gbps Ethernet interface, so I was including the Ethernet frame of 24 bytes, hence 52 bytes.

When you are looking at pps bytes or bits per second you really need to include the overhead of all protocols being used.   Example:  

29 bytes at 500,000 packets per second is  14,500,000 bytes per second and 116,000,000 bits per second.
52 bytes at 500,000 packets per second is  26,000,000 bytes per second and 208,000,000 bits per second.

You can see that on an empty UDP packet you are actually processing almost twice as much data at the Ethernet level because with an empty UDP packet the Ethernet header almost as big as the UDP packet.  Thus on an Ethernet network if you don't take into account the Ethernet overhead and you have a lot of small packets you could easily be miscaclulating the traffic per second by a lot.  

I would escalate the problem with Juniper and get a more senior tech involved.  We have never had an issue with support, but we have ISG's and they are ScreenOS based firewalls and we only have 300 Mbps of Internet bandwidth.
0
FireBallITAuthor Commented:
scrdear glitjr we have bought this product refurbished and they could not assign the product to us, it has been nearly 3 months. I really do not think to talk with them any more i had called them too many times.

Their last answer is , like the device is not their production , i just want them to add our device to customer panel so we should buy an support contract easily .

Customer Notes :
Hi Cahit,

As referenced previously by my colleague the device SN AB4209AA0014 was
purchased outside of the Juniper Authorised Reseller channels.
In order to register this device into your company name and for you to
purchase support you must request a quote for re-instatement fee's
from your authorised Juniper Reseller.

Please reference the below URL to locate a Juniper Reseller in your
area.

https://www.juniper.net/us/en/partners/locator/

Please also reference the below URL which relates to the Juniper
re-instatement and inspection policy.

http://www.juniper.net/support/990222.pdf

Regards,
Jason Roberton

Open in new window



forget about them please.

SRX connected to the router with 10G we have other methods in plan like
creating an address book with datacenter ip blocks. and adding rate limit for egress port of router to srx for UDP something like that.

But we mainly want to know what is the main problem. SRX connected to the router with 10G uplink. only attack machine connected with 1G . So yes pps is a big number for 1G but for 10G connection  ?


as you should see on image
185.9.157.15 is connected to a switch on Xe-1/0/1
192.168.1.15 is ip address of the srx on ge-0/0/1

different interfaces infected from the same attack
0
giltjrCommented:
Can you post your current config?

That is the problem buy anything out side normal sales channel.   You can get a support contact, but first you will need  to through an authorized reseller, and what they are going to do is make you pay the cost of support going back to when the box was last under support.  So if it has been out of support for 3 years, they will make you pay for 3 years of support to get current and then by a new contact.

The stats on the Gbps interface include the Ethernet overhead.  You have  324,387,816 bps and 675,743, that is roughly 480 bits per packet, which is 60 bytes.  By chance is the interface also using tag'ed VLANs?  That add a little more overhead.
0
FireBallITAuthor Commented:
no we do not use vlan


version 12.1X44-D45.2;
system {
    root-authentication {
    }
    name-server {
        195.175.39.39;
        8.8.8.8;
    }
    services {
        ssh;
        telnet;
        web-management {
            http {
                interface [ ge-0/0/1.0 xe-1/0/0.0 xe-1/0/1.0 ];
            }
        }
    }
    license {
        autoupdate {
            url https://ae1.juniper.net/junos/key_retrieval;
        }
    }
}
interfaces {
    ge-0/0/1 {
        gigether-options {
            no-auto-negotiation;
        }
        unit 0 {
            family inet {
                inactive: rpf-check {
                    fail-filter rpf-filter;
                    mode loose;
                }
                filter {
                    input BlokKural;
                }
                address 192.168.1.95/24;
            }
        }
    }
    xe-1/0/0 {
        unit 0 {
            family inet {
                filter {
                    input BlokKural;
                }
                sampling {
                    input;
                    output;
                }
                address 37.123.100.122/29;
            }
        }
    }
    xe-1/0/1 {
        description "Ex4500 Downlink";
        unit 0 {
            family inet {
                address 185.9.157.1/27;
            }
        }
    }
}
forwarding-options {
    sampling {
        input {
            rate 100;
        }
        family inet {
            output {
                flow-server 192.168.10.206 {
                    port 2055;
                }
            }
        }
    }
}
snmp {
    location izmir;
    contact "Cahit Eyigunlu";
    community SALAY {
        authorization read-only;
    }
}
routing-options {
    static {
        route 0.0.0.0/0 next-hop 37.123.100.121;
    }
}
policy-options {
    prefix-list block.zeusCC {
        185.9.156.2/32;
    }
    prefix-list unblock.zeusCC;
}
security {
    idp {
        idp-policy Blocker {
            rulebase-ips {
                rule BlockTTL {
                    match {
                        from-zone any;
                        source-address any;
                        to-zone any;
                        destination-address any;
                        application default;
                        attacks {
                            custom-attacks Block_TTL;
                        }
                    }
                    then {
                        action {
                            drop-packet;
                        }
                        ip-action {
                            ip-block;
                            target source-address;
                            timeout 6000;
                        }
                        notification;
                    }
                }
            }
        }
        active-policy Blocker;
        custom-attack Block_TTL {
            recommended-action drop-packet;
            severity major;
            time-binding {
                count 10;
                scope source;
            }
            attack-type {
                signature {
                    protocol-binding {
                        udp {
                            minimum-port 0 maximum-port 65535;
                        }
                    }
                    context packet;
                    direction any;
                    shellcode intel;
                    protocol {
                        ipv4 {
                            ttl {
                                match equal;
                                value 0;
                            }
                        }
                    }
                }
            }
        }
        security-package {
            url https://services.netscreen.com/cgi-bin/index.cgi;
        }
    }
    screen {
        ids-option Protection {
            icmp {
                ip-sweep threshold 1000;
                fragment;
                large;
                flood threshold 1000;
                ping-death;
            }
            ip {
                bad-option;
                record-route-option;
                timestamp-option;
                security-option;
                stream-option;
                spoofing;
                source-route-option;
                loose-source-route-option;
                strict-source-route-option;
                unknown-protocol;
                tear-drop;
            }
            tcp {
                syn-fin;
                fin-no-ack;
                tcp-no-flag;
                syn-frag;
                port-scan threshold 1000;
                syn-ack-ack-proxy threshold 512;
                syn-flood {
                    alarm-threshold 512;
                    attack-threshold 200;
                    source-threshold 750;
                    destination-threshold 1000;
                    timeout 5;
                }
                land;
                winnuke;
            }
            udp {
                udp-sweep threshold 500000;
            }
            limit-session {
                source-ip-based 500;
            }
        }
    }
    forwarding-process {
        application-services {
            session-distribution-mode hash-based;
        }
    }
    policies {
        from-zone trust to-zone untrust {
            policy default-permit {
                match {
                    source-address any;
                    destination-address any;
                    application any;
                }
                then {
                    permit;
                }
            }
        }
        from-zone untrust to-zone trust {
            policy default-permit {
                match {
                    source-address any;
                    destination-address any;
                    application any;
                }
                then {
                    permit {
                        application-services {
                            idp;
                        }
                    }
                }
            }
        }
    }
    zones {
        security-zone trust {
            screen Protection;
            host-inbound-traffic {
                system-services {
                    all;
                }
                protocols {
                    all;
                }
            }
            interfaces {
                ge-0/0/1.0;
                xe-1/0/1.0;
            }
        }
        security-zone untrust {
            screen Protection;
            host-inbound-traffic {
                system-services {
                    all;
                }
                protocols {
                    all;
                }
            }
            interfaces {
                xe-1/0/0.0;
            }
        }
    }
}
firewall {
    family inet {
        filter BlokKural {
            term Allow-NTP {
                from {
                    source-address {
                        192.168.1.1/32;
                    }
                    protocol udp;
                    port ntp;
                }
                then accept;
            }
            term Block-NTP {
                from {
                    protocol udp;
                    port ntp;
                }
                then {
                    discard;
                }
            }
            term 0 {
                from {
                    packet-length [ 29 1048 ];
                }
                then {
                    discard;
                }
            }
            term 500 {
                then accept;
            }
            term 1 {
                from {
                    protocol udp;
                    tcp-flags 0x00;
                }
                then {
                    discard;
                }
            }
        }
    }
}

Open in new window

0
FireBallITAuthor Commented:
And also icmp performance is bad. Now we are getting an icmp attack , i redirected traffic to the srx and it hased nearly crashed ping times get 20x bigger

there sis something wrong with tat

icmpicmp2icmp3


           2015-05-15 22:55:05.852      77.223.129.51   178.20.231.1    53      58197   705     0x00    17      3200
           2015-05-15 22:55:06.021      89.233.169.69   178.20.231.1    11      0       74      0x00    1       3200
           2015-05-15 22:55:06.021      77.74.11.9      178.20.231.1    11      0       74      0x00    1       3200
           2015-05-15 22:55:06.021      185.15.196.228  178.20.231.1    53      44932   705     0x00    17      3200
           2015-05-15 22:55:06.021      185.22.185.130  178.20.231.1    53      36270   705     0x10    17      3200
           2015-05-15 22:55:06.021      176.103.210.249 178.20.231.1    3       3       110     0x18    1       3200
           2015-05-15 22:55:06.195      212.68.57.214   178.20.231.1    53      48220   705     0x00    17      3200
           2015-05-15 22:55:06.195      185.84.221.191  178.20.231.1    25830   46120   1494    0x00    17      3200
           2015-05-15 22:55:06.195      176.53.125.191  178.20.231.1    53      38888   705     0x00    17      3200
           2015-05-15 22:55:06.195      176.40.48.200   178.20.231.1    27416   38542   82      0x00    17      3200
           2015-05-15 22:55:06.195      185.40.194.173  178.20.231.1    19823   38944   894     0x00    17      3200
           2015-05-15 22:55:06.196      176.241.150.241 178.20.231.1    27754   44839   71      0x10    17      3200
           2015-05-15 22:55:06.367      185.9.209.34    178.20.231.1    12987   49526   1240    0x00    17      3200
           2015-05-15 22:55:06.367      95.154.247.242  178.20.231.1    3       1       74      0x00    1       3200
           2015-05-15 22:55:06.367      185.12.187.29   178.20.231.1    10136   62613   977     0x00    17      3200
           2015-05-15 22:55:06.367      185.39.24.252   178.20.231.1    3       3       110     0x00    1       3200
           2015-05-15 22:55:06.367      185.47.5.151    178.20.231.1    12375   62360   528     0x00    17      3200
           2015-05-15 22:55:06.367      91.191.170.73   178.20.231.1    11      0       74      0x00    1       3200
           2015-05-15 22:55:06.538      176.40.48.172   178.20.231.1    53      61288   82      0x00    17      3200
           2015-05-15 22:55:06.539      78.135.109.94   178.20.231.1    53      33573   705     0x00    17      3200
           2015-05-15 22:55:06.539      185.47.4.246    178.20.231.1    17035   49862   528     0x00    17      3200
           2015-05-15 22:55:06.539      212.68.42.212   178.20.231.1    53      33036   705     0x00    17      3200
           2015-05-15 22:55:06.539      176.10.213.104  178.20.231.1    26797   34418   1494    0x00    17      3200
           2015-05-15 22:55:06.706      77.223.153.119  178.20.231.1    53      52954   705     0x00    17      3200
           2015-05-15 22:55:06.706      212.68.57.28    178.20.231.1    53      58082   705     0x10    17      3200
           2015-05-15 22:55:06.707      185.39.27.55    178.20.231.1    3       3       110     0x10    1       3200
           2015-05-15 22:55:06.707      185.73.203.101  178.20.231.1    53      36796   705     0x10    17      3200
           2015-05-15 22:55:06.707      185.85.190.241  178.20.231.1    53      51879   705     0x10    17      3200
           2015-05-15 22:55:06.707      185.47.6.33     178.20.231.1    14808   34807   528     0x10    17      3200
           2015-05-15 22:55:06.904      78.135.101.180  178.20.231.1    53      51001   705     0x10    17      3200
           2015-05-15 22:55:06.904      176.227.184.199 178.20.231.1    23812   51467   755     0x10    17      3200
           2015-05-15 22:55:06.905      185.14.208.207  178.20.231.1    23812   51467   64      0x10    17      3200
           2015-05-15 22:55:06.905      185.14.210.89   178.20.231.1    26604   53901   1494    0x10    17      3200
           2015-05-15 22:55:07.075      176.40.209.48   178.20.231.1    17217   64899   82      0x00    17      3200
           2015-05-15 22:55:07.075      185.85.190.234  178.20.231.1    53      49800   705     0x04    17      3200
           2015-05-15 22:55:07.075      176.53.125.247  178.20.231.1    53      52464   705     0x04    17      3200
           2015-05-15 22:55:07.075      212.68.48.105   178.20.231.1    53      40119   809     0x04    17      3200
           2015-05-15 22:55:07.075      195.74.128.75   178.20.231.1    3       3       110     0x04    1       3200
           2015-05-15 22:55:07.075      176.40.1.130    178.20.231.1    15043   44535   82      0x04    17      3200
           2015-05-15 22:55:07.247      185.22.185.136  178.20.231.1    53      50872   705     0x00    17      3200
           2015-05-15 22:55:07.247      176.43.200.81   178.20.231.1    53      42596   235     0x00    17      3200
           2015-05-15 22:55:07.247      176.40.48.200   178.20.231.1    27416   33616   82      0x00    17      3200
           2015-05-15 22:55:07.247      212.68.44.132   178.20.231.1    53      37124   705     0x10    17      3200
           2015-05-15 22:55:07.247      87.106.216.135  178.20.231.1    3       3       110     0x10    1       3200
           2015-05-15 22:55:07.247      176.53.119.218  178.20.231.1    53      49693   705     0x10    17      3200
           2015-05-15 22:55:07.420      176.53.125.206  178.20.231.1    53      50401   705     0x18    17      3200
           2015-05-15 22:55:07.420      62.117.4.21     178.20.231.1    11      0       74      0x10    1       3200
           2015-05-15 22:55:07.420      77.223.131.38   178.20.231.1    53      64178   497     0x10    17      3200
           2015-05-15 22:55:07.596      87.99.33.168    178.20.231.1    3       3       110     0x00    1       3200
           2015-05-15 22:55:07.597      195.74.128.208  178.20.231.1    3       3       110     0x00    1       3200
           2015-05-15 22:55:07.597      176.67.66.29    178.20.231.1    14584   63459   862     0x00    17      3200
           2015-05-15 22:55:07.597      78.187.159.133  178.20.231.145  50898   9958    80      0x00    17      3200
           2015-05-15 22:55:07.597      212.68.48.92    178.20.231.1    53      47753   809     0x00    17      3200
           2015-05-15 22:55:07.597      185.85.190.205  178.20.231.1    53      47710   705     0x00    17      3200
           2015-05-15 22:55:07.597      78.135.101.180  178.20.231.1    53      47066   705     0x00    17      3200
           2015-05-15 22:55:07.773      185.14.211.227  178.20.231.1    17152   37368   1494    0x10    17      3200
           2015-05-15 22:55:07.773      77.223.134.83   178.20.231.1    53      56075   705     0x10    17      3200
           2015-05-15 22:55:07.924      185.9.210.113   178.20.231.1    21489   52444   1240    0x00    17      3200
           2015-05-15 22:55:07.924      212.68.57.231   178.20.231.1    53      37663   705     0x00    17      3200
           2015-05-15 22:55:07.924      176.40.101.0    178.20.231.1    23469   49218   82      0x00    17      3200
           2015-05-15 22:55:07.924      185.73.201.78   178.20.231.1    53      61892   705     0x00    17      3200
           2015-05-15 22:55:07.924      185.85.238.203  178.20.231.1    53      61382   705     0x00    17      3200
           2015-05-15 22:55:07.924      176.31.243.162  178.20.231.1    53      60421   727     0x00    17      3200
           2015-05-15 22:55:08.103      176.109.179.245 178.20.231.1    3       3       110     0x00    1       3200
           2015-05-15 22:55:08.103      195.22.214.87   178.20.231.1    11      0       74      0x00    1       3200

Open in new window

0
FireBallITAuthor Commented:
d
this attack is coming from real resources and srx lost all connection with 100k pps :)
0
FireBallITAuthor Commented:
I realize that there was 2 attack comed.

1. ICMP
2. DNS Amplification because source ports was 53 and i re-simulated the attack. 100K pps of DNS amp attack getting down the device too
0
giltjrCommented:
Looks like you have removed a lot of stuff from the last config you posted in one of your other questions.   Just a couple of notes.

1) If possible use https instead of http, again for better security.
2) You have all 3 interfaces enabled for management.  At a minimum disable management for xe-1/0/0.  You don't want somebody trying to hit the management interface from a untrusted interface ever.  Preferably  dedicated one interface just for management and don't  allow any other traffic on that interface.
3) Disable telnet.  This is very unsecure.  If for some reason you must have telnet enable, then please only allow management access from a inside secure interface.

These won't change anything, but just makes the box a little more secure.
0
FireBallITAuthor Commented:
yes but it is test lab at the moment i do not understand why it has this problems with udp :(
0
FireBallITAuthor Commented:
additionally i do not think that is an npc / pps issue because when we activate the screen rules for threshold there is no more it creates a problem
0
giltjrCommented:
I a little confused and will need to do some more reading, but from what I can tell you should be discarding all UDP traffic coming in on xe-1/0/0.  On that interface you have filter { input BlokKural; }

In BlokKural you have  :
 
            term 1 {
                from {
                    protocol udp;
                    tcp-flags 0x00;
                }
                then {
                    discard;
                }

Which I believe says that if the input packet is UDP or TCP and has no TCP flags set, then discard the packet.
0
FireBallITAuthor Commented:
actually that rule is not working as expected it never blocked any packet. so we did not removed it.

I had made some tests

ACK attack spoof ip
DNS Amplification attack
udp spoof flood
udp same ip smal packet flood

all of them give similar results as it has 1 SPC , beside this when we apply screen rules it stops the attacks. (not as expected)

so do you have any idea about that  ? :S
0
giltjrCommented:
Relooking at BlokKural  the problem may be the order the the term.  Right before term 1 you have term 500 which accepts everything.  Firewall do not process "best" match, they process "first match"  So the way BlokKural should work is

Allow traffic from 192.168.1.1/32 if it is UDP and port 123 (NTP).

Discard traffic if UDP and port 123 (NTP) from any other address.

Discard traffic if the packet length is between 29 bytes and 1024 bytes. @@@ I would expect most traffic to be dropped here.

Allow everything else.  @@@ Since all traffic that gets here matches no other terms should be looked at.

Discard if udp or if tcp and no flags are set. @@@ this should never get hit because all other traffic was allowed by prior term.
0
FireBallITAuthor Commented:
we tryed only term 0 block udp , term 1 allow all but it does not resolve the issue too
0
FireBallITAuthor Commented:
From the juniper's mailing list i got an logical answer :



Cahit,

We ran into exact same issue with our SRX3600 a few months ago.  It took us weeks of struggle with J-TAC, before we figured out what the issue was.  In a nutshell, the behavior you've been experiencing is, believe or not, the "by-design" SRX3k behavior.  By default, an SRX3k firewall uses its lowest SPC card for both load distribution and traffic processing, in your case SPC 7.  SRX reserves only half of its SPC card capacity for traffic processing.  Moreover, once a "load" is assigned to a card, in our case it was an IPSec tunnel, it stays there even if that load keeps increasing.  You can check your SPC card CPU utilization by running the following shell command.  If you are seeing half of your CPU's at 99%, you sure are dropping packets on that SPC.  

root@hostname% vty node0.fpc7.pic0


BSD platform (XLR processor, 4096MB memory, 16384KB flash)

[flowd64]FPC7.PIC0(vty)# sh xlr cpu
CPU 04 Per second( 44)
CPU 05 Per second( 52)
CPU 08 Per second(  0)
CPU 09 Per second(  0)
CPU 10 Per second(  0)
CPU 11 Per second(  0)
CPU 12 Per second(  0)
CPU 13 Per second(  0)
CPU 14 Per second(  0)
CPU 15 Per second(  0)
CPU 16 Per second( 99)
CPU 17 Per second( 99)
CPU 18 Per second( 99)
CPU 19 Per second( 99)
CPU 20 Per second( 99)
CPU 21 Per second( 99)
CPU 22 Per second( 99)
CPU 23 Per second( 99)
CPU 24 Per second( 99)
CPU 25 Per second( 99)
CPU 26 Per second( 99)
CPU 27 Per second( 99)
CPU 28 Per second( 99)
CPU 29 Per second( 99)
CPU 30 Per second( 99)
CPU 31 Per second( 99)
Average [cpu8-31]( 66)
        xlr_show_cpu_statistics session counters:
        Current Flow sessions                                             23757
        Current Flow sessions IPv4                                        23757
        Current Flow sessions IPv6                                            0
        Current CP   sessions                                             26923
        Current CP   sessions IPv4                                        26924
        Current CP   sessions IPv6                                            0
        Current CP   sessions installed                               157369031
        Current CP   sessions installed IPv4                          157369031
        Current CP   sessions installed IPv6                                  0



Strangely enough, the rest of your SPC cards could be sitting doing nothing, as you are dropping packets on the other card.

[flowd64]FPC8.PIC0(vty)# show xlr cpu
CPU 04 Per second( 34)
CPU 05 Per second( 23)
CPU 08 Per second( 29)
CPU 09 Per second( 28)
CPU 10 Per second( 29)
CPU 11 Per second( 29)
CPU 12 Per second( 29)
CPU 13 Per second( 28)
CPU 14 Per second( 28)
CPU 15 Per second( 28)
CPU 16 Per second( 28)
CPU 17 Per second( 27)
CPU 18 Per second( 28)
CPU 19 Per second( 28)
CPU 20 Per second( 29)
CPU 21 Per second( 27)
CPU 22 Per second( 28)
CPU 23 Per second( 28)
CPU 24 Per second( 27)
CPU 25 Per second( 25)
CPU 26 Per second( 27)
CPU 27 Per second( 26)
CPU 28 Per second( 28)
CPU 29 Per second( 27)
CPU 30 Per second( 28)
CPU 31 Per second( 28)
Average [cpu8-31]( 28)
xlr_show_cpu_statistics session counters:
Current Flow sessions                                             16353
Current Flow sessions IPv4                                        16353
Current Flow sessions IPv6                                            0
Current CP   sessions                                                 0
Current CP   sessions IPv4                                            0
Current CP   sessions IPv6                                            0
Current CP   sessions installed                                       0
Current CP   sessions installed IPv4                                  0
Current CP   sessions installed IPv6                                  0


To me, this is an obvious flaw in the SRX3k design.  I am not sure whether the same applies to SRX5k or not.  Juniper's solution to this problem was to give us an extreme performance license, which dedicates your lowest SPC card entirely to load balancing and can dynamically shift loads between the rest of SPC's.

Here is the license SKU: SRX3K-EXTREME-LTU Expanded performance and capacity Extreme License for SRX3000 line

By the way, Juniper did not charge us a dime for the license, which confirms they admit the design flaw.  This feature should really be a part of Junos functionality on SRX3k out of the box, otherwise it is just a ticking bomb that will get you when you least expect it.

If I were you, I would press on your Juniper rep to get the license and re-run your tests.

Hope this helps.

Cheers,

Gennadiy
0
giltjrCommented:
Sounds like a good chance that is your problem.  I do remember reading about one of the SPC's CPU's being used for two difference purposes, and I believe I posted this in one of your other questions dealing with how much traffic a SRX can use.
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
FireBallITAuthor Commented:
Problem is aboutthe juniper architecture thank you for your help
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Hardware Firewalls

From novice to tech pro — start learning today.

Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.