Cisco 7200 ARP Flooding

Why are you running the interface as half-duplex? performance will improve if you run full duplex.
Unless you need it for something specific, i would turn off proxy arp with:
no ip proxy-arp  on the interface.

Is the router using cef?

If the switch isn't addressable, try this:
(after description and IP address)
no ip redirects
no ip directed-broadcast
no keepalive
duplex half (unless you think you can use full)
speed 100
no cdp enable

That's all you need.  If you want to really control braodcasts, give the IP address as a .252 mask and have it talk with an addressable port on the switch.  It sounds like the switch takes care of the broadcasts and right now you have duplicates.  Right now, devices think they are directly connected to the rotuer until they get an arp from the switch then back again, so of course it's going to interfere.  (I'm assuming the router is connected to the switch).

has anyone considered passive interface , prevent the router from advertising

Alright guys, per your suggestions I've applied the following to the local fast ethernet interface:

no ip redirects
no ip directed-broadcast
no keepalive
duplex half (for testing)
speed 100
no cdp enable
no ip proxy-arp

For our local LAN via fast ethernet to a switch:
AM6#h show int f4/0
FastEthernet4/0 is up, line protocol is up
  Hardware is DEC21140A, address is 00d0.c018.2c70 (bia 00d0.c018.2c70)
  Description: XO LAN
  Internet address is 66.238.54.1/24
  MTU 1500 bytes, BW 100 Kbit, DLY 100 usec,
     reliability 255/255, txload 5/255, rxload 2/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Half-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 00:10:00
  Last input 00:00:01, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:00:34
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue :0/40 (size/max)
  5 minute input rate 1000 bits/sec, 1 packets/sec
  5 minute output rate 2000 bits/sec, 2 packets/sec
     45 packets input, 4457 bytes
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog
     0 input packets with dribble condition detected
     99 packets output, 7850 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

**This interface's counters were reset just after I made the new changes.  It's still maintaining a 2:1 output.
AM6#show run int 4/0
Building configuration...

Current configuration : 219 bytes
!
interface FastEthernet4/0
 description XO LAN
 bandwidth 100
 ip address 66.238.54.1 255.255.255.0
 no ip redirects
 no ip unreachables
 no ip proxy-arp
 no keepalive
 duplex half
 arp timeout 600
 no cdp enable
end

On the flip side we've got our IP transit on f2/0:
AM6#show run int 2/0
Building configuration...

Current configuration : 107 bytes
!
interface FastEthernet2/0
 ip address 66.238.50.82 255.255.255.252
 no ip unreachables
 duplex full
end

AM6#show int f2/0
FastEthernet2/0 is up, line protocol is up
  Hardware is DEC21140, address is 00d0.c018.2c38 (bia 00d0.c018.2c38)
  Internet address is 66.238.50.82/30
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/7/512 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue :0/40 (size/max)
  5 minute input rate 4000 bits/sec, 7 packets/sec
  5 minute output rate 2000 bits/sec, 4 packets/sec
     4154239 packets input, 2926193764 bytes
     Received 5259 broadcasts, 0 runts, 0 giants, 7 throttles
     2 input errors, 0 CRC, 0 frame, 0 overrun, 2 ignored
     0 watchdog
     0 input packets with dribble condition detected
     2469998 packets output, 504109398 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

The router is running CEF and subnet-zero.  Here's the ARP debug:

AM6#
3d15h: IP ARP: creating incomplete entry for IP address: 66.238.54.69 interface FastEthernet4/0
3d15h: IP ARP: sent req src 66.238.54.1 00d0.c018.2c70,
                 dst 66.238.54.69 0000.0000.0000 FastEthernet4/0
3d15h: IP ARP: creating incomplete entry for IP address: 66.238.54.214 interface FastEthernet4/0
3d15h: IP ARP: sent req src 66.238.54.1 00d0.c018.2c70,
                 dst 66.238.54.214 0000.0000.0000 FastEthernet4/0
3d15h: IP ARP: creating incomplete entry for IP address: 66.238.54.4 interface FastEthernet4/0
3d15h: IP ARP: sent req src 66.238.54.1 00d0.c018.2c70,
                 dst 66.238.54.4 0000.0000.0000 FastEthernet4/0
3d15h: IP ARP: sent req src 66.238.54.1 00d0.c018.2c70,
                 dst 66.238.54.237 0000.0000.0000 FastEthernet4/0
3d15h: IP ARP: creating incomplete entry for IP address: 66.238.54.14 interface FastEthernet4/0
3d15h: IP ARP: sent req src 66.238.54.1 00d0.c018.2c70,
                 dst 66.238.54.14 0000.0000.0000 FastEthernet4/0
3d15h: IP ARP: sent req src 66.238.54.1 00d0.c018.2c70,
                 dst 66.238.54.69 0000.0000.0000 FastEthernet4/0
3d15h: IP ARP: sent req src 66.238.54.1 00d0.c018.2c70,
                 dst 66.238.54.4 0000.0000.0000 FastEthernet4/0
3d15h: IP ARP: sent req src 66.238.54.1 00d0.c018.2c70,
                 dst 66.238.54.214 0000.0000.0000 FastEthernet4/0undebug
3d15h: IP ARP: creating incomplete entry for IP address: 66.238.54.239 interface FastEthernet4/0

Thanks!

Here's what I think is happening:

These incomplete arp entries are coming from the WAN link (which you're thinking is really odd).  This is because you have a routing protocol trying to reach that network 66.238.54.14.  It sends the arp request but isn't connected to that network.  You probably need to check your policy-map statements or set a route map with the correct next hop to that network.  Check this article and see if it helps:

http://www.cisco.com/en/US/tech/tk365/tk80/technologies_tech_note09186a008009481d.shtml

I don't have any policy maps or route maps setup.  Are you suggesting I setup a next-hop policy on the WAN link for all addresses within that class C to go to the f4/0?

From my experience there are several reasons this ARP traffic could be occuring.  What I really need to know is, where is that 66.238.54.14 device and others like it that are showing up as Incomplete.  Is it directly connected (hub or layer 2 switch), through another router/bridge, etc.?

If you don't know, disconnect or admin down one side and see if the messages continue.  Once you've isolated where that is coming from then we can game plan better I think.

Some possibilities:
1.  A device connected to your router has an old ARP cache including a lot of devices no longer on the network.  It needs to have its cache flushed or the device rebooted.

2.  A bridge in the network isn't passing the complete MAC header and now you're seeing a stripped header in your router.  If that's the case, try routing that traffic instead of bridging it.

3.  You have a routing protocol on the wrong interface or not directly connected to its protocol neighbor, it's expecting ARP from one side and failing because the requests go out the other interface or are unable to complete because the next hop isn't bridging the request.  (Thats the previous post's assumption).

4.  You have a DOS attack from a connected device creating bad packets that are causing multiple nodes to arp continuously.  Not likely since you did some isolation before.

There are probably others, but we need to isolate them one at a time.  What protocols are you running and on what interfaces?

Here's the setup:
IP comes into f2/0 on a point-to-point FEthernet interface on a .252 subnet

IP either goes out through a DS3 to broadband customers or to a local 3Com 3300 switch with 4 network connections forming the 66.238.54.0/24 network.

The switch connects to f4/0 on the router.  On the switch are two linux boxes (currently powered down) and an IBM server running Win03 Ent. server.  The IBM has two network cards:

One NIC, referred to as the frontside goes out of the card, to an Netscreen-5 running in transparent, and then to the switch.

Second NIC, referred to as backside, goes out of the card, to a Netscreen-5 running in NAT, and then to the switch.  This second NS5 is used for a company VPN and while we know the first NS5 can do transparent VPN it wasn't working for us and a quick solution was to set it up in the fashion it is now.

In total, 9 IP addresses are currently in use:
66.238.54.1 -router
66.238.54.10 -linux (disc.)
66.238.54.11 -linux (disc.)
66.238.54.30 -game server running on linux box .10 (disc.)
66.238.54.246 -NS5 backside running NAT
66.238.54.250 -network address of frontside NS5 (essentially not used)
66.238.54.251 -IBM server via frontside
66.238.54.253 -admin interface for NS5
66.238.54.254 -second IP address on frontside IBM for alternate website

BGP is setup on this router since we have 3 class Cs to our name and will acquire a second IP feed.  Per the BGP settings, we have anchored the IPs to our router via ip route xxx.xxx.xxx.xxx Null 0.

Isolation measures taken:
Disconnected all nodes from switch; still broadcasted
Disconnected switch; via Fluke meter saw that router was still broadcasting
Disconnected IP transit, broadcast stops

Thanks for the info AutoSponge

I'm not understanding your problem. Your "sh int" shows you have a total of 1-2 pps on the router. I'm not sure why you are saying that's a problem. If you are getting too many arps, most switches have a broadcast suppression mechanism to let a certain percentage of packets through.

Furthermore, if you plug a sniffer into a switch port, the only thing you will see is broadcasts. Unicast traffic does not go out the switch port, only broadcasts and multicasts.

Finally, I would turn your keepalives back on. If your ethernet port gets disconnected, the ethernet interface will still appear up to the router.

-Eric

Yeah, I think this has something to do with BGP rather than the interface itself.  BGP is trying to establish direct connection with something but is looking at the wrong interface (apparently).  Show me what you have configured for BGP please.  Again, that first link I posted may help describe the problem (although I think the example was using EIGRP).

Eric,

I understand that without keepalive the link will appear on and indeed you are not seeing the problem.  If I plug a network tool directly into the f4/0 port I see a continuous pulse with 100% broadcast of ARP.  If the router isn't connected to the switch, it shouldn't see an ARP unless some foreign source was requesting it unappropriately.

By doing the show int for that interface, I see that my output is nearly double my input which shouldn't be, especially just after a count reset.

The issue lies as either a hardware issue or an ACL

autonomous-system 32785
!
router bgp 32785
no synchronization
bgp log-neighbor-changes
bgp deterministic-med
network 66.238.54.0 mask 255.255.255.0
network 209.125.57.0
network 209.218.23.0
neighbor 66.238.50.81 remote-as 2828
neighbor 66.238.50.81 description XO Communications:AS2828:BGP@xo.com:800-575-6398
neighbor 66.238.50.81 send-community
neighbor 66.238.50.81 soft-reconfiguration inbound
no auto-summary
!
ip classless
ip route 66.238.54.0 255.255.255.0 Null0
ip route 209.125.57.0 255.255.255.0 Null0
ip route 209.218.23.0 255.255.255.0 Null0
ip route 209.218.23.104 255.255.255.248 209.218.23.102
no ip http server
ip pim bidir-enable

call TAC you need a new backplane

Well after all of this, I have one question still, why are you using Internet address is 66.238.54.1/24 on that f4/0?  Is this just connected to a dumb hub and the whole class C is directly connected?

Auto:

That entire class C is set for f4/0.  The network is designed for game server deployment so an entire class C was allocated.

Happy:

New chassis right?  Not NPE or I/O board?

Auto:

I foregot to explain the class C requirement.  Each linux system hold about 10 game servers=10 IP addresses.  20 linux boxes to be connected to a 24 port switch.  200 IPs plus some FTP and web.

Try changing that interface to a 30-bit mask and see if it still arps like that.  If so, yes open a TAC case for sure.

I'll do that.  Why can't the router handle a /24?  Just not designed?

router is designed to handle any bitmask , its def a TAC issue

So you put statics in your config to make sure BGP always announces them. Where do those networks live? Are you really doing anything with them? Are you using something for an IGP so your 7200 thinks it knows how to get to those networks? Typically the route to null0 will have a higher AD so some other route will be preferred.

Please post the output of "sh ip route | inc 209.218.23", "sh ip route | inc 209.125.57" and "sh ip route | inc 66.238.54" if you think those networks should be reachable by your router.

-Eric

I was thinking that was an issue also Eric, but I reconsidered when I looked at the debug arp output again.  All of those incompletes are in the .54.x network.  So it's all suppose to be connected to the f4/0 and shouldn't have anything to do with his BGP neighbors (like I thought before).

Are your game servers running iptables? Could it be misconfigured in that it is not allowing arps to some of those "virtual" ip addresses?

And perhaps I'm not getting it, but I'm not seeing why you think incomplete arps are a problem. If you look at the input/output rate, you barely have any traffic going through there. Even if you were getting 100-200 arps/second, that wouldn't slow down a network or server. Finally, on what basis do you think that 2:1 ratio of output to input packets is bad/wrong/incorrect?

-Eric

I tried to focus on this... my kid is on a sugar high, so it wasn't easy... here's my line of thinking right now:

Your router is trying to forward to what it thinks is a local address because someone is sending packets to that IP.  If it's not in the ARP table, then your router will broadcast an ARP request to find out the MAC address for the destination IP.

So, if you know certain hosts aren't going to exist on that LAN interface, you can choose to drop requests to those hosts and thereby skip the whole broadcast ARP thing.  This would be why when you disconnect from the WAN you no longer get those broadcasts.  The real question is, if those are all the hosts you have (9 or whatever) but you're getting requests for  66.238.54.214, .69, .4, etc. where are they coming from and do you have the right mask on your interface (meaning, do those hosts actually live somewhere else).

I think if you shrink the LAN subnet with a higher mask and it still does this high rate of broadcast traffic, you have a problem.  Otherwise just verify that you have the right network and mask on the LAN interface.  If it won't mess up your routing, you can always choose to not redistribute that 66.238.54.0 network and probably cut down on requests to those hosts.  If that doesn't work and the traffic still bothers you, block requests at the WAN port for everything but your hosts, or drop them in a Null0 static route.

Changed f4/0 to a .192 subnet, still broadcasting and a show debug still has all of those ARP entries incomplete.

Either I have a hardware failure on the chassis or I need to setup a list to prevent requests for unused address space.  Can anyone explain how to construct a list to deny all traffic to 66.238.54.0/24 unless its one of the 9 addresses active on the block?  I'm assuming I'll put this on the WAN interface..

To answer Eric's questions, my game servers are not running IP tables.  To break it down further, we've disconnected just about all connections on this switch with the exception of one web server which has been verified and restarted multiple times.  

I'm not generally saying that a 2:1 flow is bad or that 100-200 ARPs is bad.  BUT, when 100-200 ARPs are for addresses that don't even exist and when we watch them come into the router, the ARPs are for random and sometimes sequential addresses, we think it's wrong.  Not to mention, 90% of the ARPs we get are for addresses we've never even used yet.

The bottom line here, and what has stemmed our investigation into this whole issue is that whatever this problem is, it's causing intermittent loss of connectivty to the web server thats on the network.  It may be causing other loss of connectivity, but no other machines are running services that we can go to and check.  We have a program monitoring nodes via ping which shows good connections except to the web server, and consequently the only comptuer on the network that gets traffic (a little at that).

To answer an upcoming question, yes, we've tried to examine the server as well.  In fact our original theory was the server itself, but it's brand new, we've swapped out the NIC and done an entire reload of the system.

Even if the server isn't connected to the switch the router still broadcasts like hell.

Are those 9 addresses being re-dist into the network by your router?  Try not advertising those routes and see if the traffic to the bogus addresses stop.

Or, you can try this:

SUMMARY STEPS
1. enable
2. configure terminal
3. interface type number
4. ip address ip-address mask
5. arp authorized
6. arp timeout seconds
7. end

This turns off dynamic ARP, now you need to set up static ARP for the hosts you know you'll talk to (and are able to reach):

1. enable
2. configure terminal
3. arp a.b.c.d H.H.H arpa fa4/0
(reapeat for each host)
4.  end

There's no arp authorized.

>arp ?
arpa
frame-relay
probe
snap
timeout

Went into config, and did a no arp arpa for f4/0 and got the following on arp debug:

AM6#
5d20h: IP ARP: creating incomplete entry for IP address: 66.238.54.251 interface FastEthernet4/0
5d20h: IP ARP: creating incomplete entry for IP address: 66.238.54.252 interface FastEthernet4/0
5d20h: IP ARP throttled out the ARP Request for 66.238.54.251
5d20h: IP ARP: creating incomplete entry for IP address: 66.238.54.111 interface FastEthernet4/0
5d20h: IP ARP throttled out the ARP Request for 66.238.54.251
5d20h: IP ARP: creating incomplete entry for IP address: 66.238.54.162 interface FastEthernet4/0
5d20h: IP ARP throttled out the ARP Request for 66.238.54.251
5d20h: IP ARP: creating incomplete entry for IP address: 66.238.54.48 interface FastEthernet4/0
5d20h: IP ARP throttled out the ARP Request for 66.238.54.251
5d20h: IP ARP throttled out the ARP Request for 66.238.54.251
5d20h: IP ARP throttled out the ARP Request for 66.238.54.251
5d20h: IP ARP throttled out the ARP Request for 66.238.54.251
5d20h: IP ARP throttled out the ARP Request for 66.238.54.251
5d20h: IP ARP: creating incomplete entry for IP address: 66.238.54.86 interface FastEthernet4/0u
5d20h: IP ARP: creating incomplete entry for IP address: 66.238.54.197 interface FastEthernet4/0
5d20h: IP ARP: creating incomplete entry for IP address: 66.238.54.195 interface FastEthernet4/0ndebug
5d20h: IP ARP: creating incomplete entry for IP address: 66.238.54.10 interface FastEthernet4/0
5d20h: IP ARP: creating incomplete entry for IP address: 66.238.54.95 interface FastEthernet4/0 all
All possible debugging has been turned off
AM6#

I think it's a 12.0 enhancement... too bad.  

It's time to make an ACL I think for the WAN port(s).  You need to block all requests for that net except your IPs:

interface FasterEthernet4/0
  description

I'm not trying to directly say, 'these arps are causing this system to go unavailable,' but when I've got a machine that I've already swapped NICs and all that good stuff and I go down to our rack to a look at whats going on, plug in a tester and see the router ARPing the hell out of the network and the tool shows that the router is generating 98% of the traffic on the switch with 100% broadcast, what more would I be concerned with?  It didn't used to do this and I've replaced the switch already as well as every NIC on the network, what more can I do?

I'm well aware the people ping blocks of IPs but why our's and why continuously; it never stops.  The people I know that enjoy doing that stuff usually move on to another block of IPs when there's nothing to do with the current.  The only vulnerable machine is the windows server but it sits behind a brand new NS5GT firewall and it's logs don't show and malicious attempts.

Either way, I've added the access list according to Auto's specs but made changes based on Eric's example by putting it on the WAN side of f2/0 instead of the local f4/0.  Looks like this:

 --More--         access-list 101 permit ip any host 66.238.54.1
 --More--         access-list 101 permit ip any host 66.238.54.251
 --More--         access-list 101 permit ip any host 66.238.54.246
 --More--         access-list 101 permit ip any host 66.238.54.254
 --More--         access-list 101 permit ip any host 66.238.54.253
 --More--         access-list 101 permit ip any host 66.238.54.10
 --More--         access-list 101 permit ip any host 66.238.54.11
 --More--         access-list 101 deny   ip any 66.238.54.0 0.0.0.255
 --More--         access-list 101 permit ip any any

Thankfully, this ACL worked.  But, we still lose connectivity to that one server.  Only problem is, outside can't ping or get into it, but if I log into the router, I can ping the server.  I go back to my workstation, I can't ping it.  Go back into the router, pings fine.  It does this at random intervals.  I'll probably have to start a new thread for it.

I've started a new thread about the loss of connectivity: https://www.experts-exchange.com/questions/21177453/Intermittent-Loss-of-Connectivity.html

Thanks for the help AutoSponge and Eric.