[Okta Webinar] Learn how to a build a cloud-first strategyRegister Now

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 5269
  • Last Modified:

%SW_DAI-4-DHCP_SNOOPING_DENY: 1 Invalid ARPs

Workstations started getting APIPA addresses and unable to get a valid DHCP IP address.  Not all devices on the switch are experiencing this problem which is weird.  I logged into my switch and I'm seeing a ton of %SW_DAI-4-DHCP_SNOOPING_DENY: 1 Invalid ARPs trying to contact the gateway with no success.  I proceeded to think that maybe these machines were acting as a DHCP server so I shut them all down but then it just seemed to move to other interfaces.

I'm not sure how to track down these Dynamic ARP inspection denials and need assistance.  Information on switch below.

Stack
2 x 3750G
2 x 3750E

Both running 12.2(58)SE1 IPBASEK9-M
0
MontyV
Asked:
MontyV
  • 6
  • 4
  • 4
  • +1
1 Solution
 
gheistCommented:
Your IOS is EOL January 27, 2013

You need to make link between switches trusted, otherwise you cannot re-plug cables across.
0
 
MontyVAuthor Commented:
Yes the IOS is going to be upgraded to 15 in a few weeks but apparently they have been running it for 3 years.  Considering the location of the switch we are only able to bring it down during very limited windows throughout the year unless complete loss of connectivity of course or hardware failure.

ip arp inspection trust
ip dhcp snooping trust

Both are on the trunk.  This is at the edge so the uplinks to the distribution switch then to the core are all trusted with the information above.
0
 
gheistCommented:
Reload with firmware upgrade takes about 5min for 48port switch. 12.3->15 i did not need to use saved config.
Can you afford "no ip arp inspection" ? to me it seems only viable solution until upgrade to supported release is done.
0
Prepare for an Exciting Career in Cybersecurity

Help prevent cyber-threats and provide solutions to safeguard our global digital economy. Earn your MS in Cybersecurity. WGU’s MSCSIA degree program curriculum features two internationally recognized certifications from the EC-Council at no additional time or cost.

 
Craig BeckCommented:
DAI is supported on your current code.  There's no real reason to rush into the upgrade yet.

Can you post the config from each switch please?
0
 
btanExec ConsultantCommented:
DHCP snooping and DAI can coexist and often are being deployed together.
It seems that you have indeed DAI enabled, which doesn't know the MACs being refreshed, thus dropping them. May be you need to let the box relearn them. May be better when such things happens, go back and undo what you have deployed recently (DHCP Snooping).   So when you have basic connectivity you might go further and finalize what you initially was trying to implement.

http://blog.brokennetwork.ca/2011/09/dhcp-based-security-part-3-dynamic-arp.html

Side note for snooping enabled - a Cisco router will drop a DHCP packet with a GIADDR of all zeros.  Normally this is not a problem because most DHCP clients by default do not insert option 82, and therefore there is no GIADDR at all.  But when we enabled DHCP snooping on our switch, the switch, by default, inserts option 82 with a GIADDR of all zeros, and we now have the problem we are seeing. We can either tell the switch to not insert option 82, or we can tell the router to trust DHCP packets with a GIADDR of all zeros

http://blog.brokennetwork.ca/2011/12/dhcp-based-security-part-1-dhcp.html
0
 
MontyVAuthor Commented:
Ok I found this cisco bug which seems to pertain to my setup.

https://tools.cisco.com/bugsearch/bug/CSCto71183

So I went ahead and upgraded to 15.  Feeling a little positive about this actually resolving my problem and no luck.  Was going to downgrade to 12.2 (55) but didn't want to go that far back.  Decided to remove all of the port-security that I could find throughout the stack.  Shut and No shut and still DAI Deny all over the place.  I'm checking my configs and I'm certain this is right but I think ok let’s strip all of the DCHP Snooping and ARP inspection statements out also and see what happens.  Should work right.  Well it does.  Computers, Printers and all other devices get IP addresses no problems.

Ok so I turn DHCP Snooping back on and wammo it’s not working again.  This is weird so I decide to wireshark and open a SPAN session.  Did tests with normal setup, without snooping and inspection and even trusting the individual port.  Of course everything turned off I can see the DHCP communication then the gratuitous ARPs from the machine.  Turn snooping and inspection on and I see no DHCP attempts at all and only see gratuitous ARPs from an APIPA address now.

Not sure what is stopping the computer from even sending a DHCP request when snooping is turned on.  I’ve opened a case with TAC and right now I went over the same stuff and all I got was that they are stumped and wanted me to send my pcap files to them so they can analyze to possibly send a link to another bug maybe.  Well that doesn’t help.
0
 
btanExec ConsultantCommented:
apologies I do not have the login access to the link...minimally this need Cisco to be more involved in step thru and log, and the low hanging obvious misconfig should be verified on existing settings.

Below is the overall guidance at least for DHCP snooping config, thought some pt to take note of - "DHCP Snooping Configuration Guidelines" and "Minimum DHCP Snooping Configuration" section

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12-2SX/configuration/guide/book/snoodhcp.pdf

Other pointers...

Enabling DHCP Snooping Globally- Configure this command as the last configuration step (or enable the DHCP feature during a scheduled maintenance period) because after you enable DHCP snooping globally, the switch drops DHCP requests until you configure the ports.

When DHCP snooping is disabled and DAI is enabled, the switch shuts down all the hosts because all
ARP entries in the ARP table will be checked against a nonexistent DHCP database. When DHCP
snooping is disabled or in non-DHCP environments, use ARP ACLs to permit or to deny ARP packets

Do not enter the ip dhcp snooping information option allowed-untrusted command on an aggregation switch to which any untrusted devices are connected.
0
 
MontyVAuthor Commented:
Below is the bug information.  Not sure it helped as I tried both the workaround and IOS upgrades/downgrades.  I've removed all of the settings settings including the database to start from scratch but it seems each time I start readding configs I get the errors again.  

Strip all port-security, dhcp snooping and arp inspection.
Added dhcp snooping and proper trusts on trunks, recreated database as it is saved remotely.  Client starts getting APIPA addresses.
Added arp inspection then started getting DAI logs showing snooping deny.
Currently it is set as "no ip dhcp snooping information option"

TAC is still investigating.

Port security with dot1x breaks dhcp snooping
CSCto71183
Description
Symptom:

When port security, dot1x and dhcp snooping are configured on the same interface and dot1x is removed then host can no longer send traffic.

Conditions:

3750 running 122-58.SE
3560 running 122-53.SE1

Workaround:

Remove and re-add port security      

Open Support Case
Was the description about this Bug Helpful?(0)
Details
Last Modified:
Aug 22,2014
Status:
Fixed
Severity:
3 Moderate
Product:
Cisco Catalyst 3750 Series Switches
Support Cases:
8
Known Affected Releases:      (2)
12.2(58)SE
12.2(53)SE1
Known Fixed Releases:      (2)
12.2(55)SE4
15.0(1)SE1
Download software for  Cisco Catalyst 3750 Series Switches
0
 
btanExec ConsultantCommented:
By default, no VLANs have an ARP ACL applied. Also, no additional validation of ARP packets is enabled.

I was thinking if there is already some ARP ACL applied to the list of VLANs, then such symptoms may appear and probably we can use the "no" option, removes the ARP ACL (based on log) from the list of VLANs to see if DAI log still appear as you started fresh...e.g.

no ip arp inspection filter <acl-name> vlan <list>
no ip arp inspection validate {[src-mac] [dst-mac] [ip]}

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/4_1/nx-os/security/configuration/guide/sec_nx-os-cfg/sec_arpinspect.html#wp1062787

DAI determine the validity of an ARP packet based on valid IP-to-MAC address bindings stored in a DHCP snooping binding database. This database is built by DHCP snooping if DHCP snooping is enabled on the VLANs and on the device. It can also contain static entries that user create. If the ARP packet is received on a trusted interface, the device forwards the packet without any checks. On untrusted interfaces, the device forwards the packet only if it is valid.
1
 
Craig BeckCommented:
I have seen the exact same issue you described.  The fix was to install 12.2(55)SE5 or SE8
0
 
btanExec ConsultantCommented:
Looks like going to latest iOS 12.2(55)SE9 to be safe
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/release/12-2_55_se/release/notes/OL23054.html#pgfId-1044292

Caveats Resolved in Cisco IOS Release 12.2(55)SE9
CSCtq91847 (Catalyst Switches 2960, 2960-S, 2960-SM, 3560, and 3750)
The switch fails while running 802.1X configuration.
There is no workaround.

CSCui65252
When Dynamic ARP Inspection (DAI) is enabled over port channel, it stops processing the Address Resolution Protocol (ARP) packets.
The workaround is to set up a direct link between the access switch and the DHCP server.

Caveats Resolved in Cisco IOS Release 12.2(55)SE6
CSCtx61557
The switch stops working even after a successful 802.1x authentication of the client.
There is no workaround.

Caveats Resolved in Cisco IOS Release 12.2(55)SE5
CSCtt37202
If a client switch is authorized using MAC Authentication Bypass (MAB), and then by using the 802.1x standard and dynamic VLAN assignment, the MAC address of the switch is not updated in the MAC address table of slave switches.
The workaround is to not use both the 802.1x and dynamic VLAN assignment configurations for the client switch.
0
 
Craig BeckCommented:
I found SE8 to be the most stable after (in my deployments) after trying all 12.2(55) revisions.  SE9 caused other issues which may or may not be relevant here.
0
 
btanExec ConsultantCommented:
Thanks for sharing - also I was looking back at the 12.2(58) SE1
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/release/12-2_58_se/release/notes/OL24337.html#wp1043739

(Catalyst 3750 or 3560 switches and Cisco EtherSwitch service modules) When dynamic ARP inspection is enabled on a switch or switch stack, ARP and RARP packets greater than 2016 bytes are dropped by the switch or switch stack. Some of the hardware limitation-Cisco IOS Release 12.2(58)SE1 and Later

Cisco IOS Limitations
(Catalyst 3750 switches and Cisco EtherSwitch service modules) Dynamic ARP inspection log entries might be lost after a switch failure. Any log entries that are still in the log buffer (have not been output as a system message) on a switch that fails are lost.
When you enter the show ip arp inspection log privileged EXEC command, the log entries from all switches in the stack are moved to the switch on which you entered the command.
There is no workaround. (CSCed95822)

Caveats Resolved in Cisco IOS Release 12.2(58)SE2
•CSCtq01926
When you configure a port to be in a dynamic VLAN by entering the switchport access vlan dynamic interface configuration command on it, the switch might reload when it processes ARP requests on the port.
The workaround is to configure static VLANs for these ports.
also good to be aware of
https://supportforums.cisco.com/discussion/12159141/last-word-ip-services-ios-ws-c3750-48ps-e

12.2(55)SE EoL/EoS only affects platforms (2960, 3560 and 3750) with flash memory of 32 mb and above.  Legacy 3560 and 3750 are NOT affected by announcement.   And even if you have a 3560 or 3750 with 32 mb flash, I would steadfastly recommend 12.2(55)SE train than 15.0(1)SE, 15.0(2)SE or 15.2(1)E.

If your  platform has 16 mb flash, then the "highest" IOS you can run is 12.2(55)SE.  My personal recommendation is 12.2(55)SE8 or you could try the newer 12.2(55)SE9.  I've got years of good experience with this particular version of IOS, 12.2(55)SE, and must admit this is one of the most stable version I've ever used (and I've used and tested a lot of versions).  It's sad that Cisco no longer wants to run with this code.
However the (58) has the below but working version is better to close your challenge first
Smart logging to capture and export packet flows to a NetFlow collector. This release supports smart logging for DHCP snooping or dynamic ARP inspection violations, IP source guard denied traffic, and ACL traffic permitted or denied on Layer 2 ports (Catalyst 3750 and 3560).
0
 
gheistCommented:
Either is EOL 31jan2014...
That switch comes with 64MB flash
0
 
Craig BeckCommented:
@gheist - I think they're deferred now anyway so you can't even download them from Cisco anymore.
0
 
MontyVAuthor Commented:
Honestly I have yet to find a solution for this.  They only thing that has helped was removing it entirely.  Not the best setup as we are still having issues and it seems to just trickle to other devices now as slowly other devices are experiencing the same problem.

I've tried various IOS version in both 12 and 15 varieties and I have had Cisco TAC on the phone more than a half dozen times with hours on the phone with no resolution.  They are stumped and the last thing I left with them was a pcap dump but that really didn't show much information as it seemed end point devices were not even making dhcp requests when turned on but when you remove snooping/binding then all of the sudden you see the requests again.

Right now I've been waiting for months for answers from Cisco and nothing so far.  So for now the solution was just to remove both snooping and arp.
0
 
btanExec ConsultantCommented:
thanks for sharing and do share if you have the TAC advice on  such symptoms. security measures should not break business running
0

Featured Post

 The Evil-ution of Network Security Threats

What are the hacks that forever changed the security industry? To answer that question, we created an exciting new eBook that takes you on a trip through hacking history. It explores the top hacks from the 80s to 2010s, why they mattered, and how the security industry responded.

  • 6
  • 4
  • 4
  • +1
Tackle projects and never again get stuck behind a technical roadblock.
Join Now