Solved

DHCP suddenly stopped working on our network

Posted on 2010-08-12
13
842 Views
Last Modified: 2012-08-13
Greetings,

I have an issue with DHCP at one of my work's locations.  DHCP has stopped working.  No one at that location can get DHCP addresses.  If I assign them a static address all is well, but DHCP will not get them an address.

My DHCP is served from my Cisco ASA 5510.  DHCP is running and I find no errors in the log.  A total of three requests have been denied in the 104 days the system has been online.  Monitoring this I can see that DHCPDISCOVERs are coming in and DHCPOFFERs are going out - however no DHCPREQUESTs are coming in and no DHCPACKs are going out.

Nothing change on our network today.  The only recent change was a week or so ago we added a fiber optic connection between two buildings at this site and someone mentioned spanning tree....

Could the spanning tree have anything to do with this?  If so, why didn't this happen back when it was implemented?  

Please help!!  Any assistance would be appreciated.

-Techtree
0
Comment
Question by:techtree
  • 7
  • 6
13 Comments
 
LVL 24

Expert Comment

by:rfc1180
ID: 33426000
> Monitoring this I can see that
What are you using to monitor?

The fact that you can see the client's broadcasted messages on the physical subnet to discover available DHCP servers, offers from the servers , and normal IP connectivity is working due to the fact that static addressing works tells us that IP connectivity is working from a network perspective (Possible DHCP server or DHCP client issue). The question is what is happening to the DHCP Requests? Are you able to install wireshark on the hosts to capture packets (Are they even getting the offer from the DHCP server, if so, do you see the request leaving the host?).

Do you have a network diagram that you can provide?

Billy
0
 

Author Comment

by:techtree
ID: 33426174
Billy,

Thank you for your response!

I am using the ASDM GUI to monitor the DHCP activity on the ASA 5510.

The office is currently closed and don't have anyone on-site at the moment to try wireshark.  This is likely going to have to wait until tomorrow.

I can't provide a diagram at the moment but I will try to remedy this.

More info:
One thing that I believe key in this is that the issue (DHCP not functioning) began suddenly.  No one was working on the ASA at all today and DHCP has been working flawlessly for months.

The last time DHCP stopped working at that site was it stopped working in one building due to a wireless bridge we had between the buildings (before the fiber link was installed) was the cause (reset both endpoints and issue went away).

Other than that, have not had any DHCP issues whatsoever at any of our sites in over 5 years...

-Techtree
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33426209
is this dhcp server have other scopes as well and is functioning just fine?

Billy
0
 

Author Comment

by:techtree
ID: 33431300
No.  Only the one scope.

-Techtree
0
 

Author Comment

by:techtree
ID: 33431340
This morning we rebooted that ASA to see if that would help.

Still same issue after reboot - however the reboot reset the statistics which has made them easier to see what's going on:

DHCPDISCOVER:  3697
DHCPREQUEST:  2
DHCPOFFER:  3697
DHCPNAK:  2

Total messages received by DHCP server:  3699
Total messages sent by DHCP server:  3699

Don't know if that helps figure out what might be happening but thought I would provide it just in case.

-Techtree
0
 
LVL 24

Accepted Solution

by:
rfc1180 earned 500 total points
ID: 33431362
I would focus on the DHCP server, see if you can get some packet captures before any intrusive testing; if they tests are inconclusive, then maybe try and disable the dhcp server and re-enable it. If that does not work, disconnect the inside port of the ASA and connect a laptop directly to the port with a cross-over and the DHCP again, if that fails then a reboot of the ASA is possibly needed.

Billy
0
Do You Know the 4 Main Threat Actor Types?

Do you know the main threat actor types? Most attackers fall into one of four categories, each with their own favored tactics, techniques, and procedures.

 
LVL 24

Expert Comment

by:rfc1180
ID: 33431366
>Don't know if that helps figure out what might be happening but thought I would provide it just in case.
yeah, too late on my last reply :(
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33431382
>3697
is that after the reboot?

How many hosts are in the broadcast domain? Do you have any dhcp relay helpers?
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33431424
are there any firewalls in the path from the ASA to the clients that could be blocking UDP port 68?

Either the clients are not getting the DHCPOFFER or all clients are not sending the DHCPREQUEST.

it is very apparent that the discovers and offers match, so it appears the DHCP server is working as expected.

Billy
0
 

Author Comment

by:techtree
ID: 33433531
Sorry for lack of replies - been  troubleshooting.

Got on the phone with Cisco - got a remote tech support session where they WebEx'd to my box and SSH'd to my ASA.

They discovered that only some BOOTP packets related to DHCP were coming through (with relation to our Linux clients).  The ASA does not support BOOTP for DHCP.  I suspect that Linux is pushing both BOOTP and regular DHCP but only the BOOTP is making it through.

To test I plugged a laptop directly into the LAN port on the ASA and I got a DHCP address immediately with no issue (tried on both Windows and Linux).

So I rebooted the main LAN switch that the ASA was plugged in to (and where all of our wall jacks trace to) and DHCP started working again and has now been working for about an hour.

It seems that, for some reason, the switch was blocking all DHCP traffic (both from clients and from the ASA).  Very basic config - out of the box, hostname, IP, management, and using GBIC for fiber link to other building.  No STP, QoS, or anything fancy.

Switch is NetGear GS748TR and I now have a tech support case open with NetGear support.

They told me to update the firmware on the switch and get back to them (I am behind by three versions according to the tech).

So we'll see....but for now DHCP is working again after having rebooted that main LAN NetGear switch.

Thank you for your responses and help!  I appreciate it!

Any further ideas or insight taking the new information into account would be appreciated as well.

-Techtree
0
 

Author Closing Comment

by:techtree
ID: 33433548
The given advice led directly to me discovering the answer.
0
 

Author Comment

by:techtree
ID: 33433587
The post from Billy that I marked as the accepted solution is what led me directly to be able to determine the cause of the issue.

After I got nowhere with Cisco (a rarity as they are typically very good), I followed Billy's troubleshooting steps in that post and discovered the DHCP from the ASA was working fine (I had plugged a laptop directly into the LAN port on the ASA and got an address immediately).  This ruled out the ASA which led me directly to the switch.  It was the switch the ASA and my LAN was connected to that was blocking the request - rebooting the switch fixed the issue.

Thanks, Billy!

-Techtree
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33433619
Glad to hear that you got it resolved.

Billy
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

If your business is like most, chances are you still need to maintain a fax infrastructure for your staff. It’s hard to believe that a communication technology that was thriving in the mid-80s could still be an essential part of your team’s modern I…
When it comes to security, there are always trade-offs between security and convenience/ease of administration. This article examines some of the main pros and cons of using key authentication vs password authentication for hosting an SFTP server.
Viewers will learn how to connect to a wireless network using the network security key. They will also learn how to access the IP address and DNS server for connections that must be done manually. After setting up a router, find the network security…
After creating this article (http://www.experts-exchange.com/articles/23699/Setup-Mikrotik-routers-with-OSPF.html), I decided to make a video (no audio) to show you how to configure the routers and run some trace routes and pings between the 7 sites…

746 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

13 Experts available now in Live!

Get 1:1 Help Now