[Last Call] Learn about multicloud storage options and how to improve your company's cloud strategy. Register Now

x
?
Solved

DHCP suddenly stopped working on our network

Posted on 2010-08-12
13
Medium Priority
?
869 Views
Last Modified: 2012-08-13
Greetings,

I have an issue with DHCP at one of my work's locations.  DHCP has stopped working.  No one at that location can get DHCP addresses.  If I assign them a static address all is well, but DHCP will not get them an address.

My DHCP is served from my Cisco ASA 5510.  DHCP is running and I find no errors in the log.  A total of three requests have been denied in the 104 days the system has been online.  Monitoring this I can see that DHCPDISCOVERs are coming in and DHCPOFFERs are going out - however no DHCPREQUESTs are coming in and no DHCPACKs are going out.

Nothing change on our network today.  The only recent change was a week or so ago we added a fiber optic connection between two buildings at this site and someone mentioned spanning tree....

Could the spanning tree have anything to do with this?  If so, why didn't this happen back when it was implemented?  

Please help!!  Any assistance would be appreciated.

-Techtree
0
Comment
Question by:techtree
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 7
  • 6
13 Comments
 
LVL 24

Expert Comment

by:rfc1180
ID: 33426000
> Monitoring this I can see that
What are you using to monitor?

The fact that you can see the client's broadcasted messages on the physical subnet to discover available DHCP servers, offers from the servers , and normal IP connectivity is working due to the fact that static addressing works tells us that IP connectivity is working from a network perspective (Possible DHCP server or DHCP client issue). The question is what is happening to the DHCP Requests? Are you able to install wireshark on the hosts to capture packets (Are they even getting the offer from the DHCP server, if so, do you see the request leaving the host?).

Do you have a network diagram that you can provide?

Billy
0
 

Author Comment

by:techtree
ID: 33426174
Billy,

Thank you for your response!

I am using the ASDM GUI to monitor the DHCP activity on the ASA 5510.

The office is currently closed and don't have anyone on-site at the moment to try wireshark.  This is likely going to have to wait until tomorrow.

I can't provide a diagram at the moment but I will try to remedy this.

More info:
One thing that I believe key in this is that the issue (DHCP not functioning) began suddenly.  No one was working on the ASA at all today and DHCP has been working flawlessly for months.

The last time DHCP stopped working at that site was it stopped working in one building due to a wireless bridge we had between the buildings (before the fiber link was installed) was the cause (reset both endpoints and issue went away).

Other than that, have not had any DHCP issues whatsoever at any of our sites in over 5 years...

-Techtree
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33426209
is this dhcp server have other scopes as well and is functioning just fine?

Billy
0
Technology Partners: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 

Author Comment

by:techtree
ID: 33431300
No.  Only the one scope.

-Techtree
0
 

Author Comment

by:techtree
ID: 33431340
This morning we rebooted that ASA to see if that would help.

Still same issue after reboot - however the reboot reset the statistics which has made them easier to see what's going on:

DHCPDISCOVER:  3697
DHCPREQUEST:  2
DHCPOFFER:  3697
DHCPNAK:  2

Total messages received by DHCP server:  3699
Total messages sent by DHCP server:  3699

Don't know if that helps figure out what might be happening but thought I would provide it just in case.

-Techtree
0
 
LVL 24

Accepted Solution

by:
rfc1180 earned 2000 total points
ID: 33431362
I would focus on the DHCP server, see if you can get some packet captures before any intrusive testing; if they tests are inconclusive, then maybe try and disable the dhcp server and re-enable it. If that does not work, disconnect the inside port of the ASA and connect a laptop directly to the port with a cross-over and the DHCP again, if that fails then a reboot of the ASA is possibly needed.

Billy
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33431366
>Don't know if that helps figure out what might be happening but thought I would provide it just in case.
yeah, too late on my last reply :(
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33431382
>3697
is that after the reboot?

How many hosts are in the broadcast domain? Do you have any dhcp relay helpers?
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33431424
are there any firewalls in the path from the ASA to the clients that could be blocking UDP port 68?

Either the clients are not getting the DHCPOFFER or all clients are not sending the DHCPREQUEST.

it is very apparent that the discovers and offers match, so it appears the DHCP server is working as expected.

Billy
0
 

Author Comment

by:techtree
ID: 33433531
Sorry for lack of replies - been  troubleshooting.

Got on the phone with Cisco - got a remote tech support session where they WebEx'd to my box and SSH'd to my ASA.

They discovered that only some BOOTP packets related to DHCP were coming through (with relation to our Linux clients).  The ASA does not support BOOTP for DHCP.  I suspect that Linux is pushing both BOOTP and regular DHCP but only the BOOTP is making it through.

To test I plugged a laptop directly into the LAN port on the ASA and I got a DHCP address immediately with no issue (tried on both Windows and Linux).

So I rebooted the main LAN switch that the ASA was plugged in to (and where all of our wall jacks trace to) and DHCP started working again and has now been working for about an hour.

It seems that, for some reason, the switch was blocking all DHCP traffic (both from clients and from the ASA).  Very basic config - out of the box, hostname, IP, management, and using GBIC for fiber link to other building.  No STP, QoS, or anything fancy.

Switch is NetGear GS748TR and I now have a tech support case open with NetGear support.

They told me to update the firmware on the switch and get back to them (I am behind by three versions according to the tech).

So we'll see....but for now DHCP is working again after having rebooted that main LAN NetGear switch.

Thank you for your responses and help!  I appreciate it!

Any further ideas or insight taking the new information into account would be appreciated as well.

-Techtree
0
 

Author Closing Comment

by:techtree
ID: 33433548
The given advice led directly to me discovering the answer.
0
 

Author Comment

by:techtree
ID: 33433587
The post from Billy that I marked as the accepted solution is what led me directly to be able to determine the cause of the issue.

After I got nowhere with Cisco (a rarity as they are typically very good), I followed Billy's troubleshooting steps in that post and discovered the DHCP from the ASA was working fine (I had plugged a laptop directly into the LAN port on the ASA and got an address immediately).  This ruled out the ASA which led me directly to the switch.  It was the switch the ASA and my LAN was connected to that was blocking the request - rebooting the switch fixed the issue.

Thanks, Billy!

-Techtree
0
 
LVL 24

Expert Comment

by:rfc1180
ID: 33433619
Glad to hear that you got it resolved.

Billy
0

Featured Post

[Webinar] Lessons on Recovering from Petya

Skyport is working hard to help customers recover from recent attacks, like the Petya worm. This work has brought to light some important lessons. New malware attacks like this can take down your entire environment. Learn from others mistakes on how to prevent Petya like worms.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Arrow Electronics was searching for a KVM  (Keyboard/Video/Mouse) switch that could display on one single monitor the current status of all units being tested on the rack.
If you’re involved with your company’s wide area network (WAN), you’ve probably heard about SD-WANs. They’re the “boy wonder” of networking, ostensibly allowing companies to replace expensive MPLS lines with low-cost Internet access. But, are they …
Internet Business Fax to Email Made Easy - With  eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, f…
In this brief tutorial Pawel from AdRem Software explains how you can quickly find out which services are running on your network, or what are the IP addresses of servers responsible for each service. Software used is freeware NetCrunch Tools (https…
Suggested Courses

656 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question