Solved

I have DHCP corruption

Posted on 2009-06-29
11
3,121 Views
1 Endorsement
Last Modified: 2012-08-14
I have a DHCP server that has many scopes.  I have scopes that start 172.17.X.X and the last scope in that range has DHCP/BOOTP leases with a MAC address of 3137322e31372e.  If I delete all of the bogus leases and then reconcile the scope, the bogus leases come back.  I can delete them again and reconcile again but the bogus addresses come back.  Eventually they fill up the scope and there are no IP addresses left to be given out.  This only happens to one scope and it is the last one in the 172.17 range.  Any ideas?  We recently moved the dhcp service to a new server but this issue was happening on the old server as well.
1
Comment
Question by:ExchangeAdminWVUH
  • 5
  • 5
11 Comments
 

Expert Comment

by:jonavogt
ID: 24746607
Is it just the one MAC Adress? If so you might consider giving it a fixed lease so it can't fill the scope up.
0
 
LVL 2

Expert Comment

by:timhl
ID: 24746895
A workstation that's configured to PXE boot will request an IP address from the DHCP server and not release it.  If the workstation fails to boot with its PXE configuration, it will typically reboot after a time-out period and try again, consuming another IP address since the lease hasn't run out on the first. This can go on and on until all the addresses have been leased.

First, try to track down the device on your network that has the MAC address you specified.  Your switches and routers should have the MAC address in their ARP tables.

Also, you can make sure the DHCP server is truly receiving requests from that device by tracing the packets in and out of port 67 on the DHCP server.  Chances are that the DHCP server is operating properly.

Without tracking down the device, a decent work-around would be to configure the DHCP server to hand out the same IP address to the device with the stated MAC address.
0
 

Author Comment

by:ExchangeAdminWVUH
ID: 24756203
The MAC address is fake as far as I can tell.  There is no deive on our network with that mac address.  The MAC address appears over and over in the DHCP list but the last few digits of it change.  

3137322e31372e3232352e32303300
3137322e31372e3232352e32303400
3137322e31372e3232352e32303500
3137322e31372e3232352e32303600

I can reconcile the scope after deleting the list but they come right back.  I kept it clean all day yesterday but last night it started again and filled up the scope.
DHCP-issue.bmp
0
 
LVL 2

Expert Comment

by:timhl
ID: 24760573
Notice that the valid assignments are DHCP only and the "bogus" addresses are DHCP/BOOTP.  The MAC addresses almost look like IPv6 addresses, but they're 1 byte short.

Two more things to try: First, you're probably not using IPv6 on your network in general, so make sure the server is configured to have it disabled.  Set the registry key HKLM\ SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\DisabledComponent to 0xFF, and reboot the server so the change can go into effect.

Second, set the lease time for BOOTP devices on that scope to be very short - 60 seconds.  Do this in the advanced tab by setting the "Lease duration for BOOTP clients".  This setting should not affect the default least time given to DHCP clients.  See http://technet.microsoft.com/en-us/library/dd183637(WS.10).aspx for more info.
0
 

Author Comment

by:ExchangeAdminWVUH
ID: 24764885
No luck.  We do not have IPv6 setup on the server.  I changed the lease times as suggested but it did not help.  When I delete the leases everything looks fine.  When I reconcile the scope, they show up again.  So I can delete them all day long but one the scope reconciles, they are coming back again.
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 2

Expert Comment

by:timhl
ID: 24765963
What's the server version and service pack level?   Have you tried deleting and re-creating the scope?  

Make a backup copy of the dhcp database first.  Open a command prompt, make a directory called c:\dchpBackup, and run 'netsh dhcp server backup "c:\dhcpBackup"'  Now your DHCP service can be fully recovered with 'netsh dhcp server restore "c:\dhcpBackup"'.  

Next, export the configuration of just the faulty scope with 'netsh dhcp server export c:\dhcpBackup\scope172-17-225 172.17.225.0'.  (Could you post this output file for review?)  This provides a way to restore just the scope if deleting it and re-creating it fails. If re-creating it fails, delete the scope again, and then import this file with 'netsh dhcp server import c:\dhcpBackup\scope172-17-225 172.17.225.0'.  This will bring the scope back to the way it was.

0
 

Author Comment

by:ExchangeAdminWVUH
ID: 24784598
The server is Windows 2003 Server Standard x64 and it is SP2.  We use the export every night and export a copy of the dhcp server.  We even moved this to a new server but the issue still remains.  I have deleted the scope and added it but no luck.  We reconcile our scopes every night at 12:30 am and that after that process we see the addresses listed in the server.  Every time I reconcile the scope, the issue starts.
0
 
LVL 2

Expert Comment

by:timhl
ID: 24789816
Given that the problem has followed the DHCP server to the new machine, there really could be devices on the network requesting those IP addresses?  I'd install a copy of Wireshark on the DHCP server and capture all the packets going in and out of port 67 on your primary network interface.  Peruse the DHCP REQUEST packets to see which machines are making requests.  Look at the DHCP OFFER packets to see what devices are being offered which IPs.  Most importantly, look at the packet contents to see what's going on.  

When you have a good sense of what's going on by looking at the packets, perform another trace around your delete/reconcile/review procedure at 12:30 am.  I.e. start a packet capture of port 67, and while it's running delete the "bogus" assignments, reconcile the scopes, see that the leases came back, then stop the capture.  Of the leases that came back, note some of the IPs that were assigned, and look for them in the packet capture.

If you're not up for this, then the only other thing I can suggest is setting up a 2nd DHCP server, and re-create the troubled scope from scratch so it is free of any inheritance of data in the original DHCP server database.  Disconnect it from the network while you configure it.  When it's ready, disable the scope on the production server, and re-connect the "test" DHCP server so that it handles the scope.  See if you have the same problem; if so, it's definitely due to devices on your network.  Review Microsoft's split-scope DHCP server design page, http://technet.microsoft.com/en-us/library/dd296651(WS.10).aspx to be clear about how to cleanly separate the responsibilities of multiple DHCP servers on your network.  
0
 

Author Comment

by:ExchangeAdminWVUH
ID: 24863356
I used wireshark to capture the packet information going to our dhcp server.  attached is one of the packets from the capture.  I am not real good at reading the wireshark files but i found was able to find the ports on our network that were initiating the packets.  It looks like something is sending a dhcp discover packet to our server.  The server is sending the dhcp offer packet to the device but the dhcp ack packet is never sent back to the server.  So the ip address in the dhcp offer packet is sitting in "limbo" waiting for the ack packet.  Once we reconcile the scope, the ip address in the dhcp offer packet is then added to the address leases witht he bogus mac address.  When we delete the leases, the process starts over again.  At least this is how I think it is happening.  Any idea what is causing this process?
dhcp.txt
0
 
LVL 2

Accepted Solution

by:
timhl earned 500 total points
ID: 24872606
Good work! Now you have some valuable information to work with.  The first piece of info you have is the MAC address of the device.  From the packet you provided, you see "Client MAC address: 00:23:7d:e3:65:54".  The left-most 6 digits, 00237d, tell you the vendor is Hewlett Packard.  But more importantly, with the MAC address you can search the ARP caches of your switches to find the port it's connected to, and you can find the device.

Secondly, in the packet, Option: (t=60,l=32) Vendor class identifier = "PXEClient:Arch:00000:UNDI:002001", means that the device is trying to "PXE Boot", or boot from the network.  Network Boot is build into the NIC BIOS and can be disabled in the PC's BIOS by removing it as one of the startup options. PXE boot uses "bootp" protocol, a subset of DHCP which doesn't include the concept of leasing. IP addresses assigned with bootp don't expire, and that's why eventually it consumes all the IPs in the scope.  You didn't provide any other DHCPREQUEST packets, but you could check them for other MAC addresses.  It could be just 1 PC that is cycling through the network boot process, failing each time, but consuming another IP each time.

Best course of action now is to find the device by searching the switches for its MAC address, and adjust its BIOS settings to prevent it from PXE booting.

0
 

Author Closing Comment

by:ExchangeAdminWVUH
ID: 31598198
Thank you very much.  This was exactly what was happening.  We found the server that was causing the issues and shut them off.
0

Featured Post

Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Ever wondered why you had to use DHCP options (dhcp opt 60, 66 or 67) in order to use PXE? Well, you don't!
A Cisco router can be configured as a DHCP Server. There are advantages and disadvantages in making your Cisco router work as DHCP Server. Almost all the features for windows DHCP can be configured on Cisco-based DHCP server. Some of the features me…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

758 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now