Link to home
Start Free TrialLog in
Avatar of ExchangeAdminWVUH
ExchangeAdminWVUH

asked on

I have DHCP corruption

I have a DHCP server that has many scopes.  I have scopes that start 172.17.X.X and the last scope in that range has DHCP/BOOTP leases with a MAC address of 3137322e31372e.  If I delete all of the bogus leases and then reconcile the scope, the bogus leases come back.  I can delete them again and reconcile again but the bogus addresses come back.  Eventually they fill up the scope and there are no IP addresses left to be given out.  This only happens to one scope and it is the last one in the 172.17 range.  Any ideas?  We recently moved the dhcp service to a new server but this issue was happening on the old server as well.
Avatar of jonavogt
jonavogt
Flag of Germany image

Is it just the one MAC Adress? If so you might consider giving it a fixed lease so it can't fill the scope up.
A workstation that's configured to PXE boot will request an IP address from the DHCP server and not release it.  If the workstation fails to boot with its PXE configuration, it will typically reboot after a time-out period and try again, consuming another IP address since the lease hasn't run out on the first. This can go on and on until all the addresses have been leased.

First, try to track down the device on your network that has the MAC address you specified.  Your switches and routers should have the MAC address in their ARP tables.

Also, you can make sure the DHCP server is truly receiving requests from that device by tracing the packets in and out of port 67 on the DHCP server.  Chances are that the DHCP server is operating properly.

Without tracking down the device, a decent work-around would be to configure the DHCP server to hand out the same IP address to the device with the stated MAC address.
Avatar of ExchangeAdminWVUH
ExchangeAdminWVUH

ASKER

The MAC address is fake as far as I can tell.  There is no deive on our network with that mac address.  The MAC address appears over and over in the DHCP list but the last few digits of it change.  

3137322e31372e3232352e32303300
3137322e31372e3232352e32303400
3137322e31372e3232352e32303500
3137322e31372e3232352e32303600

I can reconcile the scope after deleting the list but they come right back.  I kept it clean all day yesterday but last night it started again and filled up the scope.
DHCP-issue.bmp
Notice that the valid assignments are DHCP only and the "bogus" addresses are DHCP/BOOTP.  The MAC addresses almost look like IPv6 addresses, but they're 1 byte short.

Two more things to try: First, you're probably not using IPv6 on your network in general, so make sure the server is configured to have it disabled.  Set the registry key HKLM\ SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters\DisabledComponent to 0xFF, and reboot the server so the change can go into effect.

Second, set the lease time for BOOTP devices on that scope to be very short - 60 seconds.  Do this in the advanced tab by setting the "Lease duration for BOOTP clients".  This setting should not affect the default least time given to DHCP clients.  See http://technet.microsoft.com/en-us/library/dd183637(WS.10).aspx for more info.
No luck.  We do not have IPv6 setup on the server.  I changed the lease times as suggested but it did not help.  When I delete the leases everything looks fine.  When I reconcile the scope, they show up again.  So I can delete them all day long but one the scope reconciles, they are coming back again.
What's the server version and service pack level?   Have you tried deleting and re-creating the scope?  

Make a backup copy of the dhcp database first.  Open a command prompt, make a directory called c:\dchpBackup, and run 'netsh dhcp server backup "c:\dhcpBackup"'  Now your DHCP service can be fully recovered with 'netsh dhcp server restore "c:\dhcpBackup"'.  

Next, export the configuration of just the faulty scope with 'netsh dhcp server export c:\dhcpBackup\scope172-17-225 172.17.225.0'.  (Could you post this output file for review?)  This provides a way to restore just the scope if deleting it and re-creating it fails. If re-creating it fails, delete the scope again, and then import this file with 'netsh dhcp server import c:\dhcpBackup\scope172-17-225 172.17.225.0'.  This will bring the scope back to the way it was.

The server is Windows 2003 Server Standard x64 and it is SP2.  We use the export every night and export a copy of the dhcp server.  We even moved this to a new server but the issue still remains.  I have deleted the scope and added it but no luck.  We reconcile our scopes every night at 12:30 am and that after that process we see the addresses listed in the server.  Every time I reconcile the scope, the issue starts.
Given that the problem has followed the DHCP server to the new machine, there really could be devices on the network requesting those IP addresses?  I'd install a copy of Wireshark on the DHCP server and capture all the packets going in and out of port 67 on your primary network interface.  Peruse the DHCP REQUEST packets to see which machines are making requests.  Look at the DHCP OFFER packets to see what devices are being offered which IPs.  Most importantly, look at the packet contents to see what's going on.  

When you have a good sense of what's going on by looking at the packets, perform another trace around your delete/reconcile/review procedure at 12:30 am.  I.e. start a packet capture of port 67, and while it's running delete the "bogus" assignments, reconcile the scopes, see that the leases came back, then stop the capture.  Of the leases that came back, note some of the IPs that were assigned, and look for them in the packet capture.

If you're not up for this, then the only other thing I can suggest is setting up a 2nd DHCP server, and re-create the troubled scope from scratch so it is free of any inheritance of data in the original DHCP server database.  Disconnect it from the network while you configure it.  When it's ready, disable the scope on the production server, and re-connect the "test" DHCP server so that it handles the scope.  See if you have the same problem; if so, it's definitely due to devices on your network.  Review Microsoft's split-scope DHCP server design page, http://technet.microsoft.com/en-us/library/dd296651(WS.10).aspx to be clear about how to cleanly separate the responsibilities of multiple DHCP servers on your network.  
I used wireshark to capture the packet information going to our dhcp server.  attached is one of the packets from the capture.  I am not real good at reading the wireshark files but i found was able to find the ports on our network that were initiating the packets.  It looks like something is sending a dhcp discover packet to our server.  The server is sending the dhcp offer packet to the device but the dhcp ack packet is never sent back to the server.  So the ip address in the dhcp offer packet is sitting in "limbo" waiting for the ack packet.  Once we reconcile the scope, the ip address in the dhcp offer packet is then added to the address leases witht he bogus mac address.  When we delete the leases, the process starts over again.  At least this is how I think it is happening.  Any idea what is causing this process?
dhcp.txt
ASKER CERTIFIED SOLUTION
Avatar of timhl
timhl
Flag of United States of America image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Thank you very much.  This was exactly what was happening.  We found the server that was causing the issues and shut them off.