Random network crashes in domain...

It may be an issue with a failing switch flooding the network.  Do you have any type of network monitoring software.

Are all your network devices on one switch.

We struck an issue a while back with a D-Link switch which was teamed to a Cisco Switch, breaking the teaming fixed it along with playing around with Spanning Tree & IGMP Snooping.
On a seperate issue we also found that with a new HP C3000 (until we upgraded the IOS/Firmware of the main switch) that we had to remove a teamed port from one of the Blades to actually get things working much more efficiently. Similar issues did appear in both scenarios to what you stated but so randomly that by the time we looked into it - it had either corrected itself or only hit certain PC's on the network..

Maybe try checking that any Domain Controllers ports (teaming) is up to date & running fine as well as the switchgear having the latest firmware.

Hi ChiefIT - thanks for the quick response.  I imagine your talking about Windows Server 2003 SP1.  I have verified all are running Server 2003 SP2 and our SQL Server is running Server 2008.  If you meant the XP clients, I believe all are imaged to XP SP2 and upgraded to SP3.

PriceD - sounds interesting.  We are using 8 DLINK DGS-1224 WebSmart Switches.  The WebSmart switches provide a simple interface to view collisions, etc... but nothing too fancy.  All servers are connected to the first switch, and clients to the rest.  

As I was typing this, the issue just occured again - I will check the event viewer logs and report back what I find.

So far the logs look clean.  I do see on the PDC around the exact same time the error starts or clears itself (not sure which at this point) there are two events logged in the System Log.  This is an informational message:
Type: Information
Source: e1express
EventID: 42
Message: Intel(R) PRO/1000 PM Network Connection driver had been started.

The servers are multihomed, but one of the NIC's is disabled.

By the way, there are a total of 4 e1express events - 2 from the 7:30pm freeze and 2 from the 8:50pm freeze.

kiwistag - The DLINK DFL 800 has the most up-to-date firmware; however I believe I will have to upgrade the firmware on the switches. I will do that tonight/morning. I'm 99% certain that the network adapters are not teamed and one of them is disabled.

Cisco Switches and router??

We use a DLINK DFL 800 NetDefend and  8 DLINK DGS-1224 WebSmart Switches.

Disregard, I see you have D-link switches.

So, the duplex settings are probably not the issue.

On a client, type IPconfig /all, and let's see what problems we have with DNS.

What it sounds like is you have multiple connections BETWEEN switches, to your backbone switches.

You should have one connection per switch to your wiring backbone. Otherwise swtiching gets confused and knocks down switches.

Sorry for the delayed reply.  I have run an ipconfig /all from my laptop (Vista 32 Business) and this is what shows:

Windows IP Configuration

   Host Name . . . . . . . . . . . . : BRIAN-LAPTOP
   Primary Dns Suffix  . . . . . . . : LUCER.LUCERESEARCH.LOCAL
   Node Type . . . . . . . . . . . . : Broadcast
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : LUCER.LUCERESEARCH.LOCAL
                                       LUCER
                                       LUCERESEARCH.LOCAL

Wireless LAN adapter Wireless Network Connection:

   Connection-specific DNS Suffix  . : LUCER
   Description . . . . . . . . . . . : Intel(R) WiFi Link 5100 AGN
   Physical Address. . . . . . . . . : 00-22-FB-2C-09-82
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::a891:fde6:a668:b676%13(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.3.100(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Lease Obtained. . . . . . . . . . : Tuesday, September 15, 2009 09:38
   Lease Expires . . . . . . . . . . : Wednesday, September 16, 2009 09:37
   Default Gateway . . . . . . . . . : 192.168.3.1
   DHCP Server . . . . . . . . . . . : 192.168.3.1
   DNS Servers . . . . . . . . . . . : 204.130.255.3
                                       64.122.32.71
                                       192.168.3.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) 82567LM Gigabit Network Connecti
on
   Physical Address. . . . . . . . . : 00-21-70-EC-01-0E
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::1054:b9cb:6b6b:ae61%12(Preferred)
   IPv4 Address. . . . . . . . . . . : 172.16.1.70(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 172.16.1.1
   DNS Servers . . . . . . . . . . . : 172.16.1.2
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Bluetooth Network Connection:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Bluetooth Device (Personal Area Network)
   Physical Address. . . . . . . . . : 00-23-4D-EB-2D-1D
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Local Area Connection* 7:

   Connection-specific DNS Suffix  . : LUCER
   Description . . . . . . . . . . . : isatap.LUCER
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::5efe:192.168.3.100%23(Preferred)
   Default Gateway . . . . . . . . . :
   DNS Servers . . . . . . . . . . . : 204.130.255.3
                                       64.122.32.71
                                       192.168.3.1
   NetBIOS over Tcpip. . . . . . . . : Disabled

Tunnel adapter Local Area Connection* 11:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : isatap.{082354EA-5AC1-44D3-BAAA-91B4AEF37
006}
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Local Area Connection* 12:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #3
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Local Area Connection* 13:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 02-00-54-55-4E-01
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

CheifIT - It sounds like you may be on to something with the switches.  The consultant that set the network up in 2006 used the DLINK WebSmart switches and it looks like a wiring nightmare.  In fact, he even has our internet data circuits connected directly into the switches instead of the DLINK DFL 800 NetDefend/Firewall.

Here is a lame ascii drawing of what I mean.

[DMARC]===>[nternet T1's]===>[SWITCH1]===>[WAN Port 1 on DFL800]===>[LAN Port 1 on DFL800] ==>[SWITCH6]

OK, now I see two problems:

   Default Gateway . . . . . . . . . : 192.168.3.1<< GATEWAY/ROUTER
   DHCP Server . . . . . . . . . . . : 192.168.3.1<< GATEWAY PROVIDING DHCP
   DNS Servers . . . . . . . . . . . : 204.130.255.3<< OUTSIDE DNS SERVER
                                       64.122.32.71<< OUTSIDE DNS SERVER
                                       192.168.3.1<< GATEWAY AS A DNS SERVER

I also see problems with whom is supplying DHCP. 192.168.3.1 appears to be your gateway router. This means it is probably the router for your internet.

The problem with  non- Windows servers supplying DHCP is, it will also try to supply DNS. In doing so, it will also not store the DNS SRV records for your servers. SRV records are used for domain services, like AUTHENTICATION, and File replication.

Let me tell you what happens:
Your client will go to the router for DNS, It will see outside DNS servers, (as you can see in your IPconfig). So, it will go to outside DNS for internal DNS resolution. That means it will go to JOE ISP's DNS server in an attempt for you to contact YOUR DOMAIN SERVER. That doesn't work.

So, what you need to do is prevent your router from supplying DHCP. Then, make sure your DHCP server tells your clients to go to YOUR DNS server for DNS.

The inability for your DHCP clients to see your DNS server and Domain server will cause what appears to be intermittent communications between the client and servers.

So, here is a list of things for you to do.

THESE CAUSE PROBLEMS WITH COMMUNICATING WITH THE SERVER ON ALL DHCP CLIENTS:
-Configure your Windows servers to supply DHCP.
-Under DHCP scope options, configure YOUR DNS servers within the list of servers. Under forwarders you can put your Gateway router as a forwarder to DNS.
-Disable your router from supplying DHCP on the LAN side of the router, leaving the WAN Side alone.
-Ensure you are on SP2 for all 2003 servers.

THIS CAUSES UP/DOWN TIMES WITH THE ENTIRE NETWORK WHERE NETWORK CONNECTIONS WILL DISSAPEAR:
-Make sure your switches don't have TWO connections to your backbone switches. ONE connection only, switches are not capable of multicast like that.

Also, download DHCPloc.exe. Install it and run this program to find out if there are any other rogue DHCP servers. Yes, your router is considered a rogue DHCP server.


(ONE LAST THING)
This client appears to be multihomed:
This too causes problems unless configured right.

Get your domain's DHCP and switches in order. Then, I will help you with multihomed computers.

Hi ChiefIT - thanks again for the reply.  I will correct the settings on the wireless access point (Linksys WRT300N).

Keep in mind, this was an ipconfig /all from my laptop which has multiple interfaces (wifi, ethernet, bluetooth, etc...)

The network crashes were occuring when there were NO devices connected to the WiFi access point.  

All IP addresses on the network are static, except for those that connect to the wifi router (these are set by DHCP).  I can simply remove the Linksys WRT300N from the network to simplify troubleshooting if necessary.

The only devices that connect to the WiFi access point are 4 laptops and a handful of iPhones.

nevertheless, if you want domain access to domain services, they should seek your server for DHCP as well as DNS.

So, from what I am gathering, your WAP is supplying DHCP to WAP clients??

Then, your focus should be to get the wiring straight on your switches. We can revisit DHCP after that.

Exactly.  The switches are going to be another story.

The wiring is absolutely atrocious.  Looks like I'm going to have to pick a night to redo it.  Sadly the patch panels aren't even labelled. *sigh*

I will upgrade the firmware on the switches and then start with the MOST basic connection from the DMARC > T1's > WAN Port on DFL800 > LAN Port on DFL800 > Switch 1.  Looks like a lot of work ahead of me.  I will get this straightened out and post back.

Thanks again so far ChiefIT.

Ouch:
I hate networks like that, you could go down and purchase a LAN checker that you can check six lan ports at once.

You ... could... drop the ports back to 10Mbps Full or 100 Half Duplex and see how it goes too..

Please note I have generated another question based on an issue that came up during the re-wiring.

http://www.experts-exchange.com/Networking/Wireless/Q_24735203.html

ChiefIT - I rewired from the patch panel to the switch.

There is only one connection from the isp gateway to the firewall, and from the lan port on the firewall back to the switch.  The problem occured Sunday after the changes had been made.  Again the only error reported was on the PDC at the time of the crash:

Type: Information
Source: e1express
EventID: 42
Message: Intel(R) PRO/1000 PM Network Connection driver had been started.


Additionallly, there are 2 wireless routers that are connected to the switches.  The wifi routers are WAN addressed to our private network (172.16.1.X) and LAN addressed to their own subnet (192.168.3.X and 192.168.2.X).

So, what's the current topology?

Also what symptoms are we at now?

Its the same as before with the exception that the ISP gateway connection is now plugged into WAN port 1 on the firewall and LAN port 1 on the firewall is plugged directly into the 1st port of the 1st switch.  All servers are connected to ports 2-13 on the first switch.

Same symptoms - at a random time the entire network loses connectivity.  You can only ping the localhost, but no other server or address.  When the entire loss of connectivity occures, there are no events logged on the servers except for the PDC:

Type: Information
Source: e1express
EventID: 42
Message: Intel(R) PRO/1000 PM Network Connection driver had been started.

Exact same thing just happened again (Monday 4:15MST) on the network.  Entire loss of connectivity.

Only event showing in the log is on the PDC
Type: Information
Source: e1express
EventID: 42
Message: Intel(R) PRO/1000 PM Network Connection driver had been started.

For that reason, i think you should request moderator assistance and ask to add this to the networking zone and get assistance from there. The networking Experts will ask you for a network topology. :

The log in my firewall is littered with these Warnings:

Date - 2009-09-21 16:15:47
ipdatalen=342 udptotlen=342  

Severity - Warning

Category/ID - RULE6000051

Rule - Default_Rule

Proto - UDP

Src/DstIf - lan


Src/DstIP - 172.16.1.85 239.255.255.250

Src/DstPort - 34093 1900

Event/Action - ruleset_drop_packet drop

OK, Looks like a suspect.

Tell me something.

Does your entire network crash with all data, or just UDP data?

DNS uses TCP data. If you periodically don't get internet, I don't think this error pertains to your root problem. Instead, you probably have a router conflict between your WAN router and the Wireless routers.

All data and network functionality ceases.

Can ping localhost, but not gateway, switches, clients, etc...

UDP data is a connectionless, transport layer protocol. In other words, the sending machine will broadcast the data out, and not require a response from the distant end. This UDP problem is not your problem. UDP is generally not routeable. However, If your firewall were blocking TCP or IP as well as UDP, then I would say this is your problem.

Your issue is the network ceases at various times. In other words, you're having problems with TCP/IP as well as UDP.

That is a physical layer or data link layer problem, not a problem with UDP.  

A physical layer or data link layer problem means its either routing or Hardware conflict of the items that are responsible for routing, (like switches and routers).

This is why it would be best to get a networking engineer or Networking expert on EE to help you. I am best at domain administration.

If in your shoes, I would seek someone with a bit more experience in networking than me. So, press the request attention button and ask a moderator to put this in the networking.

OR create a new post in the networking zone, with routers.

If you choose that option. Let me know of this new post and I will show up there to work with the other experts and you on getting this resolved.

 

If you want to know about the LAYERS I was talking about. You can read this information after you have been helped:

http://en.wikipedia.org/wiki/OSI_model

Internal communications are good, but external communications are bad. This sounds more and more like a routing conflict.

Can you ping www.Google.com during the bad times?

Spanning is disabled by default (and currently) on all DGS1224 WebSmart switches.  Spanning is however enabled on the new DXS-3227 switch.

@LR Brian:

Just a little extra information on STP:

STP stands for "Spanning Tree/Portfast". If portfast is not enabled, you will default to spanning tree. Spanning tree can cause intermittent communications on the network. Portfast strips some routing informaiton that delays the packets from routing. Spanning tree takes about 45 seconds to discover routes to the nodes. That will time out XP machines or newer AND you will see intermittent network connectivity on all xp machines an newer, but not see this on 2000 machines and older.  

--A network topology will help your networking experts troubleshoot and fix this emensely.

@ikalmar: Thanks for joining us ikalmar. We need a networking tech with us.

Just a little info for you:
The wiring has been changed and the author has 2 wireless routers within the network. The wiring change was to prevent dual connections to the backbone switches of the network. The inability to multicast on these switches has caused intermittent network connectivity. From What I am see, it appears like we may have two routers that are conflicting and therefore the routes are confused. You have seen two routers mess each other up, haven't you???

Made some configuration changes as suggested.

DXS-3227 moved to the top of the stack, DGS 1224 moved to the bottom.  STP enabled on all switches, with priority of the middle switch set to 8092.

Will close question this Friday should the issue cease.

Issue occured again...  Here is what the switches look like:

Heres what it looks like

ISP Gateway ==> DFL800 Firewall WAN1

DFL800 Firewall LAN1
                      || (port 23 of 1st switch)
Switch 9 - 172.16.1.21 (DXS-3227)  = STP Enabled
                      ||
Switch 2 - 172.16.1.14 (DXS-1224) = STP Enabled
                      ||
Switch 3 - 172.16.1.15 (DXS-1224) = STP Enabled
                      ||
Switch 4 - 172.16.1.16 (DXS-1224) = STP Enabled
                      ||
Switch 5 - 172.16.1.17 (DXS-1224) = STP Enabled (Priority set to 8092)
                      ||
Switch 6 - 172.16.1.18 (DXS-1224) (Older FW, doesn't have ST option, STOP NOT enabled)
                      ||
Switch 7- 172.16.1.19 (DXS-1224) = STP Enabled
                      ||
Switch 8 - 172.16.1.20 (DXS-1224) = STP Enabled
                      ||
Switch 1 - 172.16.1.13 (DXS-1224) = STP Enabled

I have plans tonight to upgrade the firmware on all switches (should bring the .18 switch up with Spanning Tree).

Additiionally, I installed the clunky DLINK monitoring software that auto discovers the websmart switches said that .13, .14, .15, .16 were "Not alive" during the issue.