Erratic network behavior - Computers and servers losing internet connectivity

Hi everyone,
We’re having a problem with a client’s network and we’d appreciate any advice or help you can provide. Please see the below details.

--- Their network: Domain.local
--Physical servers--
Name: VH-01.
Hyper-V host: Cisco R210-2121605W
Operating System: Server 2008 R2 SP1.
Specifications:
RAM: 24GB
CPU: Intel Xeon  X5650 2.67GHz Model 44 Stepping 2
Network Cards: 2 x Broadcom BCM5709C NetXtreme II GigE, 2 x Intel 82576 Gigabit Dual Port.

--Virtual Servers--
Name: VS-01
Operating System: Windows Server SBS 2011
Specifications:
RAM: 14GB
CPU: 4 Virtual Processors.
Role: SBS 2011, DNS, DHCP, Active Directory, Exchange, SQL Database.

Name: VS-02
Operating System: Server 2008 R2 SP1.
Specifications:
RAM: 4GB
CPU: 2  Virtual Processors.
Role: Terminal Server

Name: VS-03
Operating System: Server 2008 R2 SP1.
Specifications:
RAM: 4GB
CPU: 4 Virtual Processors.
Role: Trend Micro Server (Worry-Free business Security Advanced 8.0 SP1)

--Computers--
Windows 8.1: 2
Windows 7: 5
Windows XP: 3

Router: Cisco 877 SMB Router.
Switch: Linksys 24Pt Gigabit switch.

--- Issue:
Erratically, random computers will lose the ability to browse the internet. During this period of time they can still access local resources on the server and ping the gateway router. RDP access would not work though. Strangely the affected computers cannot be pinged by the router. Other devices can ping the affected computer.
Both Windows 8.1 and Windows 7 computers have had the issue. None of the Windows XP computers. Server: VS-02 and VS-03 have experienced symptoms but not VS-01 and VH-01.

On one of the computers we have discovered the following error in the Event Log:

== The system failed to register host (A or AAAA) resource records (RRs) for network adapter with settings:
==== Adapter Name : {67018E35-2310-4D71-BACF-13747FD76F41}
==== Host Name : D-07
==== Primary Domain Suffix : Domain.local
==== DNS server list :
===== 192.168.1.25, 8.8.8.8
==== Sent update to server : <?>
==== IP Address(es) :
====  192.168.1.112
== The reason the system could not register these RRs was because the DNS server contacted refused the update request. The reasons for this might be (a) you are not allowed to update the specified DNS domain name, or (b) because the DNS server authoritative for this name does not support the DNS dynamic update protocol.
== To register the DNS host (A or AAAA) resource records using the specific DNS domain name and IP addresses for this adapter, contact your DNS server or network systems administrator.

Users can fix the issue by restarting the computer or by unplugging the Network cable and plugging it back in.


--- Diagnosis so far:
When the issue first occurred, it was thought to be only affecting one computer, a windows 8.1 device. The resolutions attempted were the following:
We accessed event log remotely and located the error found above. Through research we tried to fix the ‘A’ record permissions for the computer in DNS Manager on VS-01 by giving the user full control. We made sure the records weren’t stale and ensured the scavenging was setup correctly. We cleared the DNS cache and updated the files. We then enabled secure and non-secure dynamic updates in DNS Manager. We also restarted the DNS and DHCP services and flushed and registered the DNS on the computer.
None of the above had any effect.

Via PSEXEC we created a firewall rule on the computer to allow port 80. Doing so allowed a VNC connection. Problem appeared solved, or so we thought.

It was revealed that other users had the problem and hadn’t thought to mention it to anyone.
All windows 7 and 8 computers were suffering from the issue. Strangely Windows XP was spared.

To fix the problem this time we approached Group Policy. We created a GPO to set a firewall rule to allow Port 80. We were impatient so we manually created this rule as well. This didn’t resolve the problem this time though.

It was realized that certain update services GPOs and some specific client GPOs missing. A bit of a mystery there. We imported spares from another SBS 2011 server and fixed them.

We set up a secondary DNS (it previously only had the SBS server) in DHCP for Google’s DNS: 8.8.8.8.

We also ensured every computer had it set in the network adapter properties to register with DNS.

When VS-02 was affected this enabled us an opportunity to diagnose by logging onto the server using the console connection via the Hyper-V host. We were able to confirm that VS-02 could ping the router and the SBS server. The router could not ping the server though.
We setup another spare router and swapped them to test. The issue was still occurring.
We swapped the switch and the issue still occurred.

A red herring we encountered was the discovery that a drive in the RAID on VH-01 was failing. We replaced the drive in hopes that the failing RAID (and thus affected performance) had been the cause of the strange occurrences. No dice.

We’ve also disabled the Anti-Virus and turned off VS-02 and VS-03 to no avail.

I think that covers everything so far.
Currently we’re at a loss. Maybe we’ve missed something obvious or maybe we should burn the building down. Any help here would be much appreciated.
LVL 2
tech_tonicAsked:
Who is Participating?
 
tech_tonicConnect With a Mentor Author Commented:
ARP poisoning was occurring in the network due to a user plugging a router into the network to use as a switch.
0
 
Carol ChisholmCommented:
Spurious reverse DNS entries?
A non-existent DNS server somewhere in AD?
A non-existent DNS server in the DHCP settings?
IPv6?
0
 
tech_tonicAuthor Commented:
Hi Wizard,

Thanks for the reply. I checked reverse DNS and actually noticed a reference to the old server we migrated from, it was set as a name server, but the IP address was unknown.

After removing this record I decided to go through all of my DNS and DHCP records and discovered a lot of records which were pointing back to the old server we migrated from. Additionally I opened ADSI edit and removed any other references in there. Active Directory appeared clean.

I swear I went through DNS a thousand times, but I never actually noticed any of these. Thanks for your help, I'm really hoping this is the cause of the issue.
0
Improve Your Query Performance Tuning

In this FREE six-day email course, you'll learn from Janis Griffin, Database Performance Evangelist. She'll teach 12 steps that you can use to optimize your queries as much as possible and see measurable results in your work. Get started today!

 
Carol ChisholmCommented:
Make sure your domain controllers are not referencing the old server, as I think they can put it back in the DNS. Check also in your DNS settings that the old server is not a name server for any of your DNS zones.
0
 
tech_tonicAuthor Commented:
Hi Carol,

The domain controller is not refreshing the entries back into DNS.

I believe I may have fixed the virtual servers from losing connection, however it is still happening to users and they either have to restart their computers or renew their IP address to regain internet connection:

The server has an Intel Dual Port NIC, Dual port Broadcom NIC and a seperate management port.

It is currently only using the Intel Dual port NIC and the management port.

Currently one of the Intel adapters is used completely for the virtual host whilst the other is set as the Virtual Switch. The "Allow management operating system to share this network adapter" is ticked on this as well.

I made a change and set an IP address statically on the virtual switch as it was getting it via DHCP.
0
 
Carol ChisholmCommented:
Have you checked the permissions? Look in the event log.
0
 
Jeffrey Kane - TechSoEasyPrincipal ConsultantCommented:
We set up a secondary DNS (it previously only had the SBS server) in DHCP for Google’s DNS: 8.8.8.8.

Why did you do this?  You should ONLY have the SBS's IP address listed for the DNS Server for all machines on your network.

The reason you are getting the error is that the Google DNS Server won't allow updates from your local LAN IP's.

Please remove the 8.8.8.8 entries (both from statically set and DHCP config), refresh both SBS and a workstation and then if you are still having trouble, please post a COMPLETE ipconfig /all from both the SBS and a workstation.
0
 
tech_tonicAuthor Commented:
My own solution.
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.