Solved

Erratic network behavior - Computers and servers losing internet connectivity

Posted on 2014-04-03
8
204 Views
Last Modified: 2015-04-27
Hi everyone,
We’re having a problem with a client’s network and we’d appreciate any advice or help you can provide. Please see the below details.

--- Their network: Domain.local
--Physical servers--
Name: VH-01.
Hyper-V host: Cisco R210-2121605W
Operating System: Server 2008 R2 SP1.
Specifications:
RAM: 24GB
CPU: Intel Xeon  X5650 2.67GHz Model 44 Stepping 2
Network Cards: 2 x Broadcom BCM5709C NetXtreme II GigE, 2 x Intel 82576 Gigabit Dual Port.

--Virtual Servers--
Name: VS-01
Operating System: Windows Server SBS 2011
Specifications:
RAM: 14GB
CPU: 4 Virtual Processors.
Role: SBS 2011, DNS, DHCP, Active Directory, Exchange, SQL Database.

Name: VS-02
Operating System: Server 2008 R2 SP1.
Specifications:
RAM: 4GB
CPU: 2  Virtual Processors.
Role: Terminal Server

Name: VS-03
Operating System: Server 2008 R2 SP1.
Specifications:
RAM: 4GB
CPU: 4 Virtual Processors.
Role: Trend Micro Server (Worry-Free business Security Advanced 8.0 SP1)

--Computers--
Windows 8.1: 2
Windows 7: 5
Windows XP: 3

Router: Cisco 877 SMB Router.
Switch: Linksys 24Pt Gigabit switch.

--- Issue:
Erratically, random computers will lose the ability to browse the internet. During this period of time they can still access local resources on the server and ping the gateway router. RDP access would not work though. Strangely the affected computers cannot be pinged by the router. Other devices can ping the affected computer.
Both Windows 8.1 and Windows 7 computers have had the issue. None of the Windows XP computers. Server: VS-02 and VS-03 have experienced symptoms but not VS-01 and VH-01.

On one of the computers we have discovered the following error in the Event Log:

== The system failed to register host (A or AAAA) resource records (RRs) for network adapter with settings:
==== Adapter Name : {67018E35-2310-4D71-BACF-13747FD76F41}
==== Host Name : D-07
==== Primary Domain Suffix : Domain.local
==== DNS server list :
===== 192.168.1.25, 8.8.8.8
==== Sent update to server : <?>
==== IP Address(es) :
====  192.168.1.112
== The reason the system could not register these RRs was because the DNS server contacted refused the update request. The reasons for this might be (a) you are not allowed to update the specified DNS domain name, or (b) because the DNS server authoritative for this name does not support the DNS dynamic update protocol.
== To register the DNS host (A or AAAA) resource records using the specific DNS domain name and IP addresses for this adapter, contact your DNS server or network systems administrator.

Users can fix the issue by restarting the computer or by unplugging the Network cable and plugging it back in.


--- Diagnosis so far:
When the issue first occurred, it was thought to be only affecting one computer, a windows 8.1 device. The resolutions attempted were the following:
We accessed event log remotely and located the error found above. Through research we tried to fix the ‘A’ record permissions for the computer in DNS Manager on VS-01 by giving the user full control. We made sure the records weren’t stale and ensured the scavenging was setup correctly. We cleared the DNS cache and updated the files. We then enabled secure and non-secure dynamic updates in DNS Manager. We also restarted the DNS and DHCP services and flushed and registered the DNS on the computer.
None of the above had any effect.

Via PSEXEC we created a firewall rule on the computer to allow port 80. Doing so allowed a VNC connection. Problem appeared solved, or so we thought.

It was revealed that other users had the problem and hadn’t thought to mention it to anyone.
All windows 7 and 8 computers were suffering from the issue. Strangely Windows XP was spared.

To fix the problem this time we approached Group Policy. We created a GPO to set a firewall rule to allow Port 80. We were impatient so we manually created this rule as well. This didn’t resolve the problem this time though.

It was realized that certain update services GPOs and some specific client GPOs missing. A bit of a mystery there. We imported spares from another SBS 2011 server and fixed them.

We set up a secondary DNS (it previously only had the SBS server) in DHCP for Google’s DNS: 8.8.8.8.

We also ensured every computer had it set in the network adapter properties to register with DNS.

When VS-02 was affected this enabled us an opportunity to diagnose by logging onto the server using the console connection via the Hyper-V host. We were able to confirm that VS-02 could ping the router and the SBS server. The router could not ping the server though.
We setup another spare router and swapped them to test. The issue was still occurring.
We swapped the switch and the issue still occurred.

A red herring we encountered was the discovery that a drive in the RAID on VH-01 was failing. We replaced the drive in hopes that the failing RAID (and thus affected performance) had been the cause of the strange occurrences. No dice.

We’ve also disabled the Anti-Virus and turned off VS-02 and VS-03 to no avail.

I think that covers everything so far.
Currently we’re at a loss. Maybe we’ve missed something obvious or maybe we should burn the building down. Any help here would be much appreciated.
0
Comment
Question by:tech_tonic
  • 4
  • 3
8 Comments
 
LVL 16

Expert Comment

by:Carol Chisholm
ID: 39977374
Spurious reverse DNS entries?
A non-existent DNS server somewhere in AD?
A non-existent DNS server in the DHCP settings?
IPv6?
0
 
LVL 2

Author Comment

by:tech_tonic
ID: 39977594
Hi Wizard,

Thanks for the reply. I checked reverse DNS and actually noticed a reference to the old server we migrated from, it was set as a name server, but the IP address was unknown.

After removing this record I decided to go through all of my DNS and DHCP records and discovered a lot of records which were pointing back to the old server we migrated from. Additionally I opened ADSI edit and removed any other references in there. Active Directory appeared clean.

I swear I went through DNS a thousand times, but I never actually noticed any of these. Thanks for your help, I'm really hoping this is the cause of the issue.
0
 
LVL 16

Expert Comment

by:Carol Chisholm
ID: 39977676
Make sure your domain controllers are not referencing the old server, as I think they can put it back in the DNS. Check also in your DNS settings that the old server is not a name server for any of your DNS zones.
0
 
LVL 2

Author Comment

by:tech_tonic
ID: 39990521
Hi Carol,

The domain controller is not refreshing the entries back into DNS.

I believe I may have fixed the virtual servers from losing connection, however it is still happening to users and they either have to restart their computers or renew their IP address to regain internet connection:

The server has an Intel Dual Port NIC, Dual port Broadcom NIC and a seperate management port.

It is currently only using the Intel Dual port NIC and the management port.

Currently one of the Intel adapters is used completely for the virtual host whilst the other is set as the Virtual Switch. The "Allow management operating system to share this network adapter" is ticked on this as well.

I made a change and set an IP address statically on the virtual switch as it was getting it via DHCP.
0
Maximize Your Threat Intelligence Reporting

Reporting is one of the most important and least talked about aspects of a world-class threat intelligence program. Here’s how to do it right.

 
LVL 16

Expert Comment

by:Carol Chisholm
ID: 39990826
Have you checked the permissions? Look in the event log.
0
 
LVL 74

Expert Comment

by:Jeffrey Kane - TechSoEasy
ID: 39991179
We set up a secondary DNS (it previously only had the SBS server) in DHCP for Google’s DNS: 8.8.8.8.

Why did you do this?  You should ONLY have the SBS's IP address listed for the DNS Server for all machines on your network.

The reason you are getting the error is that the Google DNS Server won't allow updates from your local LAN IP's.

Please remove the 8.8.8.8 entries (both from statically set and DHCP config), refresh both SBS and a workstation and then if you are still having trouble, please post a COMPLETE ipconfig /all from both the SBS and a workstation.
0
 
LVL 2

Accepted Solution

by:
tech_tonic earned 0 total points
ID: 40739302
ARP poisoning was occurring in the network due to a user plugging a router into the network to use as a switch.
0
 
LVL 2

Author Closing Comment

by:tech_tonic
ID: 40745781
My own solution.
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Are you one of those front-line IT Service Desk staff fielding calls, replying to emails, all-the-while working to resolve end-user technological nightmares? I am! That's why I have put together this brief overview of tools and techniques I use in o…
Trying to figure out group policy inheritance and which settings apply where can be a chore.  Here's a very simple summary I've written which might help.  Keep in mind, this is just a high-level conceptual overview where I try to avoid getting bogge…
Windows 8 comes with a dramatically different user interface known as Metro. Notably missing from the new interface is a Start button and Start Menu. Many users do not like it, much preferring the interface of earlier versions — Windows 7, Windows X…
In this video, we discuss why the need for additional vertical screen space has become more important in recent years, namely, due to the transition in the marketplace of 4x3 computer screens to 16x9 and 16x10 screens (so-called widescreen format). …

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

19 Experts available now in Live!

Get 1:1 Help Now