Solved

Network issue - very very strange

Posted on 2011-03-21
43
312 Views
Last Modified: 2012-09-22
Hi all

Today all of a sudden, there are about 4 servers out of 20+ that are refusing inbound connections.

From the problem machines, you can ping anything and everything. Internet works.

Apart from that, nothing can communicate with these servers! Cant ping, cant browse, cant RDP. Acting very much like a firewall issue, except there is no firewall on these servers. Even uninstalled the AV to be sure.

Have done scans with Malwarebytes and Trend Online.

Looked through NIC properties, all looks ok. Changed switch ports, etc.

They are connected to a L3 managed switch and I can see the port that they are plugged into has RX/TX traffic as expected. TX is a bit on the high side in my opinion.

Free free to request any specific info.

Any ideas? I'm out of them.
0
Comment
Question by:hongedit
  • 19
  • 12
  • 5
  • +5
43 Comments
 
LVL 1

Author Comment

by:hongedit
ID: 35180942
Have updated NIC drivers also
0
 
LVL 2

Expert Comment

by:ghemstrom
ID: 35180993
What about IP configurations of the said servers compared to other servers. What about the network configuration of the network?
0
 
LVL 4

Expert Comment

by:cavp76
ID: 35181007
Event Viewer, anything strange there? Already rebooted the servers? What do they host/share?
0
 
LVL 6

Expert Comment

by:wwakefield
ID: 35181049
Might have a look at the destination on that outbound traffic also.

Are you able to take those four boxes offline temporarily?

What OS?

What server roles?

What antivirus?

Did the firewall mysteriously start up?
0
 
LVL 26

Expert Comment

by:pony10us
ID: 35181080
Don't forget to check the L3 switch for possible config changes. Look at it for traffic to/from those servers.
0
 
LVL 1

Expert Comment

by:paulms53
ID: 35181091
have any new windows updates or patches been installed recently?
0
 
LVL 1

Author Comment

by:hongedit
ID: 35181564
Some more background:

The servers with this issue is in Liverpool. They are connected via MPLS to the other sites, one of which is London.

We did an office move in London at the weekend (merging 2 offices) so some re-routing was done on the switches but nothing on the Liverpool side - everything there is the same as it always has been.

The servers are running Windows Server 2008. All host different things - File, Print, Citrix, Exchange, DFS, etc.

L3 switch config has not been changed in some time. AV is FortiClient (was McAfee).

No updates recently.

Netstat shows not a lot of incoming traffic. Is there a tool to see where outbound connections are going to (IP, port)?
0
 
LVL 1

Author Comment

by:hongedit
ID: 35181592
Experiencing some very strange issues also at the London site, which although are un-linked on face value may somehow be contributing to or affecting Liverpool.

There is a big ? over the integrity of the core switches in London (HP Procurves), there was/is a virus outbreak in London, and certain servers/Group Policy is acting up also...

0
 
LVL 26

Expert Comment

by:pony10us
ID: 35181673
First I would really check out the equipment in London.

Second, did you outside IP address change for the servers in London after the move? It could be that the routes in the other locations need to be updated with the new IP.
0
 
LVL 1

Author Comment

by:hongedit
ID: 35181835
No ip changes in London.

The office that they moved out of is still hosting the internet connection for the essential stuff (VPNs etc). They are connected via a LES so as far as the network is concerned its just Layer 2, no different to before.

Using TCPview (sysinternals) this server is just sending out local traffic. Nothing external. Wierd.

0
 
LVL 10

Expert Comment

by:TekServer
ID: 35181957
Are the servers unreachable by name, IP address, or both?
If name only, check the DNS records for possible corruption and/or bad information.

hth!
:)
0
 
LVL 1

Author Comment

by:hongedit
ID: 35181994
unreachable by name or IP.

Not even pingable by the switch it is connected to.

When I ping the server by its own hostname I get a replies from ::1: - IPv6 is off, does this have any significance?
0
 
LVL 26

Expert Comment

by:pony10us
ID: 35182256
Can the 4 servers see each other or a workstation in the same physical location see those servers?

It sounds like the switch and the NIC are not communicating or they are using DHCP and are not getting an IP address.

Try running ipconfig /all on one of the servers and confirm it is has an IP address.
0
 
LVL 1

Author Comment

by:hongedit
ID: 35182296
Yep, all have static ip addresses.

Problem exists on local physical network.

I have asked someone to cross cable into the server NIC to see if it is pingable that way, to rule out the rest of the network.

0
 
LVL 1

Author Comment

by:hongedit
ID: 35182437
Cannot ping each other through 3 different dumb L2 switches...think its safe to say this is not the network, its something on the servers!
0
 
LVL 26

Expert Comment

by:pony10us
ID: 35182481
It does sound that way.  

You mentioned updating NIC drivers.  Did you do that to all the affected servers durring the move?

paulms53: mentioned Windows updates.  Were any performed durring the move?

Was there any other significant change in hardware/software that took place at the same time as the move?
0
 
LVL 1

Author Comment

by:hongedit
ID: 35182577
Hi

All NIC drivers have been updated on the problem servers. The servers in Liverpool which have this issue arent really directly related to the move - to them, nothing has changed.

I checked Windows updates and the only thing to have happened on the Friday was an install of Windows Defender definition updates.

No changes of hardware of any kind.
0
 
LVL 4

Expert Comment

by:cavp76
ID: 35182590
Have someone plug a screen and a keyboard to the affected systems... what does the screen show?
0
 
LVL 10

Expert Comment

by:TekServer
ID: 35182608
When you ping one of those servers from itself, what kind of response times are you getting?

This may seem an odd question, so let me back up and provide some context.  I've been trying to Google your "::1:" results (not an easy task; Google doesn't much like those symbols!), and as you suspected almost everything I've found is pointing to IPv6.  But several of the pages I found were message board posts where someone was getting wildly erratic or even negative ping responses, which symptom was fixed by adding "/usepmtimer" to the boot.ini file.  I don't think that's likely to be your problem, but I figured it was worth mentioning.  ;)

Assuming the above tangent doesn't prove to be a solution for you, I would try this:
1.  Enable IPv6 on one or more of the affected servers, see if that fixes the problem.
2.  If that doesn't work, turn off IPv6 again, and consult this article to ensure that IPv6 is disabled.

I have also seen some rumors that Exchange 2007 needs IPv6, so that's something to consider as well.

Hope this dissertation yields some useful information ...
;)
0
 
LVL 1

Author Comment

by:hongedit
ID: 35182666
<1ms is all the replies.

Tried enabling IPv6, I also tried disabling via registry also (and rebooted). It now replies using v4 but still no luck with external connecting in.

I have just been told that when this first happened, the tech on site first found that the Authentication box ticked under NIC Properties (802.1x where you can drop down and choose Smart Card or EAP).

This was obviously disabled but still no luck.

Thanik you for all your suggestions, keep them coming...has to be something missed.
0
 
LVL 6

Expert Comment

by:wwakefield
ID: 35182674
No chance you have duplicate IP addesses running around the network?
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 6

Expert Comment

by:wwakefield
ID: 35182699
Or hostnames....
0
 
LVL 1

Author Comment

by:hongedit
ID: 35182724
Dont see why we would. These servers are on a different subnet to the "move" office so I doubt that anyone used an IP address my mistake.

Host names are pretty hard to duplicate by mistake too.
0
 
LVL 1

Author Comment

by:hongedit
ID: 35182769
Well, just tried changing its IP to one verified as being free and it still doesnt work :(
0
 
LVL 26

Expert Comment

by:pony10us
ID: 35182835
So:

Affected server(s) can ping itself but nothing else. Nothing can ping the server(s).

Using an L2 switch can a workstation in the same subnet ping the server?
Can the server ping the workstation?
Are you pinging by IP or Host Name?

0
 
LVL 6

Expert Comment

by:wwakefield
ID: 35182855
@pony10usL  beat me to it...
0
 
LVL 1

Author Comment

by:hongedit
ID: 35188136
Using an L2 switch can a workstation in the same subnet ping the server?
Can the server ping the workstation?
Are you pinging by IP or Host Name?

No
No
Both

The server acts the same whether its on the real network or on a dumb switch.
0
 
LVL 26

Expert Comment

by:pony10us
ID: 35189976
Okay, everthing has been eliminated except the server iteself. Since the only update was to Windows Defender have you tried turning that off?

Also, make sure the gateway and mask are correct.

In your original question you said: "From the problem machines, you can ping anything and everything. Internet works". But then you say you can't ping another machine even on an L2 switch. This part confuses me.

0
 
LVL 1

Author Comment

by:hongedit
ID: 35190470
I have tried disabling Windows Defender.

IP details are 100% correct.

To clarify:

On the problematic machines, they can seemingly operate as normal in terms of outbound network connections. They can ping the rest of the network, access the internet, etc.

But nothing can access them at all. Its exactly as if there is a firewall rejecting all inbound connections, but we know this isnt the case.

0
 
LVL 26

Expert Comment

by:pony10us
ID: 35190548
There is appears to be a contradiction here.
 
In post 35188136 you answered "No" about the server being able to ping a workstation on the the same subnet. Now you say in post 35190470 that it CAN ping the rest of the network.

0
 
LVL 1

Author Comment

by:hongedit
ID: 35190607
Apolagies, got myself in a muddle. Last response is the correct one!
0
 
LVL 26

Expert Comment

by:pony10us
ID: 35191172
So let me try to put this together again.

1. Servers in London were moved
2. Servers in Liverpool are the ones that have an issue (4 of them)
3. Servers in Liverpool can ping all devices and access everything EXCEPT the servers in London
4. Nothing can access the servers in Liverpool
5. Have tried isolating the network in Liverpool by connecting them and some workstations to an L2 switch
6. The only updates were to Windows Defender signature files
7. Have updated the NIC drivers

Now my questions
1. Are there only 4 servers in Liverpool and ALL of them have the issue?
2. Were there any changes to IP addresses in London?
3. Where is the DNS server located?
4. Have you tried clearing the ARP table on the servers in Liverpool?
5. Can the servers in Liverpool ping or tracert the servers in London? (what is the result)
6. Can the servers in London ping or tracert the servers in Liverpool? (what is the result)
0
 
LVL 1

Author Comment

by:hongedit
ID: 35191298
I like your style!

1. Yes
2. Today a server in London also has the same issue
3. Incorrect. Problem servers can even ping to London (and other sites they have). Just nothing can access them.
4. Nothing can access the problem servers. There are other servers in Liverpool with no issue, and they can be accessed fine.
5. Have patched all problem servers into L2 switch, all on same subnet, and the issue remains - they can ping out, but nothing can ping in.
6. Yes
7. Yes

1. No
2. No - additional IP's added, but no exisiting IP's changed.
3. Each site has their own DC(s) which also hosts the DNS
4. Not on the servers - I have cleared the ARP on the switches though.
5+6. Yes and No, depending on which servers...

Problematic servers can ping/trace everything, on any site as expected
No device can ping the problematic servers from anywhere.

Pings and Tracerts FROM problem servers TO anything work fine
Pings and Tracerts FROM anything TO problem servers time out.
0
 
LVL 6

Expert Comment

by:wwakefield
ID: 35192023
This screams software firewall on the problem servers.

Can we uninstall defender on server and check services to make sure the windows firewall is not operational?
0
 
LVL 26

Expert Comment

by:pony10us
ID: 35192196
I have to agree wwakefield that is software related.  Something on the servers has to have changed.

One thing that I don't seem to recall being covered here is what OS version is on the problematic servers? We keep trying to ping them but if they have ICMP turned off they will not respond.
0
 
LVL 1

Author Comment

by:hongedit
ID: 35192631
I agree, this is exactly what it feels like, but there are no visible firewalls on!

Windows Firewall is off, and disabled from starting in Services. Defender is disabled.

OS is Windows Server 2008 R2 x64. Using ping is just a way to test conectivity here...if the rest of the inbound connections worked then it wouldn't be as much of an issue.

Cant ping, trace, browse to (\\server or http), etc.

But yes, I agree, its almost as if it is just blanket denying all inbound packets.

One of the newly affected machines in London today was fresh out of the box. Literally had been on the network for 2-3 hours...then same thing happened to it.

The only 2 things I think it could be is:

1. A very intelligent virus/malware program that is blocking all inbound like a firewall, but unable to detect from Malwarebytes/Trend Online/Mcafee scans
2. Something is flying round the network and damaging the servers to act like this...but what?

Going to email techs to try repairing the winsock and tcp stacks now.
0
 
LVL 26

Expert Comment

by:pony10us
ID: 35192945
I don't know if this will help but is the RPC service running on the servers?  

http://askbobrankin.com/rpc_server_unavailable.html
0
 
LVL 4

Expert Comment

by:bjove
ID: 35193931
Do you have teaming configured on the servers?
Do you have secondary IP addresses configured on the servers?
Do you have VLAN trunking configured on the servers?
0
 
LVL 1

Author Comment

by:hongedit
ID: 35199300
Fixed....

Re-enabled the Windows Firewall service, but kept Windows Firewall off.

Dont know how or why. We disbaled the service after some servers kept mysteriously re-enabling their firewalls by themselves.
0
 
LVL 26

Expert Comment

by:pony10us
ID: 35199325
Glad to here that.  So it did all come back to the firewall. It had to be on the server(s) based on everything else that was tried.  
0
 
LVL 1

Author Comment

by:hongedit
ID: 35199382
I dont get why the Windows Firewall service has to be enabled though?

Isnt that a bit backwards - disabled the firewall and it firewall's the connection?!
0
 
LVL 26

Accepted Solution

by:
pony10us earned 500 total points
ID: 35199418
Here is a thread about the firewall service:   http://social.technet.microsoft.com/Forums/en-US/windowsserver2008r2networking/thread/1b862b1c-9de4-4f19-ac12-c13082ff24be

This comment may answer your question:

Microsoft does not support disabling the Firewall Service on any Vista or later operating system.  There are other "under the hood" dependencies. The firewall is not a "standalone" component any longer.  It is a front-end to the "base filtering engine", which is an integral component in the overall networking architecture in Vista and later versions of Windows.

As Wanderer_99 stated, just turn the firewall off.  You can do this in the Firewall Control Panel, the Windows Firewall with Advanced Security MMC snap-in, or with the "netsh advfirewall set allprofiles state off" command.   Doing so will stop the firewall from filtering any traffic.  We strongly recommend, however, that you have some firewall in place on each of your hosts.  Don't depend just on a perimeter firewall.  Most security experts will tell you that threats to your hosts more often originate inside your perimeter, than from the outside.


--------------------------------------------------------------------------------
Dave Bishop
Senior Technical Writer
Windows Server Networking User Assistance
0
 
LVL 1

Author Closing Comment

by:hongedit
ID: 35199455
I've awarded points for your contribution on the matter and the explanantion of why this happened.

Thanks to all.
0

Featured Post

Better Security Awareness With Threat Intelligence

See how one of the leading financial services organizations uses Recorded Future as part of a holistic threat intelligence program to promote security awareness and proactively and efficiently identify threats.

Join & Write a Comment

Meet the world's only “Transparent Cloud™” from Superb Internet Corporation. Now, you can experience firsthand a cloud platform that consistently outperforms Amazon Web Services (AWS), IBM’s Softlayer, and Microsoft’s Azure when it comes to CPU and …
A safe way to clean winsxs folder from your windows server 2008 R2 editions
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…
This tutorial will walk an individual through the process of transferring the five major, necessary Active Directory Roles, commonly referred to as the FSMO roles to another domain controller. Log onto the new domain controller with a user account t…

744 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

10 Experts available now in Live!

Get 1:1 Help Now