Solved

Cannot Connect to Vmware Management Port after Upgrade

Posted on 2013-11-30
14
1,281 Views
Last Modified: 2016-11-23
I have just recently upgraded an entire vcenter system from 4.1 to 5.1. As well as 10 hosts in 1 site and 1 in the other site.
All seemed to go rather well except for the 1 host in the remote site.
I cannot connect to the management port of the host, however when it was first upgraded I could. Not until I upgraded the vsphere tools on the VMs on the host did I lose the connection.
The VMs running on the host are still all connected and working.
In vcenter the host shows as offline and I cannot ping the IP address of the host.
I have tried restarting the management connection and also the whole server.
I did notice last night when I turned off the VMs that after about an hour I could connect to the management port, but then lost the connection again after about an hour.
I can login to the server via ilo so have been able to troubleshoot from there
I tested the management port from the host and I can ping the default gateway but no further. From other servers I can ping the same default gateway but not the host management port.
The host is in a vlan which is properly configured from what I can tell. Especially as it was working before the upgrade.
I have had the networking team investigate and they have confirmed the switches and vlans are correct but they cannot connect to the actual host, although the port on the switch shows it is active.
I have tried disabling the vmtools on the VMs on the host but this has had no effect.
The management port has 2 network connections with 1 running in standby, I have also tried setting the standby as active but to no avail.
The host itself is showing it has properly updated to the correct version of 5.5
The server is an HP DL380 G7 about 2 years old
I can connect to the host via ilo and have even tried changing the IP address which had no effect, as I thought maybe there was an IP clash
The differences between this host and the other hosts is it is HP and the others are Dell, also for the upgrade I upgraded the Dell servers via upgrade Manager, and the HP was initiated by upgrade manager but as there was an upgrade disk in the server I accidentally upgraded it from that, but this still seemed to upgrade fine.
Any help is much appreciated
0
Comment
Question by:CodeBlueEngineers
  • 7
  • 7
14 Comments
 

Author Comment

by:CodeBlueEngineers
ID: 39687675
Also forgot to mention the connection seems to be intermittent, ie last night I was able to connect for about an hour, then lost the connection again
0
 
LVL 117
ID: 39687877
if you can get onto the host via iLo, can you ping the default gateway?

is the default gateway set correctly?
0
 

Author Comment

by:CodeBlueEngineers
ID: 39688004
Hi there, yes I can ping the default gateway but no further. The strange part I do not understand is from other servers I can ping that same DG and it is definitely correct.
0
 
LVL 117

Accepted Solution

by:
Andrew Hancock (VMware vExpert / EE MVE) earned 500 total points
ID: 39688104
This appears to be a routing issue?
0
 

Author Comment

by:CodeBlueEngineers
ID: 39688187
I have had our networking team investigate and they have said the switching and routing is correct, and that when connected directly to the switch they cannot connect to the host.
Its strange that it happened directly after the upgrade as well.
0
 
LVL 117
ID: 39688405
There have been some networking changes in 5.1 between vmkernel and management network routing.
0
 

Author Comment

by:CodeBlueEngineers
ID: 39688531
Thanks, I have logged a job with vmware so hopefully they can find the issue. If its to do with the network routing or vmware changes as above I will allocate you the points
0
Netscaler Common Configuration How To guides

If you use NetScaler you will want to see these guides. The NetScaler How To Guides show administrators how to get NetScaler up and configured by providing instructions for common scenarios and some not so common ones.

 
LVL 117
ID: 39688630
Upload your networking screenshot.
0
 

Author Comment

by:CodeBlueEngineers
ID: 39688641
Just uploaded now, just had to cover the names of the servers.
As you can see VM's and management use the same network port.
Which is strange that it works for the VM's but not the management network
vmware-networking.PNG
0
 
LVL 117
ID: 39688682
What is glaringly obvious is the host is on a different VLAN

Check vlan configuration is correct on the trunk eg physical network config. On physical switches connected to host server and all other switches in network.

You also have a single nic with all traffic running over....so check whatever config is running on this.

Seems a little on the light your vSwitch I would trunk both NICs so both active active
0
 

Author Comment

by:CodeBlueEngineers
ID: 39688696
Yes that does seem to be what is very obvious, I have the networking team checking out the switching and vlans.
Do you think there may be a chance that the management port is being flooded?
0
 
LVL 117
ID: 39688749
We are fans of the management port on its on vSwitch0 with at least two NICs.

Same for VMs its on vSwitch1 with two NICs.

Now you have VLANs which are fine but VLAN diagnosis and asking a network team for virtualization on a specific VLAN needs specific network probes on each vlan and specialist monitoring which your network tram may have so ask them for utilisation figures for each vlan

And of course that's not errors or a total value for the pipe.
0
 

Author Comment

by:CodeBlueEngineers
ID: 39691358
Just found there is another device on the same subnet which has the same IP address as the default gateway.
Once we removed the device it started working straight away
So you were right hanccocka it was network related
0
 
LVL 117
ID: 39691402
Glad it was resolved!
0

Featured Post

VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

Join & Write a Comment

Join Greg Farro and Ethan Banks from Packet Pushers (http://packetpushers.net/podcast/podcasts/pq-show-93-smart-network-monitoring-paessler-sponsored/) and Greg Ross from Paessler (https://www.paessler.com/prtg) for a discussion about smart network …
In this article, I will show you HOW TO: Create your first Windows Virtual Machine on a VMware vSphere Hypervisor 6.5 (ESXi 6.5) Host Server, the Windows OS we will install is Windows Server 2016.
Teach the user how to configure vSphere clusters to support the VMware FT feature Open vSphere Web Client: Verify vSphere HA is enabled: Verify netowrking for vMotion and FT Logging is in place or create it: Turn On FT for a virtual machine: Verify …
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…

707 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

15 Experts available now in Live!

Get 1:1 Help Now