Troubleshoot VMWare VCenter issue
esxi vCenter 6.7, NICs for VM traffic go down suddenly.
We have three esxi server with this physical network configuration:
vmnic0 & vmnic1 – 1Gb copper interfaces for management and vMotion
• Configured via vSwitch0
• Management is isolated to vmnic0; can fail over to vmnic1
• vMotion is isolated to vmnic1; can fail over to vmnic0
vmnic2 & vmnic3 – 10Gb fiber interfaces for iSCSI storage and VM traffic; configured via DSwitch01
• VM traffic traverses both interfaces
• iSCSI traffic is load-balanced across both physical interfaces, via two vmkernel ports bound to each interface (noted below); this allows for two paths to each datastore
Starting two weeks ago on one of the esxi vmnic2 & vmnic3 when down but at the switch side the port is up. We lost contact will all server in this host, vMotion fails in these machines.
The only solution is shutdown the esxi server. When it is back NICs are up.
I open a case with VMWare, they do not find software issue, same with DELL no hardware problem.
I notice that at the time of the three events a VEEAM backup of the file server was running and the VEEAM server was in in the faulty esxi (one time in #1 and two times in #3).
Any idea how to start to troubleshoot this issue
ASKER
6.7.0 Update 3 (Build 17700523) Image profile (Updated) ESXi-6.7.0-20191204001-standard (VMware, Inc.)
This pictures were taking during the event.
The three servers have the same configuration
This event happens already three times one in esxi1 and two times in esxi3
ASKER
No sure, but I noticed that when the problem showed up the backup server and file server were in the same host. I change settings to have both servers on different host.
I have a week without the issue.
ASKER
I like your recommendation of "All services and storage networks should be on their own networks, and VMs on their own vSwitch and networks."
This will take me some time because I not expert on VMWare. I have to do a little of everything, I need to do a lot of research. Maybe add extra physical NIC.
I do not have issues with the NICs in a couple of weeks. I have to assume that the issue was relation between the new device USB Anywhere and the fileserver with the backup server.
VMware, a software company founded in 1998, was one of the first commercially successful companies to offer x86 virtualization. The storage company EMC purchased VMware in 1994. Dell Technologies acquired EMC in 2016. VMware’s parent company is now Dell Technologies. VMware has many software products that run on desktops, Microsoft Windows, Linux, and macOS, which allows the virtualizing of the x86 architecture. Its enterprise software hypervisor for servers, VMware vSphere Hypervisor (ESXi), is a bare-metal hypervisor that runs directly on the server hardware and does not require an additional underlying operating system.
TRUSTED BY
Now that's out the way!
Are you referring to
1. The host VMNIC ?
2. The virtual machine VM (maybe VCSA)
If you've had VMware and DELL look at the issue with a hands on remote approach and look at logs, we can try to help, but we are limited in our actions!
Can we have screenshots of your networking on each host ?
we've got some quick fire, recommendations
1. Management Interfaces - vmnic0 and vmnic1 - Management Only. vSwitch0.
2. vMotion Interfaces - vmnicX and vmnicY - vSwitch1 - vMotion only - enable Jumbo Frames if hardware allows.
3. iSCSI Interfaces - vmnic2, and vmnic3 - vSwitchX - iSCSI only - nothing else, dedicated Storage Network.
This can be done with physical interfaces or VLANs
All services and storage networks should be on their own networks, and VMs on their own vSwitch and networks.
By losing iSCSI connections to the SAN, would render all datastores lost and VMs hanging, at the same time no communication to them, anyway.
Is this just a single host which does this ?
Have you updated firmware and or host ESXi 6.7 recently, which build of 6.7 are you using ?
Aer you using DELL OEM ?