VM problems after ESXi upgrade
I recently upgraded ESXi Free on three physical servers, from 6.7, to the latest 7.0 Update 2. Rather than do an in-place upgrade, I simply installed the 7.2 on new USB flash drives, then re-registered the existing VMs. This seemed to go smoothly, and the VMs started up fine.
I left everything for a few days, but then started noticing problems; when I went to edit a VM, the properties page wouldn't open. Same with restarts or shutdowns; it doesn't matter whether I perform any of these operations in a browser or from VMWare Workstation, the same thing happens - the VM goes into a sort of hang state and becomes unavailable. It even happens if I tell VMs to shutdown or restart from within Windows. I can issue a power off command which eventually works, after about an hour.
If I reboot the host machine, everything seems to work as it should straight after booting. When I come back next day, problems happen again. I'll emphasise that all VMs function perfectly unless I go to a. edit one, or b. perform a power operation (shutdown, restart).
Interestingly, the older machine (which is full of HDDs) is unaffected. The two newer machines (full of SSDs) are the problem ones. The older server has a couple of RAID1 arrays, and the two problem ones are on RAID10 - one has Avago MegaRAID, and the other is an Intel RAID controller (I think).
When it was on ESXi 6.7, latterly I'd installed the Avago StorCLI on one of them, but it worked fine without it for a long time. I haven't installed any additional RAID-related software on the new 7.0U2 - is this where I'm going wrong?
It 'feels' like a storage issue - does anyone have any suggestions? Any advice greatly appreciated!
Thanks
EDIT: I've upgraded the hardware compatibility on a couple of them, and installed VMWare Tools 11.3.0, but this doesn't seem to help.
I left everything for a few days, but then started noticing problems; when I went to edit a VM, the properties page wouldn't open. Same with restarts or shutdowns; it doesn't matter whether I perform any of these operations in a browser or from VMWare Workstation, the same thing happens - the VM goes into a sort of hang state and becomes unavailable. It even happens if I tell VMs to shutdown or restart from within Windows. I can issue a power off command which eventually works, after about an hour.
If I reboot the host machine, everything seems to work as it should straight after booting. When I come back next day, problems happen again. I'll emphasise that all VMs function perfectly unless I go to a. edit one, or b. perform a power operation (shutdown, restart).
Interestingly, the older machine (which is full of HDDs) is unaffected. The two newer machines (full of SSDs) are the problem ones. The older server has a couple of RAID1 arrays, and the two problem ones are on RAID10 - one has Avago MegaRAID, and the other is an Intel RAID controller (I think).
When it was on ESXi 6.7, latterly I'd installed the Avago StorCLI on one of them, but it worked fine without it for a long time. I haven't installed any additional RAID-related software on the new 7.0U2 - is this where I'm going wrong?
It 'feels' like a storage issue - does anyone have any suggestions? Any advice greatly appreciated!
Thanks
EDIT: I've upgraded the hardware compatibility on a couple of them, and installed VMWare Tools 11.3.0, but this doesn't seem to help.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Yes, you should not see the issues on SD/HDD, or look at workarounds I posted!
ASKER
Thanks again Andrew - that's really helpful.
The upgrade was enforced really - we're prepping for a Cyber Essentials Plus onsite audit, and Nessus flags up 6.7 U3 as vulnerable - which would mean an audit fail.
It even flags up the version of VMWare Tools that ships with 7.0U2a as vulnerable - I had to download v11.3.0 from the VMWare site to make it happy.
The upgrade was enforced really - we're prepping for a Cyber Essentials Plus onsite audit, and Nessus flags up 6.7 U3 as vulnerable - which would mean an audit fail.
It even flags up the version of VMWare Tools that ships with 7.0U2a as vulnerable - I had to download v11.3.0 from the VMWare site to make it happy.
bad timing to upgrade, also you need to check what red flags occur!
Could 6.7U3, not have been fixed with a patch ? Many 6.7U3 still exist in the wild!
VMware Tools is an application like any other, and needs regular updating like any other Windows OS Patch Tuesday.
Upgrading to the latest ESXi version, often does not include the latest VMware Tools, and some ESXi updates can include or not include VMware Tools.
Could 6.7U3, not have been fixed with a patch ? Many 6.7U3 still exist in the wild!
VMware Tools is an application like any other, and needs regular updating like any other Windows OS Patch Tuesday.
Upgrading to the latest ESXi version, often does not include the latest VMware Tools, and some ESXi updates can include or not include VMware Tools.
ASKER
Just a final note about this: Andrew, you were completely right - the USB thing was the problem. Installing 7.0U2a on a fixed drive solved the issue. Interestingly though, of the three servers I thought were going to be affected, only one actually was:
Not sure if there is any correlation here, but might be useful for others.
- Old Dell PowerEdge T420 - unaffected
- Custom rack server with Intel S2600WFT motherboard, with Intel RAID controller - unaffected
- Identical custom rack server with same S2600WFT motherboard, but with Avago RAID controller - affected.
Not sure if there is any correlation here, but might be useful for others.
ESXi 7.0.2 and later is bugged and causing terrible issues because VMware did not test it!
It appears to be random there is a fix coming…,,
Thanks for feedback
It appears to be random there is a fix coming…,,
Thanks for feedback
ASKER
Thanks for your reply. So you reckon if I just installed VMWare on a fixed drive partition that would fix it? On two servers that's ok - there's on old partition I could use, but on one of them there isn't - so I'm goosed, basically. lol
I'll look into this - thanks.