VMWare VSphere ESXi and Virtual Center
Posted on 2010-09-08
Issue with Virtual Center adding a VMWare ESXi host
I currently have a single VMWare VSphere ESXi host, version (and build) number 4.1.0, 260247. This host has a 4 port Ethernet NIC, with one port connected, and a fiber channel HBA to attach to our shared storage. The host is being utilized for a demo of VMWare View, and won’t be needed for HA, DRS, etc.
I have 2 virtual machines, both Windows server 2003 on the host. Both are patched and part of a production AD domain and tree. On one virtual machine, I have installed Virtual Center, and I am utilizing the version of SQL included since our demo will be fairly small scale. The Virtual Center version is 188.8.131.5221. So, the virtual center server is running as a virtual machine (which I have done before without issue)
The problems center around adding the VMWare host to Virtual Center. I can add the host successfully. However, after about a minute, the host goes into a disconnected state. I can reconnect, and sometimes the reconnect will work, other times, I have to walk through the process of specifying the IP address, root account and password, etc. And, other times, I have to completely remove the host from Virtual Center, and re-add. But, the bottom line is once I add the host, it goes into a disconnected state within 2-3 minutes.
The error I receive the majority of the time is: A general system error occurred: Internal error: vmodl.fault.hostcommunication.
To troubleshoot, I’ve done the following so far:
1. Verify I can successfully access the host directly with the VSphere client from several different desktops as well as the virtual center machine. When doing this, I never lose connectivity with the VMWare host.
2. Run extended pings. No packets are lost
3. Verify no firewalls turned on or between my location and the vmware server
4. Successfully telnet to port 902 (or whatever the default port is; I can’t recall if it’s 902, 903, etc). But, whatever it is, when I look it up, I am able to successfully telnet to it.
5. The VMWare ESXi server is not in any DNS tables, nor are the 2 windows virtual servers. So, I’ve modified the hosts file on each of them with the appropriate settings. I can ping to all by name, including just the name as well as the FQDN.
6. The 2 virtual machines were not in the AD domain, so I requested one of the AD administrators add these in. He did, and in the process changed the names of the servers. So, I updated the hosts file.
7. In addition to the name change and updating the hosts file, I completely removed virtual center from the VM it was on, including the SQL portion. Basically, my aim was to remove any possible references to the previous VM’s names. Then, I reinstalled both and attempted to add the ESXi host back in. Still the same results.
8. Utilized the SSH and restarted the services on the ESXi host several times.
9. When examining the virtual center system logs closer, I observed several SSL certificate errors, including host name could not be verified, etc. (Unfortunately, I left the exact errors at work; I will post them when I return to work). This led me to believe I may have a corrupted SSL certificate either on the virtual center or ESXi host, or both. So, I used openssl to generate a new ssl certificate and imported it. After doing that, I now get the error: authenticity of hosts ssl certificate is not verified
I can’t get the host to go into Virtual Center now. So, it comes down to those 2 errors, and the above are the steps I’ve taken so far. The original error, A general system error occurred: Internal error: vmodl.fault.hostcommunication seems to be related to the virtual machine renames and a possibly corrupt SSL certificate. I have modified the hosts files and reinstalled virtual center, so the host renames should be out of the picture. However, my attempt to generate a new SSL certificate seems to have made things worse. At this point, I’m not really sure which way to go.
I’d appreciate any suggestions, etc so I can get back on track with this. It’s quite frustrating to spend a day and a half on something that usually only takes me a minute or two (literally). Hopefully, at least I’ll learn something interesting from all this :)