Hyper-V Guest OS unable to connect to Microsoft
Weirdest thing I've seen. Brand new W2K8 R2 Enterprise server, hypervisor install went fine. Built a new VM (also W2K8 R2), went to register Windows and got timeout error 0x80072EE2. Windows update failed with 8024402F. The hypervisor can hit microsoft.com, register, update and the like with no problems. Guest OS can hit google in a browser with no problem. Can ping other sites by name or IP with no error, but a ping attempt at MS does a DNS resolution and times out on the ping.
Basically the OS will just NOT communicate with MS anything. Kind of sounds like virus in the hosts file type behavior...but isn't. It's a brand new build. Haven't even added roles, RDP, shares, or anything. Disabled the on-system firewall and still no luck. Filtering not happening via our perimeter routing/firewall system either.
Blew away the VM, and started from scratch. Same behavior on the second and third attempts.
Not sure if this is significant, but it's an HP OEM version of the OS.
Completely at a loss for what could be causing this.
Thanks in advance for any pointers,
Basically the OS will just NOT communicate with MS anything. Kind of sounds like virus in the hosts file type behavior...but isn't. It's a brand new build. Haven't even added roles, RDP, shares, or anything. Disabled the on-system firewall and still no luck. Filtering not happening via our perimeter routing/firewall system either.
Blew away the VM, and started from scratch. Same behavior on the second and third attempts.
Not sure if this is significant, but it's an HP OEM version of the OS.
Completely at a loss for what could be causing this.
Thanks in advance for any pointers,
ASKER
Turns out MS has disabled ICMP. Didn't think to check ping from other machines. Still no connection with Win Update, Registration, or web. Turned IE Enhanced Security Configuration to off for Admin and users (temporarily) and same results.
BITS is active (starts auto when Windows Update begins, even though it was set to "Manual" initially.) I set it to Auto (Delayed Start) and fired it off again. Still no-go.
We do have a firewall, but it's not blocking anything related to this. That, and the host OS is hitting everything fine. Also put the problematic guest OS on the domain with no issues.
I've tried most of the steps on that link a few times already, but in no particular order. I think I'm going to stop, clear my head, and go through it step-by-step again in an orderly fashion - because sometimes in my haste I miss little things.
BITS is active (starts auto when Windows Update begins, even though it was set to "Manual" initially.) I set it to Auto (Delayed Start) and fired it off again. Still no-go.
We do have a firewall, but it's not blocking anything related to this. That, and the host OS is hitting everything fine. Also put the problematic guest OS on the domain with no issues.
I've tried most of the steps on that link a few times already, but in no particular order. I think I'm going to stop, clear my head, and go through it step-by-step again in an orderly fashion - because sometimes in my haste I miss little things.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I don't think our firewalling system has that limitation, but at this point I'll check anything. It will save me hair-loss.
This is the second 2008 system, kind of; if you count the hypervisor that it's running on. That machine is working fine, which is the frustration for me. If one didn't work initially, and the second failed too - that's expected, but the non VM install went pretty slick and worked out of box. Same media, same OS, same physical hardware, same NICs, and firewalling.
Another just-found situation that sucks, but is good in that it's a second data-point, is that my AV install isn't getting to the external update server, but the host system can. I'm leaning toward something getting lost between the VM and host.
This is the second 2008 system, kind of; if you count the hypervisor that it's running on. That machine is working fine, which is the frustration for me. If one didn't work initially, and the second failed too - that's expected, but the non VM install went pretty slick and worked out of box. Same media, same OS, same physical hardware, same NICs, and firewalling.
Another just-found situation that sucks, but is good in that it's a second data-point, is that my AV install isn't getting to the external update server, but the host system can. I'm leaning toward something getting lost between the VM and host.
Is this AV host is also a virtual machine? If yes, it makes sense.
Did you rule out any hardware problem, like cables, switch ports, etc.? I presume your server has multiple network ports and, I also presume, you have had separated the management traffic to the Hyper-V host from the data traffic of the virtual machines on different network ports. May be you could try to create another virtual switch from different network adapter on the Hyper-V host and plug the VMs there.
What the Windows Update log says? You should be able to find much more information about the error there. Obviously, all those problems are related, I just think that the update issue will be easier to troubleshoot. It’s located in %windir%\Windowsupdate.log
Did you rule out any hardware problem, like cables, switch ports, etc.? I presume your server has multiple network ports and, I also presume, you have had separated the management traffic to the Hyper-V host from the data traffic of the virtual machines on different network ports. May be you could try to create another virtual switch from different network adapter on the Hyper-V host and plug the VMs there.
What the Windows Update log says? You should be able to find much more information about the error there. Obviously, all those problems are related, I just think that the update issue will be easier to troubleshoot. It’s located in %windir%\Windowsupdate.log
ASKER
AV Server it's set to pull updates from is on the internet. Update log file didn't have a lot of extra information, unfortunately. Nothing jumped out at me.
Odd twist though. I went to update.microsoft.com and it spooled for a second and then started its redirection. Ended up loading the "Use your Start menu to check for updates" page - so I'm thinking there's some kind of security settings somewhere that is blocking it from connecting. No idea why the hypervisor is not doing this. I've tuned IE down ( I have also turned off Certificate Revocation, as per: http://support.microsoft.com/kb/816897) and still getting nowhere.
The fact that it's going to the rerouted page on microsoft is promising though. I have a request up on tech net too. So, if I get a solution, I'll be sure to repost.
I just got wire and ends (lending out cable bit me in the butt this time), so I'm going to put it in different switch ports and switches - just in case. I don't think it will help, but it certainly can't hurt.
Odd twist though. I went to update.microsoft.com and it spooled for a second and then started its redirection. Ended up loading the "Use your Start menu to check for updates" page - so I'm thinking there's some kind of security settings somewhere that is blocking it from connecting. No idea why the hypervisor is not doing this. I've tuned IE down ( I have also turned off Certificate Revocation, as per: http://support.microsoft.com/kb/816897) and still getting nowhere.
The fact that it's going to the rerouted page on microsoft is promising though. I have a request up on tech net too. So, if I get a solution, I'll be sure to repost.
I just got wire and ends (lending out cable bit me in the butt this time), so I'm going to put it in different switch ports and switches - just in case. I don't think it will help, but it certainly can't hurt.
If it is a fresh installation of WS 2008 there is no security setting that will prevent a local/domain administrator from activating Windows or executing Windows Update. It works without any special configurations.
Did you try to install WS2008 from an evaluation DVD on a new VM, to rule out the OEM? Do you have the same problems with it or other VMs (besides, the one with AV)? Do you have another services installed on the Hyper-V host, like anti-virus for example?
Did you try to install WS2008 from an evaluation DVD on a new VM, to rule out the OEM? Do you have the same problems with it or other VMs (besides, the one with AV)? Do you have another services installed on the Hyper-V host, like anti-virus for example?
ASKER
Two more VMs added, and same symptoms. I'm going to start looking at settings on the hypervisor itself to see if it's doing something goofy to the connections the VMs are using...even though I made sure to isolate the NICs from the host OS.
It seems more as a connection problem. You could test the DNS using nslookup. Try the following commands:
The first will test the local dns server for Microsoft update web site. The second will test one of the biggest DNS servers at 4.2.2.2.
Then you could set different DNS server addresses for you servers and try again.
You can also run a constant ping command against 4.2.2.2
nslookup update.microsoft.comC:\>nslookup
> server 4.2.2.2
> update.microsoft.comThe first will test the local dns server for Microsoft update web site. The second will test one of the biggest DNS servers at 4.2.2.2.
Then you could set different DNS server addresses for you servers and try again.
You can also run a constant ping command against 4.2.2.2
ping -t 4.2.2.2 ASKER
nslookup resolved the address, and the ping works to 4.2.2.2 (and a myriad of other sites), and of course the ping to the resolved MS update address (65.55.184.16) times out because they have ICMP reply turned off on their end.
I'm really starting to think there's something on the hypervisor that is blocking the VMs from connecting to ActiveX, or sites with anything short of plain HTML. Well, that's the theory so far. I have a feeling I'm going to have to pony up the dough and call MS tomorrow.
(Appreciate all the help, BTW!)
I'm really starting to think there's something on the hypervisor that is blocking the VMs from connecting to ActiveX, or sites with anything short of plain HTML. Well, that's the theory so far. I have a feeling I'm going to have to pony up the dough and call MS tomorrow.
(Appreciate all the help, BTW!)
There is nothing on the hypervisor that could block something in the VMs. The only way the host communicates with the virtual machines is throught its integration services.
I would check the external firewall again. You could also bypass it if it is possible. It won’t hurt just for several minutes; Windows 2008 has very good integrated firewall that blocks everything when it is set to public network profile.
Did you try an evaluation version of Windows 2008 or even Windows 7?
I would check the external firewall again. You could also bypass it if it is possible. It won’t hurt just for several minutes; Windows 2008 has very good integrated firewall that blocks everything when it is set to public network profile.
Did you try an evaluation version of Windows 2008 or even Windows 7?
ASKER
Finally got around to getting an eval copy up and running.
Same thing.
Time to start drinking and calling Redmond...
Same thing.
Time to start drinking and calling Redmond...
ASKER
90 minutes with Microsoft, and the conclusion was...
... tier-2 time. (More as situation develops.)
... tier-2 time. (More as situation develops.)
ASKER
Over 20 hours of phone/email work and MS still has no answer...
...but in my fiddling around with it, I noticed something weird.
I deleted ALL the virtual net connections, and created just one. The hypervisor uses NIC#2 w/ static address.
I set the single Virtual Switch to use NIC#1. Hypervisor works fine.
I select "Allow management operating system to share this network adapter" and now the hypervisor stars exhibiting the same symptoms.
I am using IPTABLES for firewalling. It's not blocking this MAC, IP, etc. Right now that's the direction the tech at MS wants to go. Going to run back to them with this new information.
...but in my fiddling around with it, I noticed something weird.
I deleted ALL the virtual net connections, and created just one. The hypervisor uses NIC#2 w/ static address.
I set the single Virtual Switch to use NIC#1. Hypervisor works fine.
I select "Allow management operating system to share this network adapter" and now the hypervisor stars exhibiting the same symptoms.
I am using IPTABLES for firewalling. It's not blocking this MAC, IP, etc. Right now that's the direction the tech at MS wants to go. Going to run back to them with this new information.
As I pointed in a previous post, it seams more likely a firewall problem outside of the Hyper-V host and the VMs. Is there any way to bypass your firewall and plug it directly to the Internet router? Windows 2008 has a pretty good firewall and you won’t be exposed if the server is connected directly to Internet for several minutes. Even better if you have a cheap home router, you could use it for NAT.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
Ultimately did not turn out to be any problem external to the VM and HP (Broadcomm) NIC. Firewall and network were performing as expected.
Do you have an external firewall between the Virtual machine and your DNS server? Some hardware firewalls could cause timeouts with Windows Server 2008 DNS requests, Cisco ASA is one of them.
Check if Background Intelligent Transfer Service is running on the VM. It has to be Automatic (delayed start) startup type.
The OEM DVD should not cause a problem but you need to use the serial number for virtual machines. However, you could download the evaluation version of WS 2008 R2 and install it on a VM to rule out the installation media.