_KeepItSimple_
asked on
Hyper-V Time Synchronisation Issue
Hello,
I'm having issues with the network time slipping in our virtual environment. It's a simple setup as follows:
- Windows Server 2008 R2 host OS running on a Dell R710 box.
- Host OS is called HOST1 running Hyper-V.
- There are 2 guest VMs - both Windows Server 2008 R2.
- Domain controller VM is DC1.
- File server VM is FS1.
- All servers are domain-joined (including host).
Originally, I had all guests configured with the Hyper-V Time Synchronization option disabled as per official MS guidelines. DC1, being the PDC, was configured to obtain time from europe.pool.ntp.org. The other machines, being domain members, would synchronise with DC1. However, the event logs on all machines (physical and virtual) frequently contained the following warning:
Log Name: System
Source: Microsoft-Windows-Time-Ser
Event ID: 50
Description:
The time service detected a time difference of greater than 5000 milliseconds for 900 seconds. The time difference might be caused by synchronization with low-accuracy time sources or by suboptimal network conditions. The time service is no longer synchronized and cannot provide the time to other clients or update the system clock. When a valid time stamp is received from a time service provider, the time service will correct itself.
Investigating this issue, I came across the following article:
http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchronization-in-hyper-v.aspx
This article states strongly that the Hyper-V time synchronization option should be enabled for all VMs even in a domain environment. Therefore, I enabled this option for all VMs and followed the instructions in the "Configuring the Windows Time service to use an external time source" (manually) section of the following KB article:
http://support.microsoft.com/kb/816042
In doing so, I have once again configured DC1 to access pool.ntp.org for external time.
Having done all this, I've noticed that the network time is slipping by around 30 mins per day. I'm hoping someone here can advise on why this happening.
When I start the Windows Time service on DC1, the event log states:
The time provider NtpClient is currently receiving valid time data from 1.pool.ntp.org,0x1 (ntp.m|0x1|0.0.0.0:123->79
When I start the Windows Time Service on HOST1 and FS1, the event log states:
The time provider NtpClient is currently receiving valid time data from DC1.domain.local (ntp.d|0.0.0.0:123->192.16
All seems good at this point. However, the time starts to slip from now. The odd thing is that there are no further event entries on any machine for the Time-Service. However, I have noticed events on all machines from Kernal-General stating similar to:
The system time has changed to ¿2012¿-¿01¿-¿31T10:59:45.2
So it seems that the systems are synchronizing amongst themselves but DC1 isn't bothering to obtain time from the external source after the Windows Time service is first started.
Can anyone shed some light on this behaviour and how time should be configured in this environment to prevent errors/warning appearing in the logs and time slips?
I'm currently working around it by periodically restarting the Windows Time service on DC1.
Thanks.
I'm having issues with the network time slipping in our virtual environment. It's a simple setup as follows:
- Windows Server 2008 R2 host OS running on a Dell R710 box.
- Host OS is called HOST1 running Hyper-V.
- There are 2 guest VMs - both Windows Server 2008 R2.
- Domain controller VM is DC1.
- File server VM is FS1.
- All servers are domain-joined (including host).
Originally, I had all guests configured with the Hyper-V Time Synchronization option disabled as per official MS guidelines. DC1, being the PDC, was configured to obtain time from europe.pool.ntp.org. The other machines, being domain members, would synchronise with DC1. However, the event logs on all machines (physical and virtual) frequently contained the following warning:
Log Name: System
Source: Microsoft-Windows-Time-Ser
Event ID: 50
Description:
The time service detected a time difference of greater than 5000 milliseconds for 900 seconds. The time difference might be caused by synchronization with low-accuracy time sources or by suboptimal network conditions. The time service is no longer synchronized and cannot provide the time to other clients or update the system clock. When a valid time stamp is received from a time service provider, the time service will correct itself.
Investigating this issue, I came across the following article:
http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchronization-in-hyper-v.aspx
This article states strongly that the Hyper-V time synchronization option should be enabled for all VMs even in a domain environment. Therefore, I enabled this option for all VMs and followed the instructions in the "Configuring the Windows Time service to use an external time source" (manually) section of the following KB article:
http://support.microsoft.com/kb/816042
In doing so, I have once again configured DC1 to access pool.ntp.org for external time.
Having done all this, I've noticed that the network time is slipping by around 30 mins per day. I'm hoping someone here can advise on why this happening.
When I start the Windows Time service on DC1, the event log states:
The time provider NtpClient is currently receiving valid time data from 1.pool.ntp.org,0x1 (ntp.m|0x1|0.0.0.0:123->79
When I start the Windows Time Service on HOST1 and FS1, the event log states:
The time provider NtpClient is currently receiving valid time data from DC1.domain.local (ntp.d|0.0.0.0:123->192.16
All seems good at this point. However, the time starts to slip from now. The odd thing is that there are no further event entries on any machine for the Time-Service. However, I have noticed events on all machines from Kernal-General stating similar to:
The system time has changed to ¿2012¿-¿01¿-¿31T10:59:45.2
So it seems that the systems are synchronizing amongst themselves but DC1 isn't bothering to obtain time from the external source after the Windows Time service is first started.
Can anyone shed some light on this behaviour and how time should be configured in this environment to prevent errors/warning appearing in the logs and time slips?
I'm currently working around it by periodically restarting the Windows Time service on DC1.
Thanks.
ASKER
Hi demazter,
Thanks for the response. Prior to reading the Virtual PC Guy's blog article I would have always said to disable host sync but he puts forward a compelling argument. Have you read the article? Do you disagree?
I may give your recommendation a try if no other suggestions are forthcoming.
Thanks for the response. Prior to reading the Virtual PC Guy's blog article I would have always said to disable host sync but he puts forward a compelling argument. Have you read the article? Do you disagree?
I may give your recommendation a try if no other suggestions are forthcoming.
I did have a read of it and whilst he does seem to make a compelling arguement I can only go buy my own experiences and they differ from those in the blog.
1 -> If 4 VM's on the same host are part of the same domain it makes no difference if the time is wrong as long as they are ALL the same. If they are not you WILL have service problems.
2 -> If you Sync your host with an external time source then the "startup time sync" issue is resolved because there should be no drift from the time source your host is using to that the VM PDCe is also using.
3 -> Computer clocks drift, that's a fact, virtual or physical. So by setting the host to also Sync with an external time source you are covering all bases. The host has the correct time, the PDCe has the correct time and the other VM's that use NT5DS to Sync with the PDCe will also have the correct time.
1 -> If 4 VM's on the same host are part of the same domain it makes no difference if the time is wrong as long as they are ALL the same. If they are not you WILL have service problems.
2 -> If you Sync your host with an external time source then the "startup time sync" issue is resolved because there should be no drift from the time source your host is using to that the VM PDCe is also using.
3 -> Computer clocks drift, that's a fact, virtual or physical. So by setting the host to also Sync with an external time source you are covering all bases. The host has the correct time, the PDCe has the correct time and the other VM's that use NT5DS to Sync with the PDCe will also have the correct time.
The standard recommendation as I recall, is to disable hyper-v time sync for the domain controller VMs only, and leave time sync on for all other VMs. I had forgotten to do this on one of my virtual DCs once, and the host was drifting from the rest of the network because the host was getting time from the VM, and the VM was getting time from the host, and all of this was over-riding the normal Windows NTP process.
ASKER
Thanks, kevinhsieh. I'll try removing time sync on the DC VM only and see what happens.
turn off hyper v timesync
run everything as you would in a non vm environment for time sync
set clients with gpo's
make your pdc the primary time server
it's not worth messing with it
in one of the questions the guy admits theres a problem with hyperv time services
if you are in a production environment do this:
Do this on the domain controller (PDC) http://technet.microsoft.com/en-us/library/cc786897(WS.10).aspx
w32tm /config /manualpeerlist:"0.pool.nt
Do this on remaining domain controllers
w32tm /config /syncfromflags:domhier /reliable:yes /update
Do this on Windows clients http://technet.microsoft.com/en-us/library/cc758905(WS.10).aspx
w32tm /config /syncfromflags:domhier /update
run this after each change
net stop w32time && net start w32time
there is a NTP GPO that can be applied in AD that is very easy to configure.
run everything as you would in a non vm environment for time sync
set clients with gpo's
make your pdc the primary time server
it's not worth messing with it
in one of the questions the guy admits theres a problem with hyperv time services
if you are in a production environment do this:
Do this on the domain controller (PDC) http://technet.microsoft.com/en-us/library/cc786897(WS.10).aspx
w32tm /config /manualpeerlist:"0.pool.nt
Do this on remaining domain controllers
w32tm /config /syncfromflags:domhier /reliable:yes /update
Do this on Windows clients http://technet.microsoft.com/en-us/library/cc758905(WS.10).aspx
w32tm /config /syncfromflags:domhier /update
run this after each change
net stop w32time && net start w32time
there is a NTP GPO that can be applied in AD that is very easy to configure.
ASKER
The suggestion by kevinhsieh to remove the Hyper-V time sync on the DC VM only has just put me back to where I was previously - the machines don't drift in time but I'm back to getting the following event appearing in the logs:
Log Name: System
Source: Microsoft-Windows-Time-Ser
Event ID: 50
Level: Warning
Description:
The time service detected a time difference of greater than 5000 milliseconds for 900 seconds. The time difference might be caused by synchronization with low-accuracy time sources or by suboptimal network conditions. The time service is no longer synchronized and cannot provide the time to other clients or update the system clock. When a valid time stamp is received from a time service provider, the time service will correct itself.
I'd like to stop this happening.
I'll work through the suggestion from Greg_Hejl but I'm fairly sure that I'd done all this in the first place and still got the warning event above.
I'll post back soon.
Log Name: System
Source: Microsoft-Windows-Time-Ser
Event ID: 50
Level: Warning
Description:
The time service detected a time difference of greater than 5000 milliseconds for 900 seconds. The time difference might be caused by synchronization with low-accuracy time sources or by suboptimal network conditions. The time service is no longer synchronized and cannot provide the time to other clients or update the system clock. When a valid time stamp is received from a time service provider, the time service will correct itself.
I'd like to stop this happening.
I'll work through the suggestion from Greg_Hejl but I'm fairly sure that I'd done all this in the first place and still got the warning event above.
I'll post back soon.
have you tried the suggestion I made earlier?
ASKER
I will try your suggestion, demazter. The only reason I'm putting it off is because I'd rather have a single machine using the external source. I'm not sure why though.
I'm not sure how to have the host sync with both an external source and the PDC. Can you advise?
What's confusing me is this persistent warning event that I mention above. I don't understand where it's coming from. When I previously had all machines set NOT to sync with hyper-V and had the DC using ntp.org I still got this event. I'm fairly sure if I follow your suggestion that I'll still get this event even if all system clocks are correct.
I'm not sure how to have the host sync with both an external source and the PDC. Can you advise?
What's confusing me is this persistent warning event that I mention above. I don't understand where it's coming from. When I previously had all machines set NOT to sync with hyper-V and had the DC using ntp.org I still got this event. I'm fairly sure if I follow your suggestion that I'll still get this event even if all system clocks are correct.
>>>I'm not sure how to have the host sync with both an external source and the PDC. Can you advise?
You don't. You sync the host and the PDC with the same time source. For example, I have a network with 6 ESXi hosts. All of them sync'd with a GPS NTP Server in their own right.
The PDCe that is also on one of those hosts is also Sync'd with the time server. All of the windows clients then use the PDCe by default (as long as sync with host is turned off).
The benefit of this is that at bootup they get the same time they get when the time service starts.
I had an issue recently, where PDCe and all DC's had the correct time. Exchange Servers all had the correct time but the Exchange Servers were reporting their time was incorrect.
Because one of the hosts was 2 minutes out from the other and at bootup before the time service changed it the exchange server were 2 minutes out so failed to start their services.
What I am trying to say, is if you sync the host then you don't get "drift" and the bootup time of your DC is the same as when the time service finally kicks in and sync's with the external source.
You don't. You sync the host and the PDC with the same time source. For example, I have a network with 6 ESXi hosts. All of them sync'd with a GPS NTP Server in their own right.
The PDCe that is also on one of those hosts is also Sync'd with the time server. All of the windows clients then use the PDCe by default (as long as sync with host is turned off).
The benefit of this is that at bootup they get the same time they get when the time service starts.
I had an issue recently, where PDCe and all DC's had the correct time. Exchange Servers all had the correct time but the Exchange Servers were reporting their time was incorrect.
Because one of the hosts was 2 minutes out from the other and at bootup before the time service changed it the exchange server were 2 minutes out so failed to start their services.
What I am trying to say, is if you sync the host then you don't get "drift" and the bootup time of your DC is the same as when the time service finally kicks in and sync's with the external source.
ASKER
Ok, that makes sense. I'll give that a go and post back.
Thanks.
Thanks.
ASKER
Here's what I've done:
Thanks
Disabled Hyper-V time sync on all VMs.
Ran the following command on HOST1
w32tm /config /manualpeerlist:"0.pool.nt
Restarted Windows Time service. Checked logs - sync with ntp.org ok.
Ran the same command above on DC1. Restarted Windows Time service. Checked logs - sync with ntp.org ok.
Ran the following command on FS1
w32tm /config /syncfromflags:domhier /update
Restarted Windows Time service. Checked logs - sync with DC1 ok.
I'll check back shortly and post results.Thanks
the results will be:
in the next few weeks you will discover clients using the hyperv time sync will drift off.
then you will go through all your vm's and turn off hyper v time sync and run the domheir command or set a GPO for all your machines (not the DC's)
good luck
in the next few weeks you will discover clients using the hyperv time sync will drift off.
then you will go through all your vm's and turn off hyper v time sync and run the domheir command or set a GPO for all your machines (not the DC's)
good luck
Why would the clients be using the Hyper-V time sync? All windows clients by default will use the PDCe to sync time.
when you create a VM and install windows OS it changes registry settings for time source
turning off hyper v time sync disables these registry settings, running the domheir command sets them back to default
turning off hyper v time sync disables these registry settings, running the domheir command sets them back to default
ASKER
I don't understand. I've already disabled the hyper-v time sync for all guest OS's.
Is Host1 running as a DC?
you shouldn't have two NTP Servers in your environment
if host1 is just running hyper v - it just needs the client settings
you shouldn't have two NTP Servers in your environment
if host1 is just running hyper v - it just needs the client settings
Why do you think there are 2 NTP servers? 1 Host syncing to external NTP and 1 DC syncing to external NTP. All other VM's should automatically sync with the PDCe. This is default behaviour in a Windows network.
ASKER
No, host1 is just a member server that happens to be running the hyper-v role. What you're suggesting is how I originally had the environment configured (no hyper-v sync, pdc using external ntp.org as source, all others using pdc as source). This worked without any noticeable drift but kept throwing the warning event I've mentioned above.
It's that warning event that I'm really trying to stop.
It's that warning event that I'm really trying to stop.
_KeepItSimple_
For both the Hyper-V and PDCe use the instructions in the bottom half of this document to configure the external time sync: http://support.microsoft.com/kb/816042
You shouldn't need to do anything else.
Sync Hyper-V host with external NTP
Sync PDCe with the same external NTP
All other Windows devices will automatically sync with the PDCe
For both the Hyper-V and PDCe use the instructions in the bottom half of this document to configure the external time sync: http://support.microsoft.com/kb/816042
You shouldn't need to do anything else.
ASKER
Demazter, that's what I have done. I'm letting it run for a while to see if all is well.
the command you ran on host1 makes it a NTP server and it will advertise SNTP on the network that it is a reliable time source
you should have only 1 NTP server in your environment
you should have only 1 NTP server in your environment
ASKER
Although time is no longer noticeably drifting on the network, servers HOST1 and FS1 continue to report the following event:
The DC1 server seems ok - no warnings.
This doesn't make sense to me. Where is the time difference coming from?
I'll now try going through the steps given by Greg_Hejl in post 37530001 above. However, I'm fairly sure the outcome will be the same.
Log Name: System
Source: Microsoft-Windows-Time-Service
Event ID: 50
Level: Warning
User: LOCAL SERVICE
Description:
The time service detected a time difference of greater than 5000 milliseconds for 900 seconds. The time difference might be caused by synchronization with low-accuracy time sources or by suboptimal network conditions. The time service is no longer synchronized and cannot provide the time to other clients or update the system clock. When a valid time stamp is received from a time service provider, the time service will correct itself.
The DC1 server seems ok - no warnings.
This doesn't make sense to me. Where is the time difference coming from?
I'll now try going through the steps given by Greg_Hejl in post 37530001 above. However, I'm fairly sure the outcome will be the same.
How often do you see this event?
ASKER
It occurred on FS1 10 times yesterday. The interval between events varied from 1.5 hours to 3.25 hours. However, most were around 1.75 hours apart.
Sounds like it's struggling to sync with the external time source.
ASKER
The only server that uses an external time source is DC1 and it seems ok. FS1 uses DC1 as its source.
I should have mentioned that HOST1 shows these events less frequently - none yesterday and only twice the day before.
The issue really seems to be between the member servers and the PDCe (DC1).
I should have mentioned that HOST1 shows these events less frequently - none yesterday and only twice the day before.
The issue really seems to be between the member servers and the PDCe (DC1).
on the member server in REGEDIT.
Navigate to HKEY_LOCAL_MACHINE\SYETEM\
The type entry should be set to NT5DS
Navigate to HKEY_LOCAL_MACHINE\SYETEM\
The type entry should be set to NT5DS
ASKER
That's correct on both HOST1 and FS1.
Did you not configure host1 to sync with the outside time source as per my previous posts?
ASKER
Yes but I changed the configuration within the last 30 minutes or so to try Greg_Hejl's suggestion.
OK.
I will leave you to Greg_Hejl then.
I will leave you to Greg_Hejl then.
ASKER
Don't be offended - I'm just trying all suggestions offered. Your help is greatly appreciated.
I'll leave it for 24 hours as is and see what happens.
Thanks.
I'll leave it for 24 hours as is and see what happens.
Thanks.
Not offended at all :)
It's just very difficult in situations like this when multiple experts are posting different solutions.
I have a solution I know works on client sites and it's never failed me which is why I suggested it to you.
You have gone down a different road attempting a solution by a different person which in my opinion will not resolve your issue.
I therefore have no further input to offer you.
It's just very difficult in situations like this when multiple experts are posting different solutions.
I have a solution I know works on client sites and it's never failed me which is why I suggested it to you.
You have gone down a different road attempting a solution by a different person which in my opinion will not resolve your issue.
I therefore have no further input to offer you.
ASKER
As predicted, this has got me back to square one. There is no noticeable time drift on the network but I continue to get Time-Service warnings in the event logs. These appear occasionally on HOST1 (3 in the last week) but more frequently on FS1 (53 times in the last week) and DC1 (6 times in the last week).
These events are always the same - Time-Service event 50. The full description is posted above.
It's very frustrating as the network time seems to be ok other than these pesky warnings in the event logs.
These events are always the same - Time-Service event 50. The full description is posted above.
It's very frustrating as the network time seems to be ok other than these pesky warnings in the event logs.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
ASKER
I think that's what I'm going to have to do. Annoying though as I'm the kind of guy that likes to keep my event logs clean.
Thanks for your help.
Thanks for your help.
This question has been classified as abandoned and is closed as part of the Cleanup Program. See the recommendation for more details.
They should be set to NOT sync with the VM host.
However, I have noticed in the past few weeks with both VMWare and Hyper-V that this doesn't always ring true and that the host time can still effect the VM time even though it is set to not sync.
I would recommend the following:
-> Sync the host with the external time source AS WELL
-> Set ALL VM's to NOT Sync with Hyper-V
-> Set the PDCe to sync with a reliable time source.