Experts knowledge about task scheduler internals needed
Hi experts.
On the network I administer, I rarely come about problems with windows' task scheduler. But if I do, it's very annoying... the following has occurred: I noticed a task simply stopped to work from one day to the next, that is: it stopped to follow the schedule.
History: empty since the last successful start
Next run: empty (although it is set to run every day!)
???
->recreated the task, it worked for two days, on the third day it broke again.
This happened on different computers, different OS, different tasks, different executing accounts (normally: system) - no similarities. Several other tasks on the same computers keep working in the meanwhile.
Question: Since logging (=task history) breaks, too, what possible ways can I use to shed light on this? What lowlevel approaches apart from procmon could I use?
On the network I administer, I rarely come about problems with windows' task scheduler. But if I do, it's very annoying... the following has occurred: I noticed a task simply stopped to work from one day to the next, that is: it stopped to follow the schedule.
History: empty since the last successful start
Next run: empty (although it is set to run every day!)
???
->recreated the task, it worked for two days, on the third day it broke again.
This happened on different computers, different OS, different tasks, different executing accounts (normally: system) - no similarities. Several other tasks on the same computers keep working in the meanwhile.
Question: Since logging (=task history) breaks, too, what possible ways can I use to shed light on this? What lowlevel approaches apart from procmon could I use?
MHMAdmins
Try to create a service account with administrative privaleges and run the task scheduler to see if it runs or not. Also make sure the accounts are not locked out when the task scheduler is running as I've seen this happen as well. You also may want to verify the recovery tab settings for the task scheduler service and make sure it's set to restart.
ASKER
Hi.
Service accounts are not possible before 2008 R2, this domain is at 2008.
The accounts are usually system and of course not locked out.
Thing is: history is empty - that means, task scheduler is not even aware that it had anything to do. Also, as we monitor services all the time, we would notice any service failure immediately. Also please imagine what would happen if the service went down... all tasks would fail. As I told you, it only affects single tasks.
Service accounts are not possible before 2008 R2, this domain is at 2008.
The accounts are usually system and of course not locked out.
Thing is: history is empty - that means, task scheduler is not even aware that it had anything to do. Also, as we monitor services all the time, we would notice any service failure immediately. Also please imagine what would happen if the service went down... all tasks would fail. As I told you, it only affects single tasks.
I would assume the server in question is patched up to current update status?
ASKER
Yes. And apart from that, it shows no anomalies. Please note that it's not one server but win7, 2008, 2008 R2 and vista - all do it, though very, very rarely.
Doing changes, like setting inactive and back to active, does not reschedule?
ASKER
Yes they do, but after one or two days they start to bug again.
Just a guess: Maybe the time is corrected in big amounts occasionally, and the scheduler gets out of sync. That can't happen with NTP, which does only small adjustments, but if time sync runs per domain controller and the "older" time service (W2000), or only gets sync'd once a day or on log on, time might jump back and forth.
ASKER
No luck with that. And please note, other daily running tasks don't have that problem.
Whether the time skews have an effect depends on when they are corrected, and if the correction spans the trigger time of tasks.
Run a simple test task doing nothing (but logging, maybe) every minute. That should show if there is a generic issue with the scheduler, as you assume.
Sadly, you do not have any means to debug the scheduler directly.
Run a simple test task doing nothing (but logging, maybe) every minute. That should show if there is a generic issue with the scheduler, as you assume.
Sadly, you do not have any means to debug the scheduler directly.
ASKER
That will not do. All other tasks run well, it's not that I can not run tasks. It's that I can't be sure that my tasks run in the future. I need to get insight on its internal mechanisms, everything else is playing around.
Last bit of playing was setting the task in question to repeat every minute - it worked on and on! But I am sure, if I reset it to daily, it will again cease to reschedule tormorrow or the day after.
Last bit of playing was setting the task in question to repeat every minute - it worked on and on! But I am sure, if I reset it to daily, it will again cease to reschedule tormorrow or the day after.
I assume you unchecked any "force stop" options in the Settings tab of the task?
As already said, there are no means to debug - besides using a kernel debugger, of course. So all you have are indirect approaches. Even ProcMon won't help you (much) here.
As already said, there are no means to debug - besides using a kernel debugger, of course. So all you have are indirect approaches. Even ProcMon won't help you (much) here.
What makes this task special? Is it a certain program it's calling? If it's this certain task that is eventually failing, it seems related to the application it's trying to run rather than task scheduler.. Can you run the program/script/app "by hand" and have it work consistently? I'd make sure of that if you haven't already.
-rich
-rich
ASKER
No, there's no force stop set. Thanks for flagging.
Any event errors we can check?
The Task Scheduler service has several dependencies>
Any other background services set to run such disk defrager?
Anti virus interfering?
Add the task to exceptions?
If you could explain the task would help pin point it a bit better
The Task Scheduler service has several dependencies>
Any other background services set to run such disk defrager?Anti virus interfering?
Add the task to exceptions?
If you could explain the task would help pin point it a bit better
We stop using task scheduler and use this now for the same issues.
http://www.freebyte.com/fbtaskscheduler/
CT
http://www.freebyte.com/fbtaskscheduler/
CT
as you advise this is occurring on multiple machines, and that other tasks on the same machines are OK, you have already narrowed this down quite a bit.
The only likely causes left are:
A) The task itself is the issue:
What does the task do?
How do you distribute it? manual setup or centralised rollout?
does it run OK if you run it manually?
Does the task produce any event logs around the time its supposed to run?
B) The time is the issue
Could you try setting it to run at a very different time to see if it runs and reruns OK?
Are there any other tasks running at the time this runs? are backups running around the time it fails?
Do any of the working tasks run at a similar time to the one with the issue?
can you deliberately set the time on a machine to be wrong and monitor to see if the task runs?
The only likely causes left are:
A) The task itself is the issue:
What does the task do?
How do you distribute it? manual setup or centralised rollout?
does it run OK if you run it manually?
Does the task produce any event logs around the time its supposed to run?
B) The time is the issue
Could you try setting it to run at a very different time to see if it runs and reruns OK?
Are there any other tasks running at the time this runs? are backups running around the time it fails?
Do any of the working tasks run at a similar time to the one with the issue?
can you deliberately set the time on a machine to be wrong and monitor to see if the task runs?
ASKER
> What makes this task special? Is it a certain program it's calling
Nothing special and it is not failing. It is just not rescheduling.
> Any event errors we can check?
No errors.
> as you advise this is occurring on multiple machines, and that other tasks on the same machines are OK...
not quite. At the moment, there is only one machine with that task being a problem. But there have been others with totally different tasks before. Same problem: rescheduling does not occur after it rescheduled for months.
> We stop using task scheduler and use this now for the same issues...
I am trying to avoid that as the task scheduler is a core component and has to work.
Nothing special and it is not failing. It is just not rescheduling.
> Any event errors we can check?
No errors.
> as you advise this is occurring on multiple machines, and that other tasks on the same machines are OK...
not quite. At the moment, there is only one machine with that task being a problem. But there have been others with totally different tasks before. Same problem: rescheduling does not occur after it rescheduled for months.
> We stop using task scheduler and use this now for the same issues...
I am trying to avoid that as the task scheduler is a core component and has to work.
Some programs can't be run by NT/system, and have to use a user account. The System profile may work once, but then it may stop, i don't remember why that is though. I'd make sure you're giving the whole path to the exe and not relying on default paths. Depending on the OS, have a look at the "Start in" setting too.
-rich
-rich
ASKER
Let me try to get our focus on something else, because again and again, it is not that the task is not being executed - the task even ran for months - it's that it someday somehow stops to reschedule and therefore the scheduler does not even try to execute it any more.
I hoped to get closer to the internals of scheduler. What components are involved? How does rescheduling work? What is being read while rescheduling and how could that fail? How could I monitor the failure? I don't think "you do not have any means to debug the scheduler directly" is perfectly true. Nearly anything in windows can be given a closer look. Maybe not so many people have tried with scheduler because normally, it is very reliable.
I will start another thread in a microsoft forum and keep you updated.
I hoped to get closer to the internals of scheduler. What components are involved? How does rescheduling work? What is being read while rescheduling and how could that fail? How could I monitor the failure? I don't think "you do not have any means to debug the scheduler directly" is perfectly true. Nearly anything in windows can be given a closer look. Maybe not so many people have tried with scheduler because normally, it is very reliable.
I will start another thread in a microsoft forum and keep you updated.
There are Microsoft qualified experts here at EE could be away for the week end :)
Has there been any viruses detected on this system ?
there is some bugs with it, as shown in the feedback comments here
Task Scheduler
http://msdn.microsoft.com/en-us/library/windows/desktop/aa383614(v=vs.85).aspx
Task Scheduler Reference
http://msdn.microsoft.com/en-us/library/windows/desktop/aa383608(v=vs.85).aspx
And this from Wiki
Bugs
On Windows 2000 and Windows XP, tasks assigned to run with SYSTEM privileges do not function when the computer is prepared for disk imaging with sysprep. Sysprep changes security identifier (SID) to avoid duplication but does not update scheduled tasks to use the new SID.
Consequently, all SYSTEM scheduled tasks fail to run on the imaged computers.
There is no solution for this problem but one may reschedule the tasks to work around the issue.
On Windows Vista or Windows Server 2008, where Service Pack 2 is not installed, repeating scheduled tasks may be repeated incorrectly and the next execution time displayed in Task Scheduler may be wrong.
Microsoft does not elaborate how exactly wrong is this repetition, but a patch is available for this bug.
On Windows Vista, 7, 2008, and 2008 R2: The MMC Component says that you are running "Task Scheduler 1.0" when in fact you are running 2.0, this is a Trivial bug so it wasn't noticed, and is likely due to the re-write of the task scheduler.
The Version has been corrected to 2.0 in Windows 8, and 2012
http://en.wikipedia.org/wiki/Windows_Task_Scheduler#Bugs
Look forward to your updates from Microsoft
Has there been any viruses detected on this system ?
there is some bugs with it, as shown in the feedback comments here
Task Scheduler
http://msdn.microsoft.com/en-us/library/windows/desktop/aa383614(v=vs.85).aspx
Task Scheduler Reference
http://msdn.microsoft.com/en-us/library/windows/desktop/aa383608(v=vs.85).aspx
And this from Wiki
Bugs
On Windows 2000 and Windows XP, tasks assigned to run with SYSTEM privileges do not function when the computer is prepared for disk imaging with sysprep. Sysprep changes security identifier (SID) to avoid duplication but does not update scheduled tasks to use the new SID.
Consequently, all SYSTEM scheduled tasks fail to run on the imaged computers.
There is no solution for this problem but one may reschedule the tasks to work around the issue.
On Windows Vista or Windows Server 2008, where Service Pack 2 is not installed, repeating scheduled tasks may be repeated incorrectly and the next execution time displayed in Task Scheduler may be wrong.
Microsoft does not elaborate how exactly wrong is this repetition, but a patch is available for this bug.
On Windows Vista, 7, 2008, and 2008 R2: The MMC Component says that you are running "Task Scheduler 1.0" when in fact you are running 2.0, this is a Trivial bug so it wasn't noticed, and is likely due to the re-write of the task scheduler.
The Version has been corrected to 2.0 in Windows 8, and 2012
http://en.wikipedia.org/wiki/Windows_Task_Scheduler#Bugs
Look forward to your updates from Microsoft
ASKER
Making only little progress here: like merete quoted wikipedia, task scheduler in win vista pre SP2 has known bugs with rescheduling. We have SP2 installed, this should not be the issue.
But: as I could verify, on vista and windows 7, if you setup 2 daily triggers, let's say one at 02:00 pm and one at 01:59 pm, the GUI shows the next run to be 02:00 pm although it does indeed execute the next run at 01:59 - this does not occur if you setup the 01:59 one first. So this is a still bug in vista, win 7 even, but not in win8, where it finally got corrected.
[Another off-topic bug: version info of the task scheduler show 1.0 in vista, in 7 and in 8/2012 server - all wrong, never got corrected, wikipedia is wrong here.]
So we can see there are indeed bugs with rescheduling still around. I will do some more tests and then contact MS.
But: as I could verify, on vista and windows 7, if you setup 2 daily triggers, let's say one at 02:00 pm and one at 01:59 pm, the GUI shows the next run to be 02:00 pm although it does indeed execute the next run at 01:59 - this does not occur if you setup the 01:59 one first. So this is a still bug in vista, win 7 even, but not in win8, where it finally got corrected.
[Another off-topic bug: version info of the task scheduler show 1.0 in vista, in 7 and in 8/2012 server - all wrong, never got corrected, wikipedia is wrong here.]
So we can see there are indeed bugs with rescheduling still around. I will do some more tests and then contact MS.
How are you testing McKnife
If I may, I read up about this and thought it might help
Navigate to the General Tab of the Scheduled Task and select "run with highest privleges". the user account is on SYSTEM and use a command that allows authentication parameters to be passed in the code such as the PowerShell cmdlet Start-BitsTransfer.
Also, fill in the correct start in path or arguments as necessary under the edit actions tab.
use a .bat file to kick it off, so fill in the start in box.
Workaround:
Navigate to the General Tab of the Scheduled Task and select "Run only when user is logged on". Now the scheduler will execute the .ordertest scheduled task.
This of course requires the user to be logged on but at least the .bat will execute on schedule.
Source
http://stackoverflow.com/questions/10757473/task-scheduler-not-executing-batch-bat-file-with-mstest-commands
Two Minute Drill: Quickly test Task Scheduler
http://blogs.technet.com/b/askperf/archive/2011/06/10/two-minute-drill-quickly-test-task-scheduler.aspx << included with this page at the bottom>
Troubleshooting Task Scheduler
Applies To: Windows 7, Windows Server 2008 R2, Windows Server 2012, Windows Vista
http://technet.microsoft.com/en-us/library/cc721846.aspx
And finally this one
Project Description
This project provides a single assembly wrapper for the 1.0 and 2.0 versions of Task Scheduler found in all Microsoft operating systems post Windows 98. It simplifies the coding, aggregates the multiple versions, provides an editor and allows for localization support.
Task Scheduler Managed Wrapper
http://taskscheduler.codeplex.com/
If I may, I read up about this and thought it might help
Navigate to the General Tab of the Scheduled Task and select "run with highest privleges". the user account is on SYSTEM and use a command that allows authentication parameters to be passed in the code such as the PowerShell cmdlet Start-BitsTransfer.
Also, fill in the correct start in path or arguments as necessary under the edit actions tab.
use a .bat file to kick it off, so fill in the start in box.
Workaround:
Navigate to the General Tab of the Scheduled Task and select "Run only when user is logged on". Now the scheduler will execute the .ordertest scheduled task.
This of course requires the user to be logged on but at least the .bat will execute on schedule.
Source
http://stackoverflow.com/questions/10757473/task-scheduler-not-executing-batch-bat-file-with-mstest-commands
Two Minute Drill: Quickly test Task Scheduler
http://blogs.technet.com/b/askperf/archive/2011/06/10/two-minute-drill-quickly-test-task-scheduler.aspx << included with this page at the bottom>
Troubleshooting Task Scheduler
Applies To: Windows 7, Windows Server 2008 R2, Windows Server 2012, Windows Vista
http://technet.microsoft.com/en-us/library/cc721846.aspx
And finally this one
Project Description
This project provides a single assembly wrapper for the 1.0 and 2.0 versions of Task Scheduler found in all Microsoft operating systems post Windows 98. It simplifies the coding, aggregates the multiple versions, provides an editor and allows for localization support.
Task Scheduler Managed Wrapper
http://taskscheduler.codeplex.com/
ASKER
Hi Merete. Please be aware that this is a bug, it's not being used incorrectly. Those tests and troubleshooting thints don't apply.
I will have a look at the wrapper, though, to see wether it recognizes some tasks as being malformed.
-----------------------
I did further tests and found out very interesting things: if I delete the problematic task and recreate it with another name, it runs without problems and reschedules correctly, even if I restart the computer. If, however, I choose to use the old tasks name, it does work too as long as I don't restart the computer. I confirmed that with both problematic tasks: if I restart it, the task with the new name still works, the one with the old name is broken. The problem also appears if I recreate it with another name, export it to the old name and reimport it - the reimported one is not rescheduling after a restart.
So this scheduler is a big mess, if you ask me. For those of you, who never ran into such problems, it might be reliable, yes.
What I will do next is script something to show me on how many computers we have tasks that are not rescheduling. If more than one or two, I will open a case with MS, otherwise, I will just work around it.
I will have a look at the wrapper, though, to see wether it recognizes some tasks as being malformed.
-----------------------
I did further tests and found out very interesting things: if I delete the problematic task and recreate it with another name, it runs without problems and reschedules correctly, even if I restart the computer. If, however, I choose to use the old tasks name, it does work too as long as I don't restart the computer. I confirmed that with both problematic tasks: if I restart it, the task with the new name still works, the one with the old name is broken. The problem also appears if I recreate it with another name, export it to the old name and reimport it - the reimported one is not rescheduling after a restart.
So this scheduler is a big mess, if you ask me. For those of you, who never ran into such problems, it might be reliable, yes.
What I will do next is script something to show me on how many computers we have tasks that are not rescheduling. If more than one or two, I will open a case with MS, otherwise, I will just work around it.
Is that name in any way special? Like containing special chars, having a specific length, ...?
ASKER
Nothing special, no blanks or special characters, length below 10 chars.
So changing one arbitrary char, or appending one, is all you need to do to have it working for all times? Really strange ...
ASKER
My script showed so far that there are only two computers network wide with this phenomenon. Sadly, trying to work around it the same way simply did not work out. It is even the same task... perfectly the same (task deployed by policy, same client OS). Recreating it using a different name this time does not survive a reboot... :(
Will further investigate. As I have no support contract for these computers, I cannot open a case with MS. Their forum will be my last resort.
Will further investigate. As I have no support contract for these computers, I cannot open a case with MS. Their forum will be my last resort.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Regarding why renaming might help: The registry is organized as binary tree for fast access. Any change on a key (not sure about value names) can change the way the info is stored, and what is stored on the same memory page.
ASKER
End of story:
As the two systems we are experiencing it on right now are both vista and we don't have a single support agreement for vista running, I thought, well, let's move the tasks over to win8 enterprise. After verifying that I could import the corruption from vista to another vista system, I imported all task files and registry entries to win8 to make the whole task tree look like vistas. Well, win8 swallows it and does not complain AND the task in question IS rescheduling. That made me think the tasks are not corrupt but vista is.
I am pretty sure I found out, what was going on (attention, long but interesting story to follow):
Some time ago, we wanted to deploy a task network wide. Using the DC which was 2008, we could only deploy xp style tasks, not vista style tasks that have more options. But with RSAT from win7, we are able to create such vista style deployable tasks! ... or so I thought. Deploying that one task failed and even more, it corrupted a dozen of vista workstations' tasks! After you opened task scheduler on those machines, you were confronted with messages like "catastrophic failure". So we looked up all corrupt tasks, found out that those were all built-in and were able to recreate those. Case solved... or so I thought. I didn't even mention this incident here, as it's been 3 years ago and until now we did not have any problems.
But as it seems, the higher-version-tasks when deployed that way cannot be swallowed by vista and 2008 although they should. And although all visible error could be corrected, there had still been invisible ones.
Lessons learned: bugs in task scheduler, bugs in task deployment via Group policy preferences (GPP). Never ever try to deploy a vista generation task to vista via GPP, it simply explodes. Use a startup script command line with schtasks.exe /create /xml yourtask.xml /TN taskname instead. :)
And although we had this problem with not only vista, with others, it was sufficient to recreate the task, so this proved to be two different problems!
Thanks folks, closing.
As the two systems we are experiencing it on right now are both vista and we don't have a single support agreement for vista running, I thought, well, let's move the tasks over to win8 enterprise. After verifying that I could import the corruption from vista to another vista system, I imported all task files and registry entries to win8 to make the whole task tree look like vistas. Well, win8 swallows it and does not complain AND the task in question IS rescheduling. That made me think the tasks are not corrupt but vista is.
I am pretty sure I found out, what was going on (attention, long but interesting story to follow):
Some time ago, we wanted to deploy a task network wide. Using the DC which was 2008, we could only deploy xp style tasks, not vista style tasks that have more options. But with RSAT from win7, we are able to create such vista style deployable tasks! ... or so I thought. Deploying that one task failed and even more, it corrupted a dozen of vista workstations' tasks! After you opened task scheduler on those machines, you were confronted with messages like "catastrophic failure". So we looked up all corrupt tasks, found out that those were all built-in and were able to recreate those. Case solved... or so I thought. I didn't even mention this incident here, as it's been 3 years ago and until now we did not have any problems.
But as it seems, the higher-version-tasks when deployed that way cannot be swallowed by vista and 2008 although they should. And although all visible error could be corrected, there had still been invisible ones.
Lessons learned: bugs in task scheduler, bugs in task deployment via Group policy preferences (GPP). Never ever try to deploy a vista generation task to vista via GPP, it simply explodes. Use a startup script command line with schtasks.exe /create /xml yourtask.xml /TN taskname instead. :)
And although we had this problem with not only vista, with others, it was sufficient to recreate the task, so this proved to be two different problems!
Thanks folks, closing.