Solved

Experts knowledge about task scheduler internals needed

Posted on 2013-05-28
31
1,064 Views
Last Modified: 2013-06-12
Hi experts.

On the network I administer, I rarely come about problems with windows' task scheduler. But if I do, it's very annoying... the following has occurred: I noticed a task simply stopped to work from one day to the next, that is: it stopped to follow the schedule.
History: empty since the last successful start
Next run: empty (although it is set to run every day!)
???
->recreated the task, it worked for two days, on the third day it broke again.

This happened on different computers, different OS, different tasks, different executing accounts (normally: system) - no similarities. Several other tasks on the same computers keep working in the meanwhile.

Question: Since logging (=task history) breaks, too, what possible ways can I use to shed light on this? What lowlevel approaches apart from procmon could I use?
0
Comment
Question by:McKnife
  • 14
  • 7
  • 3
  • +4
31 Comments
 
LVL 9

Expert Comment

by:MHMAdmins
Comment Utility
Try to create a service account with administrative privaleges and run the task scheduler to see if it runs or not. Also make sure the accounts are not locked out when the task scheduler is running as I've seen this happen as well. You also may want to verify the recovery tab settings for the task scheduler service and make sure it's set to restart.
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
Hi.
Service accounts are not possible before 2008 R2, this domain is at 2008.
The accounts are usually system and of course not locked out.
Thing is: history is empty - that means, task scheduler is not even aware that it had anything to do. Also, as we monitor services all the time, we would notice any service failure immediately. Also please imagine what would happen if the service went down... all tasks would fail. As I told you, it only affects single tasks.
0
 
LVL 9

Expert Comment

by:MHMAdmins
Comment Utility
I would assume the server in question is patched up to current update status?
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
Yes. And apart from that, it shows no anomalies. Please note that it's not one server but win7, 2008, 2008 R2 and vista - all do it, though very, very rarely.
0
 
LVL 68

Expert Comment

by:Qlemo
Comment Utility
Doing changes, like setting inactive and back to active, does not reschedule?
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
Yes they do, but after one or two days they start to bug again.
0
 
LVL 68

Expert Comment

by:Qlemo
Comment Utility
Just a guess: Maybe the time is corrected in big amounts occasionally, and the scheduler gets out of sync. That can't happen with NTP, which does only small adjustments, but if time sync runs per domain controller and the "older" time service (W2000), or only gets sync'd once a day or on log on, time might jump back and forth.
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
No luck with that. And please note, other daily running tasks don't have that problem.
0
 
LVL 68

Expert Comment

by:Qlemo
Comment Utility
Whether the time skews have an effect depends on when they are corrected, and if the correction spans the trigger time of tasks.

Run a simple test task doing nothing (but logging, maybe) every minute. That should show if there is a generic issue with the scheduler, as you assume.
Sadly, you do not have any means to debug the scheduler directly.
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
That will not do. All other tasks run well, it's not that I can not run tasks. It's that I can't be sure that my tasks run in the future. I need to get insight on its internal mechanisms, everything else is playing around.

Last bit of playing was setting the task in question to repeat every minute - it worked on and on! But I am sure, if I reset it to daily, it will again cease to reschedule tormorrow or the day after.
0
 
LVL 68

Expert Comment

by:Qlemo
Comment Utility
I assume you unchecked any "force stop" options in the Settings tab of the task?
As already said, there are no means to debug - besides using a kernel debugger, of course. So all you have are indirect approaches. Even ProcMon won't help you (much) here.
0
 
LVL 38

Expert Comment

by:Rich Rumble
Comment Utility
What makes this task special? Is it a certain program it's calling? If it's this certain task that is eventually failing, it seems related to the application it's trying to run rather than task scheduler.. Can you run the program/script/app "by hand" and have it work consistently? I'd make sure of that if you haven't already.
-rich
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
No, there's no force stop set. Thanks for flagging.
0
 
LVL 69

Expert Comment

by:Merete
Comment Utility
Any event errors we can check?
The Task Scheduler service has several dependencies>
service task schedAny other background services set to run such disk defrager?
Anti virus interfering?
Add the task to exceptions?
If you could explain the task would help pin point it a bit better
0
 
LVL 23

Expert Comment

by:ComputerTechie
Comment Utility
We stop using task scheduler and use this now for the same issues.

http://www.freebyte.com/fbtaskscheduler/

CT
0
Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

 
LVL 27

Expert Comment

by:Steve
Comment Utility
as you advise this is occurring on multiple machines, and that other tasks on the same machines are OK, you have already narrowed this down quite a bit.

The only likely causes left are:
A) The task itself is the issue:
     What does the task do?
     How do you distribute it? manual setup or centralised rollout?
     does it run OK if you run it manually?
    Does the task produce any event logs around the time its supposed to run?
B) The time is the issue
    Could you try setting it to run at a very different time to see if it runs and reruns OK?
    Are there any other tasks running at the time this runs? are backups running around the time it fails?
    Do any of the working tasks run at a similar time to the one with the issue?
    can you deliberately set the time on a machine to be wrong and monitor to see if the task runs?
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
> What makes this task special? Is it a certain program it's calling
Nothing special and it is not failing. It is just not rescheduling.
> Any event errors we can check?
No errors.
> as you advise this is occurring on multiple machines, and that other tasks on the same machines are OK...
not quite. At the moment, there is only one machine with that task being a problem. But there have been others with totally different tasks before. Same problem: rescheduling does not occur after it rescheduled for months.
> We stop using task scheduler and use this now for the same issues...
I am trying to avoid that as the task scheduler is a core component and has to work.
0
 
LVL 38

Expert Comment

by:Rich Rumble
Comment Utility
Some programs can't be run by NT/system, and have to use a user account. The System profile may work once, but then it may stop, i don't remember why that is though. I'd make sure you're giving the whole path to the exe and not relying on default paths. Depending on the OS, have a look at the "Start in" setting too.
-rich
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
Let me try to get our focus on something else, because again and again, it is not that the task is not being executed - the task even ran for months - it's that it someday somehow stops to reschedule and therefore the scheduler does not even try to execute it any more.

I hoped to get closer to the internals of scheduler. What components are involved? How does rescheduling work? What is being read while rescheduling and how could that fail? How could I monitor the failure? I don't think "you do not have any means to debug the scheduler directly" is perfectly true. Nearly anything in windows can be given a closer look. Maybe not so many people have tried with scheduler because normally, it is very reliable.
I will start another thread in a microsoft forum and keep you updated.
0
 
LVL 69

Expert Comment

by:Merete
Comment Utility
There are Microsoft qualified experts here at EE could be away for the week end :)
Has there been any viruses detected on this system ?
there is some bugs with it, as shown in the feedback comments here
Task Scheduler
http://msdn.microsoft.com/en-us/library/windows/desktop/aa383614(v=vs.85).aspx
Task Scheduler Reference
http://msdn.microsoft.com/en-us/library/windows/desktop/aa383608(v=vs.85).aspx
And this from Wiki
Bugs
On Windows 2000 and Windows XP, tasks assigned to run with SYSTEM privileges do not function when the computer is prepared for disk imaging with sysprep. Sysprep changes security identifier (SID) to avoid duplication but does not update scheduled tasks to use the new SID.
Consequently, all SYSTEM scheduled tasks fail to run on the imaged computers.
There is no solution for this problem but one may reschedule the tasks to work around the issue.

On Windows Vista or Windows Server 2008, where Service Pack 2 is not installed, repeating scheduled tasks may be repeated incorrectly and the next execution time displayed in Task Scheduler may be wrong.
Microsoft does not elaborate how exactly wrong is this repetition, but a patch is available for this bug.

On Windows Vista, 7, 2008, and 2008 R2: The MMC Component says that you are running "Task Scheduler 1.0" when in fact you are running 2.0, this is a Trivial bug so it wasn't noticed, and is likely due to the re-write of the task scheduler.
The Version has been corrected to 2.0 in Windows 8, and 2012
http://en.wikipedia.org/wiki/Windows_Task_Scheduler#Bugs
Look forward to your updates from Microsoft
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
Making only little progress here: like merete quoted wikipedia, task scheduler in win vista pre SP2 has known bugs with rescheduling. We have SP2 installed, this should not be the issue.

But: as I could verify, on vista and windows 7, if you setup 2 daily triggers, let's say one at 02:00 pm and one at 01:59 pm, the GUI shows the next run to be 02:00 pm although it does indeed execute the next run at 01:59 - this does not occur if you setup the 01:59 one first. So this is a still bug in vista, win 7 even, but not in win8, where it finally got corrected.
[Another off-topic bug: version info of the task scheduler show 1.0 in vista, in 7 and in 8/2012 server - all wrong, never got corrected, wikipedia is wrong here.]

So we can see there are indeed bugs with rescheduling still around. I will do some more tests and then contact MS.
0
 
LVL 69

Expert Comment

by:Merete
Comment Utility
How are you testing McKnife
If I may, I read up about this and thought it might help
Navigate to the General Tab of the Scheduled Task and select "run with highest privleges".  the user account is on SYSTEM and use a command that allows authentication parameters to be passed in the code such as the PowerShell cmdlet Start-BitsTransfer.
Also, fill in the correct start in path or arguments as necessary under the edit actions tab.
 use a .bat file to kick it off,  so fill in the start in box.
Workaround:
Navigate to the General Tab of the Scheduled Task and select "Run only when user is logged on". Now the scheduler will execute the .ordertest scheduled task.
This of course requires the user to be logged on but at least the .bat will execute on schedule.
Source
http://stackoverflow.com/questions/10757473/task-scheduler-not-executing-batch-bat-file-with-mstest-commands

Two Minute Drill: Quickly test Task Scheduler
http://blogs.technet.com/b/askperf/archive/2011/06/10/two-minute-drill-quickly-test-task-scheduler.aspx << included with this page at the bottom>
Troubleshooting Task Scheduler
Applies To: Windows 7, Windows Server 2008 R2, Windows Server 2012, Windows Vista
http://technet.microsoft.com/en-us/library/cc721846.aspx
And finally this one
Project Description
This project provides a single assembly wrapper for the 1.0 and 2.0 versions of Task Scheduler found in all Microsoft operating systems post Windows 98. It simplifies the coding, aggregates the multiple versions, provides an editor and allows for localization support.
Task Scheduler Managed Wrapper
http://taskscheduler.codeplex.com/
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
Hi Merete. Please be aware that this is a bug, it's not being used incorrectly. Those tests and troubleshooting thints don't apply.

I will have a look at the wrapper, though, to see wether it recognizes some tasks as being malformed.
-----------------------
I did further tests and found out very interesting things: if I delete the problematic task and recreate it with another name, it runs without problems and reschedules correctly, even if I restart the  computer. If, however, I choose to use the old tasks name, it does work too as long as I don't restart the computer. I confirmed that with both problematic tasks: if I restart it, the task with the new name still works, the one with the old name is broken. The problem also appears if I recreate it with another name, export it to the old name and reimport it - the reimported one is not rescheduling after a restart.

So this scheduler is a big mess, if you ask me. For those of you, who never ran into such problems, it might be reliable, yes.

What I will do next is script something to show me on how many computers we have tasks that are not rescheduling. If more than one or two, I will open a case with MS, otherwise, I will just work around it.
0
 
LVL 68

Expert Comment

by:Qlemo
Comment Utility
Is that name in any way special? Like containing special chars, having a specific length, ...?
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
Nothing special, no blanks or special characters, length below 10 chars.
0
 
LVL 68

Expert Comment

by:Qlemo
Comment Utility
So changing one arbitrary char, or appending one, is all you need to do to have it working for all times? Really strange ...
0
 
LVL 53

Author Comment

by:McKnife
Comment Utility
My script showed so far that there are only two computers network wide with this phenomenon. Sadly, trying to work around it the same way simply did not work out. It is even the same task... perfectly the same (task deployed by policy, same client OS). Recreating it using a different name this time does not survive a reboot... :(

Will further investigate. As I have no support contract for these computers, I cannot open a case with MS. Their forum will be my last resort.
0
 
LVL 53

Accepted Solution

by:
McKnife earned 0 total points
Comment Utility
Ok, now this is almost solved, no further comments needed.
I noticed that rescheduling times ("next run") disappear after a restart. What happens on a restart? The scheduler service restarts. So nothing special about the restart, I noticed the problem also occurs if I chose to restart the service called "schedule" (net stop schedule & net start schedule, executed as system account). And what does that service do on restart? I reads through the files in c:\windows\system32\tasks and through the registry entries below HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule

...and at least one of those was corrupt causing the rescheduling of new, perfectly healthy items to fail! I found out, when I deleted the keys HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Schedule\TaskCache\Tree, ...\Boot, ...\Logon, ...\Plain and so on. Newly created tasks, whatever their name might be, have no problem at all afterwards.

So what do we have here? We have task scheduler tasks that are corrupted - no problem with that. But why would those tasks influence only some, but not all other tasks?  Why would (on some computers ) a simple rename solve the problem? Why not on others? What programming art makes this possible :) ?

Finally, yes, I know articles like this http://www.ghacks.net/2011/07/28/fix-the-task-image-is-corrupt-or-has-been-tampered-with-errors/ where people found out how to resolve corrupted tasks problems... thing is, at the linked article, task scheduler directly warns you. So upon executing, it sees corrupted tasks and tells you - why not for me? How complicated can these tasks and this buggy scheduler software be that it does indeed discover some formats and tells you "corrupted" but not others? How would we ever know there are corrupted tasks? Gee....

So my final step will be to export/import these task cripples to a computer that has a software insurance and therefore MS support and have them sort this stuff out. They will need to go through dozens of tasks and analyze which one is the culprit and what software bug is responsible for turning a single corrupt task into such a show stopper.

I'll spread the news.
0
 
LVL 68

Expert Comment

by:Qlemo
Comment Utility
Regarding why renaming might help: The registry is organized as binary tree for fast access. Any change on a key (not sure about value names) can change the way the info is stored, and what is stored on the same memory page.
0
 
LVL 53

Author Closing Comment

by:McKnife
Comment Utility
End of story:
As the two systems we are experiencing it on right now are both vista and we don't have a single support agreement for vista running, I thought, well, let's move the tasks over to win8 enterprise. After verifying that I could import the corruption from vista to another vista system, I imported all task files and registry entries to win8 to make the whole task tree look like vistas. Well, win8 swallows it and does not complain AND the task in question IS rescheduling. That made me think the tasks are not corrupt but vista is.

I am pretty sure I found out, what was going on (attention, long but interesting story to follow):
Some time ago, we wanted to deploy a task network wide. Using the DC which was 2008, we could only deploy xp style tasks, not vista style tasks that have more options. But with RSAT from win7, we are able to create such vista style deployable tasks! ... or so I thought. Deploying that one task failed and even more, it corrupted a dozen of vista workstations' tasks! After you opened task scheduler on those machines, you were confronted with messages like "catastrophic failure". So we looked up all corrupt tasks, found out that those were all built-in and were able to recreate those. Case solved... or so I thought. I didn't even mention this incident here, as it's been 3 years ago and until now we did not have any problems.
But as it seems, the higher-version-tasks when deployed that way cannot be swallowed by vista and 2008 although they should. And although all visible error could be corrected, there had still been invisible ones.

Lessons learned: bugs in task scheduler, bugs in task deployment via Group policy preferences (GPP). Never ever try to deploy a vista generation task to vista via GPP, it simply explodes. Use a startup script command line with schtasks.exe /create /xml yourtask.xml /TN taskname instead. :)

And although we had this problem with not only vista, with others, it was sufficient to recreate the task, so this proved to be two different problems!

Thanks folks, closing.
0

Featured Post

IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

Join & Write a Comment

Article by: Lee
Windows 7 Ultimate and Enterprise (and 2008 R2) introduced a new feature you may not be aware of - Boot from VHD.   Boot from VHD (or what Microsoft refers to asNative Boot allows you to install Windows to a VHD (Virtual Hard Disk) file that is t…
You might have come across a situation when you have Exchange 2013 server in two different sites (Production and DR). After adding the Database copy in ECP console it displays Database copy status unknown for the DR exchange server. Issue is strange…
This tutorial will walk an individual through the steps necessary to join and promote the first Windows Server 2012 domain controller into an Active Directory environment running on Windows Server 2008. Determine the location of the FSMO roles by lo…
This Micro Tutorial will give you basic overview of the control panel section on Windows 7. It will depth in Network and Internet, Hardware and Sound, etc. This will be demonstrated using Windows 7 operating system.

763 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

7 Experts available now in Live!

Get 1:1 Help Now