Link to home
Start Free TrialLog in
Avatar of daiwhyte
daiwhyteFlag for United Kingdom of Great Britain and Northern Ireland

asked on

Setting up Alerts

Hi,

I have a Windows 2003 server which runs two applications which are responsible for certain tasks which run every minute. Most of the time the tasks run without a hitch but every couple of weeks, the apps fails and displays an error message (the apps are VB.net apps)

I would like to setup an alert which captures these errors and alerts me imediately via email, how can this be done using windows alerts?

Thanks
Avatar of farazhkhan
farazhkhan
Flag of Pakistan image

Hi,

You have to setup a custom script which collects information for you and send mail. Then you can configure that script to schedule through scheuled task.

Regards,
Faraz H. Khan
Avatar of daiwhyte

ASKER

Ive already setup an email alert and just need to work out how to detect the error which the apps throws and run the email script. Need to know where how to detect the error??
Hi,

Well, if I am getting you corrcect, then you should get all kind of alerts/errors under Windows 2003 event viewer.

Regards,
Faraz H. Khan
Hello Dai

You say these tasks run every minute.

Is that to say that they run sequentially, or are they timed to run at separate times each minute?

How are you scheduling the tasks?  eg. Task Scheduler, VBScript running in background and looping as a timer, or some 3rd-party scheduling application?

What commands are each of the processes being launched using?  ie. calling an *.exe directly from the command line and with parameters, or via a scripted method such as a batch file that sends the parameters to the *.exe

If they are being called from a batch file or similar, then you may be able to (within the same script) query the return code when the process completes to test if the process completed with or without errors, and trigger your email alert if not.

If the processes, or the resultant errors thereof, are being logged to the event log, then you should be able to use a VBScript that uses WMI and query the event for the details of the error code either at a time in between the one minute events, or some similar method.

It might be possible to create a new custom event log specifically for your purpose using the EventCreate command, and have a looping VBScript monitor for a specific event code.  The script in the code snippet is an example from:
http://www.cruto.com/resources/vbscript/vbscript-examples/logs/eventlog/Monitor-Event-Logs-in-Real-Time.asp
that checks for EventCode 533.  That can obviously be modified to suit, as can the method of alert.

Without having an idea how the processes are being called, run, and logged, it's hard to make any solid suggestions.

Bill
 
strComputer = "."
Set objWMIService = GetObject("winmgmts:" _
    & "{impersonationLevel=impersonate, (Security)}!\\" & _
        strComputer & "\root\cimv2")

Set colMonitoredEvents = objWMIService.ExecNotificationQuery _    
    ("Select * from __instancecreationevent where " _
        & "TargetInstance isa 'Win32_NTLogEvent' " _
            & "and TargetInstance.EventCode = '533' ")

Do
    Set objLatestEvent = colMonitoredEvents.NextEvent
     strAlertToSend = objLatestEvent.TargetInstance.User _ 
         & " attempted to access DatabaseServer."
     Wscript.Echo strAlertToSend
Loop

Open in new window

Ive just checked the event log and nothing is logged in any of the sections when the application fails. Is there a way for Windows 2003 Server can just alert me when any errors appear on screen? Its worth noting that I have no scripting experience so go easy on me guys.
Hmmm.

I am not a programmer, but if you have any programmers within your company they may be able to compile an Event Watcher service:
http://www.codeproject.com/KB/cs/LogWatcherRSSFeeds.aspx
http://www.siccolo.com/Articles/CodeProject/LogWatcher_RSS_feed/build_log_watcher_rss_feed.html

I don't really know how you would intercept an error message on screen other than to have it flagged as a logged "Event", but even then you would need to be monitoring Event Logs in real time.

I assume you have read this page?:
http://technet.microsoft.com/en-us/library/cc779190(WS.10).aspx
The caveat with that "Notify Me of Missed Tasks" alert setting is that it only notifies you only of Task Scheduler's failure, not of individual missed tasks. Tasks that fail to run because of corrupt or missing executables DO NOT start off a notification.

I had wondered about creating a new custom Event Log purely for your couple of programs that you have scheduled.  You can periodically query the Event Logs using VBScript.  There is the EventCreate program in Windows (http://support.microsoft.com/kb/324145) but it only allows you to add your new event to the System or Application logs, not to its own log file.  It is possible to first create a new  Nevertheless, you can create custom events and then query them.

There is some discussion here about the concept:
http://serverfault.com/questions/47953/windows-event-log-email-notification
Amongst a couple of suggestions about monitoring software, a VBScript example is presented that uses the WMI "Win32_NTLogEvent" Class.

It is possible to create a new category in Event Viewer with its own Event Log file either using scripting, or possibly manually in Regedit.  To do it manually in Regedit, you would need to know what each of the registry keys hold, and what the values mean and do.  The root registry key containing all the Event Log settings is:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog
and there is some technial detail about it here:
http://technet.microsoft.com/en-us/library/cc776244(WS.10).aspx
(expand and view the sub-categories).

Have you tried creating custom events for logging your programs?

That might be the best approach. If you can at least get the programs' activities logged, then it gives a usable method for several possible approaches, or else you are stuck with workarounds.

You didn't say how the program was being launched via scheduled tasks, and it has a bearing on how errors could be trapped.  If it is launched by a batch file, then that batch file can be written to run the program, wait until the process completes, and then check the Return Code using the built-in variable named %ErrorLevel% that is stored on exit.  A Zero is success, and a 1 or other value signifies something else, depending on the program.

Scheduled tasks retains results in the file:
C:\WINDOWS\SchedLgU.Txt
but it's not very easy to get specific results from it.
I'm not sure if you are aware, but the status, results, etc of Scheduled Tasks can instead be queried on using the command SCHTASKS (C:\Windows\System32\schtasks.exe).

If you run the command    schtasks /?    from as command window it will show you the options available to you.  The /QUERY option is the one to use here.
The command    schtasks /query /?    shows the options below for a WinXP system.  I'm not sure if Server 2003 has any additional options:

SCHTASKS /Query [/S system [/U username [/P password]]] [/FO format] [/NH] [/V] [/?]

Parameter List:

/S     Remote system to connect to.
/U     User Name to execute command under.
/P     Password for user profile if required.
/FO  Output format. Valid values: TABLE, LIST, CSV.
/NH  Don't display column headers in TABLE and CSV formats.
/V     Verbose output.

To redirect screen output to a file, just add a   >   at the end of the command followed by the path to the report file (eg. report.txt or report.csv).

Run the following command to see the scheduled tasks on the local computer with "column headers" shown in long list view:
schtasks /query /fo list /v
or as double-quoted comma-separated fields in one continuous row for each scheduled task:
schtasks /query /fo csv /v
(or minus column headers:  schtasks /query /fo csv  /nh /v)

In CSV format you should see the "TaskName" and "Last Result" fields, which on my XP system are fields 2 and 6 respectively.  We can grab the values in those filelds out using a relatively simple one line command:

for /f "tokens=1-8 delims=," %a in ('schtasks /query /fo csv /nh /v ^| find /i "Your Task Name"') do @echo. & @echo Task: %~b, Last Exit Code: %~g

Broken down and explained (>>> "I have no scripting experience" <<<):

('schtasks /query /fo csv /nh /v ^| find /i "Your Task Name"')

Executes first. Gets a verbose CSV listing which would be shown on screen, but instead is redirected into the FIND command to search for ONLY the row containing "Your Task Name", which you would of course substitute for the real scheduled task's name.

The line is then processed by the FOR command:

for /f "tokens=1-8 delims=," %a in

which breaks the line into chunks between every comma, ie. comma-delimited "tokens" or pieces. Given that we start the tokens at %a, then each sequential chunk will be %b, %c, etc.  Note: I am looking to get the values in the fields "TaskName" and "Last Result" which should be fields 2 and 6 from these:
"HostName","TaskName","Next Run Time","Status","Last Run Time","Last Result","etc","etc"
however, the Date/Time values in fields 3 and 5 add in commas of their own to separate the Date and Time, so in actuality the values I need will be the 2nd and 8th comma-delimited "tokens".  Translating that to the starting variable %a, that means the chunks of data from the row will be %b and %g respectively.

The last part:

do @echo. & @echo Task: %~b, Last Exit Code: %~g

just does whatever you want with the data that has been stored in the variables %a and %g.  In this case I chose to just have it echo one blank line (@echo.) then echo some explanatory text and expand the values held in the variables.  The ~ in the variables %~b and %~g is just a trick to remove the double quotes.

Note:
If used in a batch file, every instance of % has to be doubled up, so %a would have to be %%a and %%~a, and %b would have to be %%b and %%~b, etc.

Note Also: The "Last Result" from a schtasks query is the Scheduled Tasks result, and NOT the error code of your program that failed.  Nevertheless, a Zero is still a success and a 1 or other non-zero value is NOT a success.
http://support.microsoft.com/kb/308558

Used in a batch file you can tailor the action dependent on the "Last Result" value, which is demonstrated below with echoed text to the screen.  The check      IF %~g NEQ 0    will only do something if the return code is Not Equal To (NEQ) to Zero.

for /f "tokens=1-8 delims=," %a in ('schtasks /query /fo csv /nh /v ^| find /i "Your Task Name"') do if %~g NEQ 0 @echo Task: %~b FAILED with Exit Code: %g at Last Run Time: %~f %~e

In a batch file that would be like this, which includes an IF TRUE and an IF FALSE output:

@echo off
for /f "tokens=1-8 delims=," %%a in ('schtasks /query /fo csv /nh /v ^| find /i "Your Task Name"') do (
    if %%~g NEQ 0 (
       echo.
       echo Task: %%~b FAILED with Exit Code: %%g at Last Run Time: %%~f %%~e
       echo.
       pause
    ) else (
       echo.
       echo No errors reported for Task: %%~b at Last Run Time: %%~f %%~e
       echo.
       pause
    )
)


OK, so that's the theory involved with querying SCHTASKS for the last result of your named task, but how can you use that to alert you?

Well, that's why I was curious to know if your program is actually being called from a batch file, and it is the batch file which is scheduled to run.  That would be easy, just add the additional content above and use a command line emailer to send you an alert if the process fails.

Otherwise, you could create such a check batch file and schedule that to run a very short time after your main scheduled task, and alert you on failure.  Given that your processes are scheduled to run every minute, it would be hard to determine how long each process takes and try to squeeze in a batch file to check the results.

Another method would be to have a Visual Basic Script running and looping, and doing very much what the example batch file would be doing, ie. checking last results.

You will see such a looping VBScript here:
http://serverfault.com/questions/33108/getting-alerted-when-a-windows-scheduled-task-fails

Say your *.vbs file was named "Check_Task_Result.vbs", once started with the command    CScript //NoLogo Check_Task_Result.vbs   it would keep running in the background.  That could be adapted, but I have to admit that VBScripting is not my strongest point.

Sorry if this seems pieced together.  Some of the ins and outs I have been reading are a bit over my head, so I've just talked about the methods that I do understand to a reasonable degree.  Hopefully you can get some useful information to work with.

Bill
Bill, a very thorough response, thank you very much. I too am not a strong coder by any means and this problem is causing me no end of grief since the program fails every couple of days and its a hour or two before manual checking picks it up. Maybe we could look at this another way?

Is it possible to detect how many windows are open on the server desktop at any one time? The reason I say this is the server is never touched by anyone. Once the starts up, 6 programs starts up thats it, they just run merrily for days. So when the server is in this state, we could run a vb script every 15 minutes to check how many windows are open, the result should always be 6, if not then I would like to call a batch file which is on the server (it generates an email to the support staff - I original used this when trying to setup alerts in the Alerts and Performance Monitor).

Thanks again for your indepth answer,

Dai
Hi Dai

The first and most obvious command I would look at running to check what program windows are open would be TASKLIST.  It has a lot of different options available which can be seen by typing the command   TASKLIST /?
The most useful of the options in your example would be the filter to look for a matching "WindowTitle" of running tasks.

The following command would check specifically to see if a window with "Calculator" (ie. the Windows calculator) in the title bar was running on the same computer.  If not, it would give a one line message that it was not, but if running it would provide a long comma-separated report of the various values for the running task:

tasklist /v /fi "WindowTitle EQ Calculator" /fi "Status EQ Running" /fo CSV /nh

/v = Verbose output
/fo CSV = separate values double-quoted and comma separated as "fields"
/fi "WindowTitle EQ Calculator" - Filter tasks only to windows with "Calculator" in title bar
/fi "Status EQ Running" - Filter results to running tasks only.

The results would be something a 2-line 9-field report like this (but I have created a blank line between the column header row and the matching values row):

"Image Name","PID","Session Name","Session#","MemUsage","Status","User Name","CPU Time","Window Title"

"calc.exe","3216","Console","0","240 K","Running","DC845WN\Bill","0:00:00","Calculator"

Without the /v switch, the results would be restricted to the following 5 results fields:

"Image Name","PID","Session Name","Session#","Mem Usage"
"calc.exe","3216","Console","0","1,036 K"

The  /nh  switch could be used to lose the column headings row in the results.

If the Windows calculator was NOT found to be running, the following single-line result would be issued:
INFO: No tasks running with the specified criteria.

There are a number of other useful filters that can be subjected to a criterion test with EQ, NE, GT, LT, GE, LE (respectively Equal to, Not Equal to, Greater Than, Less Than, Greater To or Equal To, and Less Than or Equal To):

STATUS - tests if Running or Not Responding
IMAGENAME - Tests for a matching program executable name
PID - Tests for a matching Process ID Number
SESSION - Session number
SESSIONNAME - Session name
CPUTIME - Up Time
MEMUSAGE - Memory usage in KB
USERNAME - Tests for tasks running under User name (in [domain\]user format)
SERVICES - Tests for matching Windows Service Name
WINDOWTITLE - The one exemplified earlier
MODULES - Tests for matching DLL name in running tasks.

The ImageName filter could also be used to test if the named program.exe had a status of "Running".

The example command could be used in a batch file to do certain things dependent on the results, and to do this we can break up the results line into comma-separated chunks and only use the ones we need.

An easy test would be to issue the command that provides verbose results, split the output into chunks, and test if one or more of the values held in those chunks matched.  To split the results into chunks (referred to as TOKENS), we use the   FOR   command.  The following batch file would assign each of the comma-separated values returned by the tasklist command to a variable name starting with %a and ending with %i and present them on separate lines:

@echo off

for /f "tokens=1-10 delims=," %%a in ('tasklist /v /fi "WindowTitle EQ Calculator" /fi "Status EQ Running" /fo CSV /nh') do (
    @echo %%a
    @echo %%b
    @echo %%c
    @echo %%d
    @echo %%e
    @echo %%f
    @echo %%g
    @echo %%h
    @echo %%i
)
pause

The 6th Token ie. the data value held in the variable %%f is the "Status" result, and if the Windows calculator is open the variable will contain "Running".  The following batch file demonstrates a test to see if %%f contains "Running":

@echo off

for /f "tokens=1-9 delims=," %%a in ('tasklist /v /fi "WindowTitle EQ Calculator" /fi "Status EQ Running" /fo CSV /nh') do set STATUS=%%~f

if "%STATUS%"=="Running" (
    echo Calculator is Open
    ) else (
    echo Calculator is Closed
   )
)
pause


Breaking the command down into its component parts:

The single-quoted TASKLIST command in parentheses is run first, resulting in the comma-separated single-line report without any column headers.

The FOR command is told to treat commas as field delimiters and look at all 9 fields.  The 6th of them will be the Status field which should be storing the double-quoted text string "Running".

The part on the other side of the TASKLIST command, ie.
do set STATUS=%%~f
just removes the double-quotes and sets the texts string as a new variable named %STATUS%.

The IF/ELSE block just does a case-sensitive test to see whether the variable named %STATUS% does contain the unquoted     Running     text string, and echoes the results.  Those   ECHO commands could be replaced with other commands that do whatever you need, such as:

if "%STATUS%"=="Running" (
    goto :EOF
    ) else (
    ** Insert command to email admin about failure here **
   )
)


In this example, if the Windows Calculator was running, the batch file would quit (goes to the End Of File) and close the command window again very quickly.  If Calculator was found not to be running it would execute the email alert command and then quit, as long as you removed the  PAUSE  at the end which is just there for testing.

To schedule such a batch file to run every 15 minutes, or at closer intervals,  would be easy enough to do using Task Scheduler.  My preference is an old and very small utility program named "schedule" written by a guy named Eric Phelps.  I am not sure though if it would run OK on anything past Windows XP.  You have the files  "schedule.exe" and "schedule.ini" in the same directory and manually edit the INI file with your settings which are explained fully in the Readme file.  Once launched, the program will continue to run until it its process is killed, but uses practically zero resources.

http://www.ericphelps.com/schedule/index.htm
http://www.ericphelps.com/schedule/schedule.zip

It's worth a look.

OK, so you wanted to check and see if there were 6 running programs on the server.

Note: If running TASKLIST from a computer other than the server, then use the command  TASKLIST /?  to see the options for searching running tasks on a remote computer.

You could perhaps do as I did above and use TaskList to seuentially check the status of the Window Titles of the 6 programs (assuming they do run with a GUI), OR you could instead use the  "ImageName" filter and extract the "Image Name" tokens out for testing against the known names of the executables, performing an IF/ELSE test on each and falling down through to the next test, or exiting with an error if one program wasn't running.  Maybe a bit complex though.

Another option would probably be possible but lengthier:

TASKLIST /v /fi "Status EQ Running" /fo CSV /nh

This would list (as CSV) all running programs.  It would be possible to whittle down those results to only your 6 programs using something like the FIND command and use the final results to count them, or to jump out to an error situation immediately one of the programs was not found in the final task listing.

Let me know what you think about any of the loose suggestions so far.

Bill
Hi Bill, Ive read through your extensive comments and I think we are on the right track but for me to complete this challenge, Im going to need a little more assistance if I may due to my coding skills are not strong at all.

I need to run my batch file if there is more than six windows open and not six programs. Ive had a go at using the tasklist but its not capturing the windows open, its only capturing the programs running, is there anyway I can use a command to detect how many windows are open?

I'm sorry daiwhyte and TheLearnedOne, but I kept forgetting to return to this question with an update.

To be honest, I hit a stone wall with this and intended to return and say so.  I knew how to perform the necessary tests from a batch file to satisfy the originally asked question - but only if the names of the programs or program windows were known and their status could be tested with TASKLIST.

Where I couldn't get anything to work was with the supplementary requirements asked, which involved only running a process if there were 6 program windows open.  It's because of the fact that my batch coding knowledge does have some deficiences when it gets more complicated.

I will try again and see if I can find a way of doing this, but if I haven't, then I'm sorry but I'll just need to let the automated closure (With points refund) run its natural course.
ASKER CERTIFIED SOLUTION
Avatar of BillDL
BillDL
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
Start Free Trial
Your right, a bit clunky but it does the job. Thanks for your efforts on  this one Bill.
Thank you Dai.

The separate FOR statements could probably be integrated into one single statement without the need to create a text file, but that's where I begin to get tangled up and find it harder to see where it isn't working.  A more experienced "DOS" batch programmer could make it smaller and neater, and a VBS method would probably be more fluent.  By the way, I forgot to remove the REM from in front of the last line, so the text file remains until the batch file creates it freshly with the next run.

Bill