[Last Call] Learn how to a build a cloud-first strategyRegister Now

x
?
Solved

Software or setting to kill high cpu utilization processes automatically

Posted on 2005-05-11
14
Medium Priority
?
2,176 Views
Last Modified: 2009-07-29
There are a couple of apps on my Citrix Server that go to 25% cpu utilization and just stay that way.
Is there a setting or software that monitors and kills processes that start to use too much cpu?

-Greg
0
Comment
Question by:gsgi
14 Comments
 

Accepted Solution

by:
mwrye earned 1400 total points
ID: 13980698
There is a software called T-Scale by RTO Software.  It allows you to set up scalability rules on your applications.  That way if one of the applications become rogue and try to take over too much resource on the server the application will scale it back.  It allows it to run but does not allow it to take up some much of the server's resources so that the other applications can run.

It does cost but was not too expensive.  We found that we dramatically reduced rogue (applications taking over the server) while still being able to utilize all the necessary apps.


http://www.rtosoft.com/products/TScale/TScale.html
0
 
LVL 13

Author Comment

by:gsgi
ID: 13981314
ok, i'll look at it.  My experience is that once an app has 25% of the cpu sustained
for 5 minutes, it is hung up and needs to be killed.

-Greg
0
 
LVL 9

Expert Comment

by:MrBillisMe
ID: 13981607
gsgi...there are processes that use resources if they are doing a paging task that might use 100% of CPU for extended periods, this particularly true when a server is involved with exchange, mailsweeps and bisql.
0
[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

 

Expert Comment

by:mwrye
ID: 13981613
Is it always the same app that is hanging? Or is it a variety of applications that are showing that behavior?

What version of Citrix are you on (XP, 1.8, etc.) and what OS?

I have several database applications that when running large processes try to take over the server.  They are not hung the application was just optimized in developement to use as much resources as it could get to complete the transactions.  These sometimes will run for hours and try to take as much of the available resources as they can the entire time.

I did have some problems at one point with 16 bit applications hanging with large resource utilization.  These also had a tendency to show as still active connections even though the user had actually logged off.  Citrix Hotfix XE102W081 resolved that problem.

There is also a reg key that you can enter to set timeouts for disconnected applications - are the sessions that are hung disconnected or are they still active?

Monica
0
 
LVL 13

Author Comment

by:gsgi
ID: 13981785
Hello mwrye, and MrBillisMe:

It is Citrix XPs FR3.  I have twenty sessions on a new dual xeon 3.0 w/ 3gb of ram and windows 2000.
Hmmm, I am going to look into that 16 bit stuff, maybe something is still in that mode.

The issue is that the vendor of the canned database we seek to run successfully hasn't
gotten around to replacing the old code with more efficient code in one of their apps.
This app, chart.exe, also calls another app viper.exe to show dental x-rays.  viper.exe
likes to hang up (I think) because it can't find its dongle, yes I wrote dongle!  Then
chart.exe is at 25% cpu.  It is easily killable, but the citrix session will not end without a
ctrl-esc and killing the citirx neighborhood with end task on the client.  Chart.exe remains
hung on the citrix box.  Also, another hand off, query.exe load ms word to print prescriptions.
When that hand off breaks down query.exe is at 25% of cpu just like chart.exe.
Unfortunately, unlike chart.exe at 25%, the same user can try to print the prescription
again, and if there is a query.exe at 25% (hung) for a user and they try again, the new
query.exe hangs again, also at 25%.  So 18 of my users are running fine, albeit slowly
using 25% of the system and two have hung up processes - chart.exe mostly, sometimes
query.exe.

There are also two other services running on the main sql server, a terminal server
only that at times spike to 50% each and are so far impossible to kill without a reboot.
One is viper and the other is a service that is looking for the MASTER superkey dongle
and then a separate superkey dongle licenses.
I've tried pskill from sysinternals and it failed to kill the pid.  

Thanks,
Greg
0
 

Expert Comment

by:mwrye
ID: 13982768
Hmmm...FR3 should have fixed the issues with the 16-bit apps (it included the Hotfix XE102W013 which should have corrected the way it ran 16-bits).

So the Query.exe you mention is not the Query.exe that Metaframe uses to query the farm - correct?  It is part of the Chart.exe application?

The only issue I know about with hanging at 25% is when the Binder is called in TS for Office Applications (your Word).  Any idea if the call to Word for your prescriptions calls the Binder tool?  Microsoft's solution is to not install the extra tools like this on Terminal Servers (Citrix) - known issues - no know fix that I know of.  I believe it also only applies to Office XP - are you running Office XP.

Defintely sounds like it is a hang - not a load issue.  

It sounds like the Chart and Viper programs are SQL.  What version of SQL is it running (2000, 7, Enterprise, Standard, etc).

If 2000 Enterprise - Have you tried running the SQL Profiler (or running the monitoring in Enterprise Manager) to see if the SQL is actually deadlocking or erroring out?  If it is deadlocked then it could sit there for a while before it will clear and the session would be still utilizing resources.

Monica
0
 
LVL 13

Author Comment

by:gsgi
ID: 13983596
Query.exe is definitely there thing I am almost positive that it used to query an x-base database and
they rewrote it to get the info out of sql 2000 standard server, which is what we use.

We are still using office 97.

I do have a sql enterprise running this for another client.  I have meant to profile it but just haven't
had time to learn the profiler yet.

-Greg


0
 

Expert Comment

by:mwrye
ID: 13985867
When you get time to set that up it can be very helpful in finding exactly what is locking.  It is graphical to setup the Trace - Open Profile, File - New Trace, General Tab - select where to put the file, Event Tab - I would select the Error and Locking areas, under Data Columns - select the info you want to see (I would capture Users, SPID, Objects, Start Time, Stop Time, Duration and of course any other data you want to see), Filters - I would set it to just your one database.

Capture this while things are running OK to create a least sort of a Baseline.  Then run it again periodically until you get the actual hung processes.  

Of course there can be some slight impact on processing speed while running the Profiler (same as with the System Manager piece of Windows).

There is a monitoring tool that I use on the Citrix Servers (this is not SQL related but system related) that shows much greater depth when monitoring than simply looking at the processes within Windows or Citrix.  We have found it to be very helpful and is only like $40 or so. It will actually allow you down to the Thread level on the process - maybe if you could kill the thread you could kill the whole process without having to reboot:

http://www.iarsn.com/taskinfo.html

The only other Software that I am aware of that will try to kill hung processes is:

http://www.softpedia.com/get/Internet/Servers/Server-Tools/EasyServ-Lite.shtml

I have never used it myself so I do not know how good it really is.  

I wonder if any software will be able to kill it automatically for you since you said you sometimes could not get it to kill when using the Windows utilities to try to kill it.

I will keep pondering the issues.  Sorry not able to give a definitive answer for your issue.

Monica


0
 
LVL 13

Author Comment

by:gsgi
ID: 13986701
I appreciate your help.  Since Chart.exe and query.exe on the Citrix box are easy
to kill once hung up, what is the best way for my users to recover.  Is their easiest
course of action to hit ctrl-esc on the client, bring up task bar on the client, and
kill the citrix ica?  Now with the citrix session finally ended, why isn't citrix cleaning
up the hung processes?  Can't I set a log off and terminate all their stuff setting?

Thanks,
Greg
0
 

Expert Comment

by:mwrye
ID: 13988520
What client are you running?  Web Client or the full ICA client?  
Are they using the Citrix Desktop or a single published App?
Is this the only application stuff they are running from the Citrix?  
Are you running NFuse or managed connections via the ICA Client?  
Are the Apps running in Seamless Mode or can they see the ICA Connection center?  

I have logged off hung applications in different ways - depending on the conection client.  

You can definetely do a Log Off of the Session in the Management Console which should kill the session on the user's machine.  Not sure if you can automate that - never seen that done - just right click the session and log off user.  That is a clean way to do it.

Are the sessions actually stopping in Citrix?  Not disconnected but reset?  Have you enabled the ICAEnableKeepAlive?  If you set this very high then sometimes unexpected disconnects will not kill the session on the Server (because if it tries to reconnect from a drop it can pick back up).  Mine is set high and when a user closes their connection other than gracefully (ending task on the computer and stuff like that) then I have to reset the Citrix Session to clear it out completely.

If the session no longer shows up and the process is still running on Citrix then it is probably because the hung application is still trying finish what it was doing.  Even after I reset the sessions through Citrix I occassionally have this happen or I will find that the connection between my Database and my Client is still active even after killing the session.  I then have to kill the user session between my DB and the Client (through the Database itself).

Most Databases that I work with split processing between the DB Server and the Client.  In the case of Citrix the client is of course the Citrix Server.  If the Client is trying to run something on it's local machine and gets hung it will continue to run until the process is killed or the server rebooted (which of course will kill the hung process).  That sometimes means that even though the Citrix Client is logged off - the process continues to run on the Citrix Server.  When that happens I have always just gone into Citrix and killed the process manually (or by batch script).  It is like they are in a debug mode which won't let me kill it but it is just looping.

The TaskInfo product allows me to kill the threads (maybe to specific DLLs that are hidden from the regular Citrix view) which may be stopping the processes from dying after the disconnect.

Sorry to ramble.  Hope this is what you were asking.  If not, then please let me know.

Monica



0
 
LVL 13

Author Comment

by:gsgi
ID: 13992703
We are using the citrix ICA client add connection thing.  It loads a main application (office.exe) which has a toolbar that lets them get to the other components of the dental database, chart.exe, famfile.exe and ledger.exe.   ledger.exe also used to hang at around 25% but they updated its code and it rarely crashes now.  So it is a matter of time before they similarly enhance chart.exe.  These programs will call up others needed, like word.

I am not on site everyday.  So the solution as to logging off people is going to need to be taught to a lay person.  Right now, I have them ctrl-esc then click the task bar on the client to bring up the task manager and kill the citrix ica session.  It is at this point, I'd like the server to go, "oh, this user is gone, ok i'll cream all running processes."

The sessions are fine.  chart.exe is hung and can not be used or ended.  The other programs are fine, like ledger.exe.  The ica goes away off course when all the programs that are started included office.exe end.  Since chart can not be ended, the ica session is effectively un-endable, so its stuck and must be killed from the client or from the citrix server itself.

 ICAEnableKeepAlive - what does this do?  I have not altered it from the default.

Once chart.exe goes to 25% it will stay like that for hours, it must be terminated on the server.

I do not think the database if suffering, but I'll look into it...  Its cpu util seems pretty constant and
unaffected by chart.exe or query.exe at 25% cpu and effectively hung up.

Ok, thanks for rambling.  I appreciate all of your input.  And I apologize for the delay in my response
but today was extremely hectic and I didn't want to rush a response.

-Greg





0
 

Expert Comment

by:mwrye
ID: 13997728
Have you tried having your user's do Ctrl-F1?  This should take them to the Task Manager of the published app they are in - they could try to kill the hung process themselves at that point (instead of killing the whole ICA connection).

There is a way to build a script using the Query Process to find processes that have been running at a high percent (say specified at 25%) for a set length of time.  I am not completely positive how to write that script.  I found the following information on the Citrix web site while looking for something else:

"You could use the dos command line command rkill the usage is rkill /nkill \\servername process.exeif you want to do it by process ID then you could userkill /view \\servername >list.txtthen pull the pid for each copy of the process running and pass it to the rkill command.

Only on Win2000 one would have to use "query process" to find all the PID's and then "tskill" to terminate them. I would however appreciate some guidance on how to the pull the PID's from the text file and pass it on to the tskill command.

Is it possible to forcibly disconnect users signed on to an application once the logon time expires? Posted: Oct 22, 2002 10:25 AM Rating: Not Rated    Click to rate       

If you put this line into a cmd script you should be ok.for /f "tokens=4" %%x in ('Query Process outlook.exe') do tskill %%xobviously replacing outlook with the name of the process you would like to kill.I hope this helped. "

I have a feeling this won't really help you though since I believe this is sort of what 'PSKills" is supposed to do - and if I am write it couldn't end the process in some cases so it might not work.

If I can piece together the script I will put it in here for you (but may be a couple of days till I can look at it).

Monica

0
 
LVL 13

Author Comment

by:gsgi
ID: 13998481
Thank you very much.  I will be away for a few days.
I appreciate all of your help very much.

-Greg
0
 

Assisted Solution

by:Wittmares
Wittmares earned 600 total points
ID: 14024032
I recently found an app called Threadmaster that will throttle CPU usage on apps when they start hogging too much CPU.  Works like a champ!!

http://threadmaster.tripod.com/

The page is rather straightforward.  Enjoy!
0

Featured Post

Free Tool: Port Scanner

Check which ports are open to the outside world. Helps make sure that your firewall rules are working as intended.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

NTFS file system has been developed by Microsoft that is widely used by Windows NT operating system and its advanced versions. It is the mostly used over FAT file system as it provides superior features like reliability, security, storage, efficienc…
In the below post we have mentioned the best hosting type for startups. Also, check out some of the superlative web hosting companies that are proposing affordable web hosting solutions to host your startup website.
Screencast - Getting to Know the Pipeline
When cloud platforms entered the scene, users and companies jumped on board to take advantage of the many benefits, like the ability to work and connect with company information from various locations. What many didn't foresee was the increased risk…
Suggested Courses

830 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question