Solved

Using more threads to process ASP.NET web service requests?

Posted on 2010-09-01
16
394 Views
Last Modified: 2012-05-10

Is there a setting somewhere in IIS 6/.NET 2.0/Windows Server 2003 to allocate threads more aggressively to handling pending ASP.NET web service requests?  When we have more than a couple of long-running web service requests arrive at approximately the same time, some of those requests end up queued behind others even though there are free threads remaining in the ASP.NET thread pool.

In our server's configuration files, we have tried changing the processModel attributes maxWorkerThreads, maxIoThreads, and minWorkerThreads, but have been unable to substantially affect the observed "queuing" behavior.  We also played around with the httpRuntime attributes minFreeThreads and minLocalRequestFreeThreads, also without success.  We also substantially increased maxConnection in the connectionManagement section in case that was involved, but again it did not help.

The web service request threads are not bound by CPU, I/O or any other measure that we are aware of.  Mostly they are waiting on responses from remote systems.  Essentially the server is sitting there nearly idle, but still refuses to process additional web service requests until the first few it received have finished.  It seems like there should be a way to tweak this behavior when there are both unused threads and plenty of idle CPU available.  

Any ideas?
0
Comment
Question by:HICKORYTECH
  • 8
  • 7
16 Comments
 
LVL 30

Expert Comment

by:MlandaT
ID: 33579736
long running processes.... does the web service need to return a result based on that processing? if so.. there is little you can actaully do by way of programming... besides optimising the long runing processing... otherwise.. you can call the long running process in a seperate thread... just do the normal thing to create a thread.. alternatively... queue the requests into MSMQ or something and then process them there... since your processing involves other servers (and you have no control over what happens there)... it's best not to block the calling application.

http://www.google.co.za/url?q=http://msdn.microsoft.com/en-us/library/system.threading.thread.start.aspx&sa=U&ei=epd-TJDVLof6lweR9ajQBg&ved=0CBoQFjAB&usg=AFQjCNEgtVuAkHBg9CKJkFBaJD2XpRI_OA
0
 

Author Comment

by:HICKORYTECH
ID: 33579845
Yes, a result does need to be returned based on the long-running (perhaps five seconds) request processing.  (We've heavily optimized the processing already ... it used to take minutes in some cases.)
0
 
LVL 30

Expert Comment

by:MlandaT
ID: 33579909
5 seconds... ok... maybe try spawning a thread then. that will probably be the easiest... but remember that it does not guarantee that the other servers you are calling dont get overloaded in any case.. how many hits are we talking about?
0
 

Author Comment

by:HICKORYTECH
ID: 33580058
Different requests to this server reach out to different external servers, so we're not concerned about that ... the WS requests incoming to the first server are the chokepoint.  I've verified this using a test app that just sleeps for a few seconds for each web service request ... the requests still queue up in the same way.

We can see the request queuing begin to happen with as few as four overlapping web service requests (a number of web service requests which a single page load in our application can easily launch).

One clarification, please: I thought spawning a thread would only be of benefit if we didn't need to return a result?  Perhaps I misunderstood?
0
 
LVL 30

Expert Comment

by:MlandaT
ID: 33580179
ooops! sorry... i misread your comment... "Yes, a result does need to be returned".

in that case... I'm afraid you have few options perhaps consider a proper load balanced web farm? but you havent quite given a good estimate of the number of hits you get... so a web farm might be overkill. other option is to resort to an aternative notification mechanism... e.g. an email notificaiton?.. then the calling app does not 'have' to wait for a response. but again.. depends on whether the outcome of the web service call is needed as input on other processing or not...
0
 
LVL 30

Expert Comment

by:MlandaT
ID: 33580193
"few as four overlapping web service requests"... what is your environment?
0
 

Author Comment

by:HICKORYTECH
ID: 33580372
We have a thin client application used over local corporate networks.  The web service requests usually get launched after a page has loaded, to retrieve additional types of information that are useful to the user but which don't need to be loaded before the page displays, in order to get the bulk of the page in front of the user's eyes as quickly as possible.

Before you ask, the additional information from these web service calls invariably needs to update the main page contents, often in significant ways, so we can't delay making the calls until the user clicks on some page control to request them.  In fact this is what spawned this research and question - users complaining that the later loading portions of the page were taking too long to appear, and this was determined to be because some of those requests were queued up somewhere inside IIS or ASP.NET and weren't even reaching our web methods until earlier requests had been fully serviced.

I can't give too specific of an estimate of the number of hits because it varies widely from one server to the next (I was trying to keep the question simple so I narrowed it to one specific server).  We do track the loads on our servers and don't see loads that would justify a web farm, especially since we can reproduce the same sort of request queuing behavior even on our servers with the lightest loads.  This is why we have been hoping there is some configuration setting we are unaware of.
0
 
LVL 30

Expert Comment

by:MlandaT
ID: 33581165
Is caching the data on the user's machine an option? then expiring those caches say once a week or something?
0
IT, Stop Being Called Into Every Meeting

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 

Author Comment

by:HICKORYTECH
ID: 33581490
Alas, client-side caching offers no benefit in our application.  The data is almost always different each time the same page is loaded, based on numerous factors, not least of which is because different accounts are viewed via the same page.  Also, even data for a single account is volatile - it can change at any time due to external factors.
0
 
LVL 30

Accepted Solution

by:
MlandaT earned 500 total points
ID: 33581657
Move the data closer to the calling application. Do away with the other calls to the other webservices. As opposed to using your own load balancing mechanisms... use the Windows Server Load Balancing mechanisms... http://www.west-wind.com/presentations/loadbalancing/networkloadbalancingwindows2003.asp

Other than that... it seems to me that there is little you can actually do to improve performance. The intermediate web service is the main bottleneck. Eliminate that web service call.. the load balancing server will forward the requests before they even hit the iis. you may also want to consider replicating your database so that you have a server that is dedicated to processing those requests for infomration and optimise the database for retreival (indexes, precalculate some things and place in intermediate tables, etc). You need a combination of approaches. Other than that... I'm afraid I cant help further.
0
 

Author Comment

by:HICKORYTECH
ID: 33581797
Hehe ... your recommendation about moving the data closer to the calling application is very much like what I already did to optimize the performance, so good call there.  :)  We can't do away with the calls entirely because the remote data is also volatile, but I moved all possible processing of that data to happen locally.

If there's no IIS/.NET tweak to help us, I have been contemplating the idea of implementing a single web service method that in one gulp takes an XML containing all of the requests from a single page, and then spins off threads for processing each request.  The advantage would be better parallelism (in theory), but this also means none of the requests would return until the longest processing one had finished.  Better than the queuing that happens now, but not much.  That is, unless I could also figure out a way to fire off multiple callbacks, one as each request completes ... but I'm not sure if that's possible.
0
 
LVL 30

Expert Comment

by:MlandaT
ID: 33582014
Well I was hoping you could avoid having to worry about thread syncronisation issues. you could however call the other web methods asynchronously and use polling on the IAsyncResult to wait for all of them to complete. It will giv you additional performance... in a sense... http://www.stardeveloper.com/articles/display.html?article=2001121901&page=1. Have a look at the concepts in that artice... might help. All the best.

PS: I'm not too familiar with all IIS settings... maybe there is something that will work for you that I am not aware of.
0
 
LVL 9

Expert Comment

by:shalabhsharma
ID: 33582964
Configure webgrden for your process thread.

A Web Garden can only be used when IIS is running in worker process isolation mode (default), and is configured on a per application pool level. The idea behind configuring an application pool to use multiple worker processes (i.e a Web Garden) is to improve scalibility and reliability, however in practice this has not always been the case with single CPU/Core systems as these worker processes are all competing for the same CPU time. However since multi processor and multi core CPU systems have become the rule rather than the exception for low/medium spec servers over the last few years the advantage of using a a Web Garden have started to make a lot more sense.

To configure a Web garden by using IIS Manager

1.In IIS Manager, expand the local computer, expand Application Pools, right-click the application pool, and then click Properties.

2.Click the Performance tab, and under Web garden, in the Maximum number of worker processes box, type the number of worker processes that you want to assign to the application pool. (You must type a number greater than 1 for the application pool to become a Web garden.

3.ck OK.

http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/659f2e2c-a58b-4770-833b-df96cabe569e.mspx?mfr=true
0
 

Author Comment

by:HICKORYTECH
ID: 33599417
Configuring a web garden was an interesting idea, but testing indicates it does not relieve the request queuing issue, unfortunately - at least not for multiple requests stemming from a single client.

Essentially we're already doing the same thing discussed in that StarDeveloper article, except via a browser rather than a WinForms app.  All of our web methods are called asynchronously via the XMLHTTP component, and then the web client waits for responses from any/all of them using onReadyStateChange.  The request queuing behavior happens entirely on the server, and in spite of the fact that the calls were made asynchronously.
0
 
LVL 30

Expert Comment

by:MlandaT
ID: 33602519
well... in that case then add more hardware, much much more hardware
0
 

Author Closing Comment

by:HICKORYTECH
ID: 33646927
Not really a solution we can implement, but eventually the response confirmed what we already suspected to be the case (that there's no easy IIS trick, and our only real option is to optimize our architecture and data flow as much as possible).
0

Featured Post

Free Trending Threat Insights Every Day

Enhance your security with threat intelligence from the web. Get trending threat insights on hackers, exploits, and suspicious IP addresses delivered to your inbox with our free Cyber Daily.

Join & Write a Comment

Setting up a Microsoft WSUS update system is free relatively speaking if you have hard disk space and processor capacity.   However, WSUS can be a blessing and a curse. For example, there is nothing worse than approving updates and they just have…
International Data Corporation (IDC) prognosticates that before the current the year gets over disbursing on IT framework products to be sent in cloud environs will be $37.1B.
In this seventh video of the Xpdf series, we discuss and demonstrate the PDFfonts utility, which lists all the fonts used in a PDF file. It does this via a command line interface, making it suitable for use in programs, scripts, batch files — any pl…
This demo shows you how to set up the containerized NetScaler CPX with NetScaler Management and Analytics System in a non-routable Mesos/Marathon environment for use with Micro-Services applications.

706 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

14 Experts available now in Live!

Get 1:1 Help Now