Static object in web services

Good morning!

We have the set of the web services developed with Net 3.5 SP1 and hosted under IIS 7.5 (w2008/x64). Those web services are completely stateless and do not use any session cache. We have app pool recycled every hour. System is working under heavy load 24x7.

The web service implementation contains class library with the logic statically referenced by the web service class. All methods in web service class are like:
void Method()

ImplementationClass GetImplementationObject()
    return new ImplementationClass();

Our implementation class has one static object with constructor which reads the data from the system catalog views from MSSQL 2008. There is also the static constructor in the implementation class which creates this static object. Web service methods call one of the method of that object which is doing some processing based on the data cached during constructor call.

The reason why we implemented it this way is because the data the object is cached is fairly static (updating one time per a few weeks). So we would like to cache it in app pool and don't read it from the database in the every web service method call.

This works just fine although one time per a few days we are getting exceptions: System.Web.Services.Protocols.SoapException: Server was unable to process request. ---> The type initializer for 'ImplementationClass' threw an exception. ---> Timeout expired.  The timeout period elapsed prior to completion of the operation or the server is not responding. The exceptions (for all sessions) last until app pool is recycled. Timeout expired could be related with SQL server, although server is up and running - other processes are working just fine.

After some time we moved the static object creation from ImplementationClass static constructor to the web methods. Basically the pattern is:
if (staticObject == null)
        if (staticObject == null)
            staticObject = StaticClass();

although it would not help. Now we are getting the following exception:
System.Net.WebException: The request failed with HTTP status 503: Service Unavailable.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)

with the following entry in the event log: Faulting application name: w3wp.exe, version: 7.5.7600.16385, time stamp: 0x4a5bd0eb Faulting module name: KERNELBASE.dll, version: 6.1.7600.16385, time stamp: 0x4a5bdfe0 Exception code: 0xe053534f Fault offset: 0x000000000000aa7d Faulting process id: 0x%9 Faulting application start time: 0x%10 Faulting application path: %11 Faulting module path: %12 Report Id: %13.

Again, it happens one time per a few days and last until app pool recycled.

We can change the implementation and get rid of the static object (put the object to the global cache, etc) although it would require some time and efforts.

I wonder, if you have the experience with the static objects in the web services and can give us some advice. We are not 100% sure if problem is related with the static object although it started to happen after we implement this feature

Thank you
LVL 13
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Richard LeeSoftware EnthusiastCommented:
What I think you need here is more information regarding the exception being thrown. The type initializer exception will be the constructor not being able to complete so you will need to examine the inner exception to identify what the real cause of the issue is. It would be handy if you log the exception or possibly using IIS7 logging facilities. I'm not sure if the event-viewer shows the complete exception.

How you handle the cache when as a static object, lazy loading or even using the HttpRuntime.Cache doesn't matter. They all should work and not impact the running of the application. The question begs to be asked, "what else are you doing within the application?". It sounds to be that the constructor is taking too long to execute. Also how are you refreshing the cache if its being loaded in the static constructor / lazy loaded?

Also its worth considering that resources are not being freed or disposed of properly. One nasty culprit is always the "lapsed listener". It's worth a mention although! Recycling the application pool ever so often wreaks of memory leak.

dwkorAuthor Commented:
Well, unfortunately we already updated the version with the initializing the object in the web methods, not via static constructor. So we missed the opportunity to log the inner exception. We can rollback and change the code, of course, but it would take a few days until it happens again..

The web service is the basically very simple middle tier which exchange the data between database and client applications. The communication between web service and client application is done via compressed .net dataset in xml format. (dataSet.ToXml() -> compressed stream). The typical signatures for the methods are:
byte[] GetData(string, ...)
void UpdateData(string, byte[],..)

The purpose of the static object is reading the most recent SQL TVP definitions from the database (in constructor) and transforms different versions of the client datasets to be compatible with TVP (in the method). Basically the transformation does 3 things: deletion of the DataColumns which are not in TVP definition, creation DataColumns which are in the definition and rearrange column sequence if needed. The TVP definition is not updating except during the system update (which is done in maintenance window), so there is no needs to refresh the cache after it's loaded.  

Interesting thing that problem does not have direct correlation with the load. Last time it happened at the middle of the night when load is substantially smaller than during the pick hours.

We also found that there are a few entries in the event log with EventId = 5011 - which leads to Although it would not give us a lot of info either..
Richard LeeSoftware EnthusiastCommented:
I do think more information is needed here to come to correct conclusion. I provided several suggestions :

1. Logging exception via code
2. Using IIS7 to log trace information
3. Monitor memory usage

Exploring SQL Server 2016: Fundamentals

Learn the fundamentals of Microsoft SQL Server, a relational database management system that stores and retrieves data when requested by other software applications.

dwkorAuthor Commented:
Thank you! Will try to collect it soon
Richard LeeSoftware EnthusiastCommented:
I await the response
dwkorAuthor Commented:
Well, it looks like the problem was related somehow with the static constructor. After some troubleshooting we found that 503 error were caused by microsoft trace framework. When we got rid of it, the method when the static object is created from the web method is working without any issues.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
.NET Programming

From novice to tech pro — start learning today.