IIS / W3WP.exe Instabilities

We have 2x 2008R2 servers clustered that is hosting our document management solution (a product called Sybrin that uses .net). It consists of a service running that communicates with SQL to store / retrieve metadata on documents our company scans / generates. The documents themselves are stored on disks that these servers have access to. 2x IIS Web Services are used by other systems to communicate to this system.

Recently we've started experiencing instability on the environment. When the instability occurs w3wp.exe CPU spikes up to 100% (normally this is around 10%). Sometimes the instability occurs even if CPU usage is normal. Trying to access the web service WSDL's times out. To get the system to work again, we recycle the application pools used by the web services. This almost always resolves the issues we are experiencing.

The following events ID's appear often:

Log Name:      Intercept
Source:        Intercept PMonitor
Date:          2015/09/21 10:48:15 PM
Event ID:      4291
Task Category: None
Level:         Warning
Keywords:      Classic
User:          N/A
Computer:      npwsybc1.africanbank.net
Description:
 Due to an unusually high system load, the threshold for functions processed per second has been exceeded. This may have resulted from time-only monitoring of a namespace with a large number of functions. Please change your configuration settings to prevent this situation. Execution time monitoring for namespaces will be disabled.

System information:
Agent version: 5.7.491.0
Windows version: 2.6.1.3.274.1.9.0;
.NET Framework: v4.0.30319
Process ID: 6168; Instance ID: 0;
InstanceName: w3wp.exe [x86];
Application Pool: Classic .NET AppPool.

Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Intercept PMonitor" />
    <EventID Qualifiers="32768">4291</EventID>
    <Level>3</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-09-21T20:48:15.000000000Z" />
    <EventRecordID>64724</EventRecordID>
    <Channel>Intercept</Channel>
    <Computer>npwsybc1.africanbank.net</Computer>
    <Security />
  </System>
  <EventData>
    <Data>
    </Data>
    <Data>
System information:
Agent version: 5.7.491.0
Windows version: 2.6.1.3.274.1.9.0;
.NET Framework: v4.0.30319
Process ID: 6168; Instance ID: 0;
InstanceName: w3wp.exe [x86];
Application Pool: Classic .NET AppPool.
</Data>
  </EventData>
</Event>


Log Name:      Application
Source:        Application Error
Date:          2015/10/15 11:23:16 AM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      npwsybc1.africanbank.net
Description:
Faulting application name: SybrinService.exe, version: 1.0.0.0, time stamp: 0x4d92ecf2
Faulting module name: KERNELBASE.dll, version: 6.1.7601.18015, time stamp: 0x50b83c8a
Exception code: 0xe0434352
Fault offset: 0x0000c41f
Faulting process id: 0xb70
Faulting application start time: 0x01d104eef16cf73b
Faulting application path: C:\Program Files (x86)\Sybrin\Sybrin DotNet Server\SybrinService.exe
Faulting module path: C:\Windows\syswow64\KERNELBASE.dll
Report Id: 5cfbc236-731e-11e5-99af-005056bc0099
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-10-15T09:23:16.000000000Z" />
    <EventRecordID>156220</EventRecordID>
    <Channel>Application</Channel>
    <Computer>npwsybc1.africanbank.net</Computer>
    <Security />
  </System>
  <EventData>
    <Data>SybrinService.exe</Data>
    <Data>1.0.0.0</Data>
    <Data>4d92ecf2</Data>
    <Data>KERNELBASE.dll</Data>
    <Data>6.1.7601.18015</Data>
    <Data>50b83c8a</Data>
    <Data>e0434352</Data>
    <Data>0000c41f</Data>
    <Data>b70</Data>
    <Data>01d104eef16cf73b</Data>
    <Data>C:\Program Files (x86)\Sybrin\Sybrin DotNet Server\SybrinService.exe</Data>
    <Data>C:\Windows\syswow64\KERNELBASE.dll</Data>
    <Data>5cfbc236-731e-11e5-99af-005056bc0099</Data>
  </EventData>
</Event>


Please advise what we can check to get this resolved. Thank you in advance.
AblSysadminSenior Systems EngineerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Radhakrishnan RSenior Technical LeadCommented:
Hi,

Open a command prompt and run the following command:
%windir%\system32\inetsrv\appcmd.exe list wp
It will list the PID and the Application Pool names. Check if any PID matches with the ones noted from the Task Manager. This is to confirm that the w3wp.exe belongs to the WsusPool Application Pool. If this does not match, then it means that the w3wp.exe for the WsusPool application is not consuming more memory. You can abandon this procedure.
If it matches, then go to Start, in the Run box, type inetmgr. This will open the IIS Manager.Navigate to the Application Pools folder. Right-click WsusPool and choose Recycling. The Edit Application Pool Recyling Settings window will open. Set 'Private memory usage' to 102400. Click Next, Finish. Close IIS Manager. Open a command promt and type IISRESET. Wait for Internet sevices to stop and start successfully. Type exit.
Dan McFaddenSystems EngineerCommented:
OK, not sure what the WsusPool has to do with anything on this issue.

There is 1 AppPool amd 1 executable that are throwing errors (according to your post above):

1.  w3wp.exe = Classic .NET AppPool
2.  executable = SybrinService.exe

First a few questions:

1. have you recently installed any OS patches?
2. have you recently installed any 3rd party software on the server?
3. have you recently updated the vendor's software, the Sybrin Service?
4. can you consistently reproduce the error?
5. do you know what requests were hitting the system when the error occurs?
6.  have you looked into the HTTP logs to try to correlate a request to the error occurring?

Also, the error messages above are post crash entries.  The Process ID is in the entry.  Trying to find the w3wp.exe with the PID listed in the message will not work because the process has already crashed.

I would also recommend contacting the vendor of the document management system.  They may be able to better help you since it is their software.

Dan
AblSysadminSenior Systems EngineerAuthor Commented:
Hi Radhakrishnan, what does 'Private memory usage' to 102400 achieve or improve in our environment?

Hi Dan,

1. No
2. No, nothing new
3. No, we've been running the same version for almost a year
4. Yesterday I figured out if a blank (or wildcard) search for documents are done using Sybrin, the system falls over a little bit later with the same symptoms. I can reproduce it.
5. It is request to either store or retrieve documents.
6. Yes, but we're not seeing anything at the same time as the crashes.

I suspect #4 might be the cause of the issue if the web services are perhaps passing blank values to Sybrin or users are searching incorrectly. Your thoughts?
Simplify Active Directory Administration

Administration of Active Directory does not have to be hard.  Too often what should be a simple task is made more difficult than it needs to be.The solution?  Hyena from SystemTools Software.  With ease-of-use as well as powerful importing and bulk updating capabilities.

Radhakrishnan RSenior Technical LeadCommented:
Hi,

Apologies, just realized that your CPU usage is high not memory. Unfortunately, the error still looks like something related to IIS and Application pool which means any one of the appool (I.e - .Net Apppool if one persists).

Is the CPU usage goes down when you stop that specific apppool? or stopping IIS?

If the server has Microsoft Host Integration Server 2010 then there is a hotfix available https://support.microsoft.com/en-us/kb/2544897 

If the issue persists, you may need to run some debugging tool and analyse it for further troubleshoot.
Dan McFaddenSystems EngineerCommented:
ablsysadmin,

IMO, this is a straight up application error.  Modifying or playing around with the AppPool configuration will probably result in much.

If the web app is crashing upon a user entering blank or null characters in the search field, then the application is not handling or validating inputs correctly.  You cannot fix or remedy this by tweaking the site or app pool.

I asked previously, have you informed the vendor/developer of the application of the issue?  Let them know that the issue is reproducible and give them exact details (step-by-step) instructions on how you can reproduce the error/failure.

A problem like yours will most likely require a resolution from the developer.

Can you post some lines from the http logs at the time the error occurs?  20 lines before and 20 lines after would be great.  

Dan
Dan McFaddenSystems EngineerCommented:
Another question, are you using SCOM to monitor your environment?

Intercept PMonitor alerting is a result of AVIcode (.NET Application Monitoring) which now comes as a part of System Center Ops Mgr.  Or do you have a monitoring utility from ViaCode (Intercept Monitor) installed?

Dan
AblSysadminSenior Systems EngineerAuthor Commented:
@Radhakrishnan

CPU does go down when recycling the app pool. But as explained, sometimes the issue occurs with CPU being at normal levels.

We don't use Microsoft Host Integration Server 2010.

@Dan

We've escalated to the vendor. Part of the issue here is that we're on a rather old version of Sybrin. We can't upgrade the version now due to it being so late in the year (too many critical systems rely on this and the call was made to live with the risk of this instability vs putting new code in place). What I'm trying to identify is why this started recently in light of the info I've supplied.

We had this event today when it crashed again:
Log Name:      Application
Source:        Application Error
Date:          2015/10/21 12:27:44 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      npwsybc1.africanbank.net
Description:
Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7a5f8
Faulting module name: PerfMon.dll, version: 5.7.491.0, time stamp: 0x4d01a3ca
Exception code: 0xc0000409
Fault offset: 0x000f59c4
Faulting process id: 0x1758
Faulting application start time: 0x01d10beb1c7a65d5
Faulting application path: C:\Windows\SysWOW64\inetsrv\w3wp.exe
Faulting module path: C:\Program Files\Quest Software\Foglight for .NET\Agent\v5.7.491\PerfMon.dll
Report Id: 5d125913-77de-11e5-99af-005056bc0099
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2015-10-21T10:27:44.000000000Z" />
    <EventRecordID>156642</EventRecordID>
    <Channel>Application</Channel>
    <Computer>npwsybc1.africanbank.net</Computer>
    <Security />
  </System>
  <EventData>
    <Data>w3wp.exe</Data>
    <Data>7.5.7601.17514</Data>
    <Data>4ce7a5f8</Data>
    <Data>PerfMon.dll</Data>
    <Data>5.7.491.0</Data>
    <Data>4d01a3ca</Data>
    <Data>c0000409</Data>
    <Data>000f59c4</Data>
    <Data>1758</Data>
    <Data>01d10beb1c7a65d5</Data>
    <Data>C:\Windows\SysWOW64\inetsrv\w3wp.exe</Data>
    <Data>C:\Program Files\Quest Software\Foglight for .NET\Agent\v5.7.491\PerfMon.dll</Data>
    <Data>5d125913-77de-11e5-99af-005056bc0099</Data>
  </EventData>
</Event>

From W3SVC1 log at the same time:
2015-10-21 10:27:35 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.47 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 15
2015-10-21 10:27:36 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.42 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 15
2015-10-21 10:27:36 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.78 Axis/1.3 200 0 0 249
2015-10-21 10:27:36 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.42 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 15
2015-10-21 10:27:36 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.42 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 0
2015-10-21 10:27:36 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.143 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 15
2015-10-21 10:27:36 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.142 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 15
2015-10-21 10:27:36 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.56 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 93
2015-10-21 10:27:36 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.142 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 0
2015-10-21 10:27:37 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.44 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 77
2015-10-21 10:27:37 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.143 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 15
2015-10-21 10:27:37 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.143 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 0
2015-10-21 10:27:37 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.142 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 15
2015-10-21 10:27:37 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.142 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 0
2015-10-21 10:27:37 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.44 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 0
2015-10-21 10:27:37 10.130.124.131 GET / - 80 - 10.30.118.12 - 200 0 0 0
2015-10-21 10:27:37 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.44 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 15
2015-10-21 10:27:37 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.143 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 15
2015-10-21 10:27:37 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.48 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 109
2015-10-21 10:27:37 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.47 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 0
2015-10-21 10:27:37 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.143 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 15
2015-10-21 10:27:37 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.56 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 109
2015-10-21 10:27:37 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.44 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 93
2015-10-21 10:27:37 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.44 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 0
2015-10-21 10:27:38 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.42 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 0
2015-10-21 10:27:38 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.48 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 15
2015-10-21 10:27:38 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.143 Mozilla/4.0+[en]+(WinNT;+I) 500 0 0 15
2015-10-21 10:27:41 10.130.124.131 GET / - 80 - 10.130.118.12 - 200 0 0 0
2015-10-21 10:27:41 10.130.124.131 GET / - 80 - 10.130.118.11 - 200 0 0 0
2015-10-21 10:27:44 10.130.124.131 GET / - 80 - 10.30.118.11 - 200 0 0 31
2015-10-21 10:27:46 10.130.124.131 GET / - 80 - 10.130.118.12 - 200 0 0 0
2015-10-21 10:27:46 10.130.124.131 GET / - 80 - 10.130.118.11 - 200 0 0 0
2015-10-21 10:27:47 10.130.124.131 GET / - 80 - 10.30.118.12 - 200 0 0 0
2015-10-21 10:27:51 10.130.124.131 GET / - 80 - 10.130.118.12 - 200 0 0 0
2015-10-21 10:27:51 10.130.124.131 GET / - 80 - 10.130.118.11 - 200 0 0 0
2015-10-21 10:27:54 10.130.124.131 GET / - 80 - 10.30.118.11 - 200 0 0 0
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.42 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 2293
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.47 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 4679
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.47 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 4211
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.47 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 4789
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.48 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 2839
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.56 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 2449
2015-10-21 10:27:55 10.130.124.131 GET / - 80 - 10.130.118.12 - 200 0 0 0
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.143 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 1310
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.57 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 140
2015-10-21 10:27:55 10.130.124.131 GET / - 80 - 10.130.118.11 - 200 0 0 0
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.48 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 202
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.142 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 1856
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.143 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 1762
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.44 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 2199
2015-10-21 10:27:55 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.142 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 2480
2015-10-21 10:27:55 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.47 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 311
2015-10-21 10:27:55 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.42 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 3088
2015-10-21 10:27:55 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.42 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 2917
2015-10-21 10:27:55 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.57 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 358
2015-10-21 10:27:55 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx - 80 - 10.130.124.48 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 1169
2015-10-21 10:27:55 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx WSDL 80 - 10.130.124.142 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 5553
2015-10-21 10:27:55 10.130.124.131 POST /SybrinCSWebService/sybrinWebServiceCS.asmx WSDL 80 - 10.130.124.44 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 4336
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.143 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 140
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.143 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 171
2015-10-21 10:27:55 10.130.124.131 POST /SybrinWebService/SybrinWebServiceNet2.asmx - 80 - 10.130.124.142 Mozilla/4.0+[en]+(WinNT;+I) 200 0 0 202

The line in BOLD looks out of the ordinary, I'll ask what it is.
Dan McFaddenSystems EngineerCommented:
I believe the last number in your http log configuration is probably Time Taken.  The bold entry means that to service that request, it took 31ms.

That does not look odd to me.

What I see that concerns me is above the bold line. There are a series of http error code 500 entries.  The lines all contain a hit on this URL stem:

/SybrinWebService/SybrinWebServiceNet2.asmx

Each http 500 (Internal Server Error) is preceded by a successful hit (http 200) on this URL stem:

/SybrinCSWebService/sybrinWebServiceCS.asmx

As a troubleshooting aid, I recommend that you enable all fields in the http logs.  You are using the default setup which leaves out many fields that could help troubleshoot the issue.

Dan
AblSysadminSenior Systems EngineerAuthor Commented:
I've enabled more fields in the http logs and will supply more information as soon as we've had another crash. Thank you for you patience in assisting with this.
AblSysadminSenior Systems EngineerAuthor Commented:
Hi Guys

Apologies for the 4 month delayed response.
I have since determined that the issue was related to old application code.
Some of the app users were running wildcard queries (blank search parameters) which resulted in the app falling over with these IIS errors.
As a workaround, we removed these peoples access because they had an alternate method of getting the data they were looking for.
As a solution, we are rewriting/redesigning the application.

Once again, thank you guys for your assistance. much appreciated

Regards

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
AblSysadminSenior Systems EngineerAuthor Commented:
The old application code was outdated and did not cater for much expansion.
rewriting the app not only provides a fix for the issue at hand but caters for future growth as well.
it was the best solution for us.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft IIS Web Server

From novice to tech pro — start learning today.