Jack_A_Roe
asked on
Service Unavailable - unusual Application Pool problem
One evening our web front end SharePoint server (MOSS on Win Srv 2003) spontaneously began spitting out W3SVC 1011 events. The day after our internal portal was unavailable. "Could not be found" first, and then "Service Unavailable", when the application pool is automatically shut down.
I have googled this literally for hours, and tried several solutions from MS Knowledgebase and others, but nothings helps.
I also have W3SVC warnings 1009, 1010, and 1013, and the problem is according to these related to the application pool. I have tried with different service accounts, seen to that the accounts have the rights and permissions it needs; created different application pools, and so forth and so on, but nothing makes any difference. - Other sites works.
I have googled this literally for hours, and tried several solutions from MS Knowledgebase and others, but nothings helps.
I also have W3SVC warnings 1009, 1010, and 1013, and the problem is according to these related to the application pool. I have tried with different service accounts, seen to that the accounts have the rights and permissions it needs; created different application pools, and so forth and so on, but nothing makes any difference. - Other sites works.
ASKER
Thanks for answering, but no, no updates were installed at the time, or for several days if not weeks prior to this. The update the KB article refers to is not installed.
Are these .net errors? Can you paste the event log errors here?
ASKER
Here these W3SVC warnings are. I also have a recurring .NET error in the application log (se below); I can get rid of this by deleting files in the file system, but there's really no point, AFAIKS. Also, there are errors telling me that crawling can't begin because the content sources can't be found. Note that there's nothing wrong with the SQL server or its databases.
Central Administration works without problems. Test sites w. test application pools too. But if I put my portal site in the CA application pool, there's no difference.
Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1010
Date: 2010-01-13
Time: 14:32:42
User: N/A
Computer: MYSERVER
Description:
A process serving application pool 'My Application Pool' failed to respond to a ping. The process id was '5952'.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1011
Date: 2010-01-13
Time: 14:36:38
User: N/A
Computer: MYSERVER
Description:
A process serving application pool 'My Application Pool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '636'. The data field contains the error number.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 6d 00 07 80 m..€
Event Type: Error
Event Source: .NET Runtime 2.0 Error Reporting
Event Category: None
Event ID: 1000
Date: 2010-01-13
Time: 15:14:25
User: N/A
Computer: MYSERVER
Description:
Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d6968e, faulting module kernel32.dll, version 5.2.3790.4480, stamp 49c51f0a, debug? 0, fault address 0x0000bef7.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 00 70 00 70 00 6c 00 A.p.p.l.
0008: 69 00 63 00 61 00 74 00 i.c.a.t.
0010: 69 00 6f 00 6e 00 20 00 i.o.n. .
0018: 46 00 61 00 69 00 6c 00 F.a.i.l.
0020: 75 00 72 00 65 00 20 00 u.r.e. .
0028: 20 00 77 00 33 00 77 00 .w.3.w.
0030: 70 00 2e 00 65 00 78 00 p...e.x.
0038: 65 00 20 00 36 00 2e 00 e. .6...
0040: 30 00 2e 00 33 00 37 00 0...3.7.
0048: 39 00 30 00 2e 00 33 00 9.0...3.
0050: 39 00 35 00 39 00 20 00 9.5.9. .
0058: 34 00 35 00 64 00 36 00 4.5.d.6.
0060: 39 00 36 00 38 00 65 00 9.6.8.e.
0068: 20 00 69 00 6e 00 20 00 .i.n. .
0070: 6b 00 65 00 72 00 6e 00 k.e.r.n.
0078: 65 00 6c 00 33 00 32 00 e.l.3.2.
0080: 2e 00 64 00 6c 00 6c 00 ..d.l.l.
0088: 20 00 35 00 2e 00 32 00 .5...2.
0090: 2e 00 33 00 37 00 39 00 ..3.7.9.
0098: 30 00 2e 00 34 00 34 00 0...4.4.
00a0: 38 00 30 00 20 00 34 00 8.0. .4.
00a8: 39 00 63 00 35 00 31 00 9.c.5.1.
00b0: 66 00 30 00 61 00 20 00 f.0.a. .
00b8: 66 00 44 00 65 00 62 00 f.D.e.b.
00c0: 75 00 67 00 20 00 30 00 u.g. .0.
00c8: 20 00 61 00 74 00 20 00 .a.t. .
00d0: 6f 00 66 00 66 00 73 00 o.f.f.s.
00d8: 65 00 74 00 20 00 30 00 e.t. .0.
00e0: 30 00 30 00 30 00 62 00 0.0.0.b.
00e8: 65 00 66 00 37 00 0d 00 e.f.7...
00f0: 0a 00 ..
Central Administration works without problems. Test sites w. test application pools too. But if I put my portal site in the CA application pool, there's no difference.
Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1010
Date: 2010-01-13
Time: 14:32:42
User: N/A
Computer: MYSERVER
Description:
A process serving application pool 'My Application Pool' failed to respond to a ping. The process id was '5952'.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Event Type: Warning
Event Source: W3SVC
Event Category: None
Event ID: 1011
Date: 2010-01-13
Time: 14:36:38
User: N/A
Computer: MYSERVER
Description:
A process serving application pool 'My Application Pool' suffered a fatal communication error with the World Wide Web Publishing Service. The process id was '636'. The data field contains the error number.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 6d 00 07 80 m..€
Event Type: Error
Event Source: .NET Runtime 2.0 Error Reporting
Event Category: None
Event ID: 1000
Date: 2010-01-13
Time: 15:14:25
User: N/A
Computer: MYSERVER
Description:
Faulting application w3wp.exe, version 6.0.3790.3959, stamp 45d6968e, faulting module kernel32.dll, version 5.2.3790.4480, stamp 49c51f0a, debug? 0, fault address 0x0000bef7.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 41 00 70 00 70 00 6c 00 A.p.p.l.
0008: 69 00 63 00 61 00 74 00 i.c.a.t.
0010: 69 00 6f 00 6e 00 20 00 i.o.n. .
0018: 46 00 61 00 69 00 6c 00 F.a.i.l.
0020: 75 00 72 00 65 00 20 00 u.r.e. .
0028: 20 00 77 00 33 00 77 00 .w.3.w.
0030: 70 00 2e 00 65 00 78 00 p...e.x.
0038: 65 00 20 00 36 00 2e 00 e. .6...
0040: 30 00 2e 00 33 00 37 00 0...3.7.
0048: 39 00 30 00 2e 00 33 00 9.0...3.
0050: 39 00 35 00 39 00 20 00 9.5.9. .
0058: 34 00 35 00 64 00 36 00 4.5.d.6.
0060: 39 00 36 00 38 00 65 00 9.6.8.e.
0068: 20 00 69 00 6e 00 20 00 .i.n. .
0070: 6b 00 65 00 72 00 6e 00 k.e.r.n.
0078: 65 00 6c 00 33 00 32 00 e.l.3.2.
0080: 2e 00 64 00 6c 00 6c 00 ..d.l.l.
0088: 20 00 35 00 2e 00 32 00 .5...2.
0090: 2e 00 33 00 37 00 39 00 ..3.7.9.
0098: 30 00 2e 00 34 00 34 00 0...4.4.
00a0: 38 00 30 00 20 00 34 00 8.0. .4.
00a8: 39 00 63 00 35 00 31 00 9.c.5.1.
00b0: 66 00 30 00 61 00 20 00 f.0.a. .
00b8: 66 00 44 00 65 00 62 00 f.D.e.b.
00c0: 75 00 67 00 20 00 30 00 u.g. .0.
00c8: 20 00 61 00 74 00 20 00 .a.t. .
00d0: 6f 00 66 00 66 00 73 00 o.f.f.s.
00d8: 65 00 74 00 20 00 30 00 e.t. .0.
00e0: 30 00 30 00 30 00 62 00 0.0.0.b.
00e8: 65 00 66 00 37 00 0d 00 e.f.7...
00f0: 0a 00 ..
Based on those errors you are going to need to do some deeper debugging. Check out the debug diag tool. It will generate a report when your process crashes that should hopefully point you towards a fix: http://support.microsoft.com/kb/919792
Yeah I agree with Tray these look heavy.
Debug diag might be able to help but that is very heavy going I am not sure how man here will be able to help with that dump. Likely you will need MS support help with that unless it is something very obvious in teh dump.
Debug diag might be able to help but that is very heavy going I am not sure how man here will be able to help with that dump. Likely you will need MS support help with that unless it is something very obvious in teh dump.
ASKER
Thanks for the replies. I've run the debag diag tool already, but it's pretty hard to interpret the outcome. There are several errors of the following type at the time of the crash (se below). And the crash---temination of processes---appears at couple of minutes after you access the site from a browser, at the same time a login box appears for the user.
In w3wp__PID__3612__Date__01_
This exception originated from msvcr80!_CxxThrowException
Review the faulting call stack for thread 16 to determine root cause for the exception.
Please follow up with vendor Microsoft Corporation for problem resolution concerning the following file: C:\WINDOWS\WinSxS\x86_Micr
Perhaps I need to try to rebuild this web application, but if it's bad code, it might follow me to the new one. I'm reluctant to call Microsoft Support, cause you usually end up talking to an Indian guy with an Irish accent, and I simply don't understand what they are saying.This exception originated from msvcr80!_CxxThrowException
Review the faulting call stack for thread 16 to determine root cause for the exception.
Please follow up with vendor Microsoft Corporation for problem resolution concerning the following file: C:\WINDOWS\WinSxS\x86_Micr
ASKER
One more thing, please note that while those dlls seem to be corrupt or something of the kind, every other web site works, for instance, Central Administration. If I put the portal site in the application pool in which CA is run, I get the same errors. If I put Central Administration in the application pool mentioned in the events above ("Portal Application Pool"), CA works. So the problem seems not to be the application pool, or the web service in it self, but this particular site. It is very strange because no one was working with the site when it decided to report the event, and no one had for several days (christmas holidays).
ASKER
Update: While I can't browse to http://portal, I can browse to subsites, as in http://portal/sales, and I can browse to libraries on http://portal and read the document, if accessing them directly with apropriate URL:s. When I'm at a subsite, I can browse around as usual, however if I click "Portal", it is not accessible.
I installed a new SharePoint server, with a new farm, and a web application named Portal; removed the content db created in the process, and added a copy of the "portal db", which worked fine. However, the symptoms are exactly the same on the new server farm, as in the old.
So it certainly seems like it is the default.aspx page which is corrupt. I can access the master page and so forth in SharePoint Designer for instance, and I can create a new start page with the old hierarchy, fully browsable and so forth---but as soon as I go to the original start page, things hangs and I get the W3SCV-warning 1011 above, on the new and on the old farm.
I installed a new SharePoint server, with a new farm, and a web application named Portal; removed the content db created in the process, and added a copy of the "portal db", which worked fine. However, the symptoms are exactly the same on the new server farm, as in the old.
So it certainly seems like it is the default.aspx page which is corrupt. I can access the master page and so forth in SharePoint Designer for instance, and I can create a new start page with the old hierarchy, fully browsable and so forth---but as soon as I go to the original start page, things hangs and I get the W3SCV-warning 1011 above, on the new and on the old farm.
ASKER CERTIFIED SOLUTION
membership
This solution is only available to members.
To access this solution, you must be a member of Experts Exchange.
http://support.microsoft.com/kb/2009746