Solved

WIN2k Web Server IIS crashes every day

Posted on 2002-05-08
38
426 Views
Last Modified: 2008-02-26
It's got to be the most frustrating problem I have ever encountered.

Our WIN2k web server IIS is crashing every so often.  Sometimes it goes fine for 3-4 days and then 2-3 times a day.  The server doesn't crash, just IIS.  However, in the services it shows IIS running fine.  When we go to restart it comes up with an error.  The only way to fix the problem is to reboot the server.

We have been troubleshooting this for a whole month and no solution has been found.

PLEASE HELP.  As far as I am concerned this is worth a whooping 500 points to anyone who can figure this wierd anomoly out.
0
Comment
Question by:dustygulleson
  • 18
  • 13
  • 2
  • +5
38 Comments
 
LVL 32

Expert Comment

by:jhance
ID: 6997788
IIS and W2K are very reliable.  My guess is that you have a hardware problem:

1) Check your RAM and swap it out for some known good modules.

2) Make sure ALL hardware is W2K SERVER supported by Microsoft in the HCL.  Remove or replace any questionable items.

3) Make sure you have the proper driver versions for all your hardware.

4) Overclocking? Stop it.  Under-cooling?  Fix it.

5) 3rd party software?  Remove/disable it to eliminate it as a cause.

6) W2K SPs and HotFixes up to date?  Perhaps you have a hacker crashing your system.  Got a firewall?  Make sure non-required ports are blocked from public access.
0
 

Author Comment

by:dustygulleson
ID: 6997797
JHANCE,

My response:

1) Check your RAM and swap it out for some known good modules.
-- The hardware is a DELL WebApp 1 GiG Mghz, 1 GiG RAM, RAID 5 SCSI 76 Meg

2) Make sure ALL hardware is W2K SERVER supported by Microsoft in the HCL.  Remove or replace any questionable
items.
-- All DELL direct.  Brand new build.

3) Make sure you have the proper driver versions for all your hardware.
-- Have done a thorough check on all drivers.  Latest drivers are installed.

4) Overclocking? Stop it.  Under-cooling?  Fix it.
-- Haven't touched it. Ever.

5) 3rd party software?  Remove/disable it to eliminate it as a cause.
-- Have PCAnywhere, DeepMetrix Livestats 6.0 XSP (Web Traffic Analytics).

6) W2K SPs and HotFixes up to date?  Perhaps you have a hacker crashing your system.  Got a firewall? Make sure non-required ports are blocked from public access.

-- All HotFixes upto date.  Installing WatchGuard 700 this weekend.


Any other thoughts?
0
 
LVL 3

Expert Comment

by:Corvax021899
ID: 6997958
Any event log errors?
I would remove or disable any none necessary software.  Just 2k And IIS to see if it's not a 3rd party software that is causing the problem.

0
Enterprise Mobility and BYOD For Dummies

Like “For Dummies” books, you can read this in whatever order you choose and learn about mobility and BYOD; and how to put a competitive mobile infrastructure in place. Developed for SMBs and large enterprises alike, you will find helpful use cases, planning, and implementation.

 

Author Comment

by:dustygulleson
ID: 6997964
Response to CORVAX

The event was the first thing we looked at.  It has nothing.  We initially saw a problem with the server having the inability to browse to the PDC.  We then removed the PDC altogther since it wasn't necassary.  I would be happy to zip up the event log to show you if your still interested.

I turned off the Livestats Web Analytics tool tonight and will track it.  I don't think this is the issue because we tested the deployment on our development server with no problems.

I will maintain PCAnywhere on it simply because it has no history of incompatibility with WIN2K servers AND I don't feel like driving the 30 minutes to the datacenter to do a change.
0
 

Author Comment

by:dustygulleson
ID: 6997967
JHANCE

How can you tell if you have bad RAM.  Is there a diagnostic tool you know about that can test it?

Also, how could bad RAM impact IIS?
0
 
LVL 2

Expert Comment

by:pssiew
ID: 6998450
dusty,

your RAM is most probably ECC so it might not be RAM.

You can check dell's support site for diagnostic tools. They usually have one for servers

Are you running any websites off your IIS ? Rogue asp scripts can cause havoc with IIS.

Since you know its IIS causing the problem, I would suggest you put in either ADPlus or Exception Monitor to get a dump file on IIS, then debug it to see what is the cause if IIS crashing.

For ADPlus - http://support.microsoft.com/directory/article.asp?ID=KB;EN-US;Q286350&

For Exception Monitor - http://www.microsoft.com/technet/treeview/default.asp?url=/TechNet/prodtechnol/iis/downloads/ixcptmon.asp

Hope this helps
0
 
LVL 3

Expert Comment

by:hnminh
ID: 6998501
"However, in the services it shows IIS running fine"
Did you mean services control panel had shown Web was running but actually it was not?

"When we go to restart it comes up with an error"
Could you please provide the error message?

Did you check server performance when we crashed? Anything eatup CPU resource?

How many connection at a time are allowed on the server?

Do you use ASP for the site? Do those ASP required other library (for example to connect to oracle DB or anything like that)....

Some idea that hope it help!
0
 
LVL 3

Expert Comment

by:hnminh
ID: 6998503
oh, my my english :(. The 3rd question should be read "... performance when the server crashed?..."
0
 
LVL 7

Expert Comment

by:franka
ID: 7007406
I don't think it hardware realted.

I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

Can you please provide more info?

Does IIS hang or really crash?

which error does IIS show after restart?
0
 

Author Comment

by:dustygulleson
ID: 7007548
HEY ALL:

I actually think we found the problem.  He goes:

Here is the IIS log from one of our web sites:

IIS Log File (CCCU.Org Only):
2002-05-13 16:13:50 209.17.75.51 - W3SVC8 ERWEB1 204.176.38.91 80 GET /pagenotfound.asp 404;http://www.cccu.org/<Rejected-By-UrlScan>?~/docLib 403 5 329 183 0 HTTP/1.1 www.cccu.org Microsoft-WebDAV-MiniRedir/5.1.2505 - -
2002-05-13 16:13:50 209.17.75.51 - W3SVC8 ERWEB1 204.176.38.91 80 GET /pagenotfound.asp 404;http://www.cccu.org/<Rejected-By-UrlScan>?~/docLib 403 5 329 183 0 HTTP/1.1 www.cccu.org Microsoft-WebDAV-MiniRedir/5.1.2505 - -
2002-05-13 16:13:50 209.17.75.51 - W3SVC8 ERWEB1 204.176.38.91 80 GET /pagenotfound.asp 404;http://www.cccu.org/<Rejected-By-UrlScan>?~/ 403 5 329 148 0 HTTP/1.1 www.cccu.org Microsoft-WebDAV-MiniRedir/5.1.2505 - -


The key thing to not is the " Microsoft-WebDAV-MiniRedir/5.1.2505."

WebDAV does not properly handle a particular type of specially malformed request. If a continuous stream of such requests were sent to an affected server, it could degrade the server’s performance to the point where it would be unable to perform useful work.  An attacker could use this vulnerability to temporarily disrupt service on an affected server. During such an attack, the server would be unable to service existing HTTP sessions or accept new ones.  This is strictly a denial of service vulnerability. There is no capability to use this vulnerability to compromise data on the system or to take any kind of administrative action on it.  WebDAV is installed by default on IIS 5 web servers, and has to be installed manually on IIS 4.  

So basically this was a type of DOS attack using the WebDAV vulnerability (only for DOS, no compramise).  

You can find out more about this at Microsoft.com:

http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q241520&ID=KB;EN-US;Q241520

This problem was solved by my senior developer after perusing line after line of logs.

I'll be back if this doesn't actually solve the problem.  However, we did trace the orginating DOS attack to Oakhurst, CA and have the local ISP going through their dial-up log to get the attackers name and address.

Cheers,
Dusty
0
 
LVL 7

Expert Comment

by:franka
ID: 7008111
if that WebDAV Denial of Service would work on your server, did you miss the security patches of the last months???
That issue is known and published at least one year.

0
 

Author Comment

by:dustygulleson
ID: 7008546
The security patches DO NOT remove WebDAV.  We thought it would have.  We had to HKEY the removal.
0
 
LVL 7

Expert Comment

by:franka
ID: 7008779
right, they didn't remove webdav (MS recommend those NTFS deny settings) but they remove that denial of service weakness of it
0
 

Author Comment

by:dustygulleson
ID: 7009066
I want to believe that, but we had all the latest hotfixes and security patches installed and this still took place.

Anyway, that doesn't seem to be the problem.  The server crashed just a little bit ago.

I've had it.  We are rebuilding it this weekend.

0
 
LVL 7

Expert Comment

by:franka
ID: 7009832
if there are only 3 or some more of these webdav requests that would be no problem.

can you please add some info, especially what I asked in my first comment?

are you using components?
0
 

Author Comment

by:dustygulleson
ID: 7010005
That log with the webDAV was an excerpt, not the whole log.

One CGI Component = United Binary's AutoImageEffects
http://www.unitedbinary.com/

Never had a problem until that one site cccu.org went live.  But UBB is being used for another site.
0
 
LVL 7

Expert Comment

by:franka
ID: 7010752
that's not not enough info...
please read my first comment.

btw: why don't you deny access to that cccu.org ip to get sure.

0
 
LVL 7

Expert Comment

by:franka
ID: 7010761
btw2: why still using pcanywhere instead of terminal services in admin mode?
0
 
LVL 7

Expert Comment

by:franka
ID: 7032380
still awainting more info from you!
0
 

Author Comment

by:dustygulleson
ID: 7032727
sorry for the delay.  please bear with me.  a lot of deadlines in these next two weeks.
0
 
LVL 7

Expert Comment

by:franka
ID: 7066049
2 weeks are over
0
 

Author Comment

by:dustygulleson
ID: 7066053
thanks franka.  

I AM SOOOOOO FRUSTRATED.

We bought a BRAND NEW dual 1.4 DELL with 1 GIG ram and put the one web site on it.  All the latest updates etc.

AND IT STILL HAS PROBLEMS.

The problem again in synopsis:
Occasionaly the site stops showing up or the timesout because the server dishes it up soooo slowly.  The site can be pinged but can not be browsed to.

When you view the event log you see NO errors.  When you view the performance log you see no problems either.

When we go to restart IIS the server freezes and requires a hard reboot.

What the heck!  Any ideas anyone.  I am raising the point value to 1000.

0
 

Author Comment

by:dustygulleson
ID: 7066054
I will add another 1000 points to anyone who can provide a solution.
0
 

Author Comment

by:dustygulleson
ID: 7066056
BTW:

This site, CCCU.org, is the ONLY web site on this box.
0
 
LVL 7

Expert Comment

by:franka
ID: 7066060
I repeat my first comment:

I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

Is it true? Are you admin or developer or both? please contact your developers!

Can you please provide more info?

Does IIS hang or really crash?

which error does IIS show after restart?

any security tab reports in your eventvwr?

Internal server errors 500? check the iis logs
0
 

Author Comment

by:dustygulleson
ID: 7066065
I repeat my first comment:

ANSWERS TO YOUR QUESTIONS:
I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"  Our developer SWEARS it's not the code since this code is running on another web page with 0% problem.

Can you please provide more info?

--> Tell me what you want beyond those addressed below.

Does IIS hang or really crash?

--> Hangs.

which error does IIS show after restart?

--> The following errors:

1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error: Incorrect function.
2.  SMTP: same as 1
3.  FTP: same as 1

The event recorded before the restart:
Event ID 7031, The IIS Admin service terminated unexpectedly...Run the configured recovery program.

any security tab reports in your eventvwr?

--> No security breaches.

Internal server errors 500? check the iis logs

--> No 500 errors.
0
 

Author Comment

by:dustygulleson
ID: 7066070
I repeat my first comment:

ANSWERS TO YOUR QUESTIONS:
I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"  Our developer SWEARS it's not the code since this code is running on another web page with 0% problem.

Can you please provide more info?

--> Tell me what you want beyond those addressed below.

Does IIS hang or really crash?

--> Hangs.

which error does IIS show after restart?

--> The following errors:

1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error: Incorrect function.
2.  SMTP: same as 1
3.  FTP: same as 1

The event recorded before the restart:
Event ID 7031, The IIS Admin service terminated unexpectedly...Run the configured recovery program.

any security tab reports in your eventvwr?

--> No security breaches.

Internal server errors 500? check the iis logs

--> No 500 errors.
0
 
LVL 7

Expert Comment

by:franka
ID: 7066074
--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"

it's not easy. any external references/includes that may delay?
checked the asp threadpool? increase it in the metabase.

--->it's not the code since this code is running on another web page with 0% problem.

but you! have the problem. is it possible that the traffic increased before the problem occured?

--> Hangs.

can you see any suspicious (asp) counters in perfmon?

--->1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error:
Incorrect function.

only "incorrect function"??? nothing more?

this is the next step you need to examine: a developer may help you. symbols are on SP cd if needed.

http://support.microsoft.com/support/kb/articles/Q286/3/50.ASP
0
 

Author Comment

by:dustygulleson
ID: 7066081
I repeat my first comment:

ANSWERS TO YOUR QUESTIONS:
I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"  Our developer SWEARS it's not the code since this code is running on another web page with 0% problem.

Can you please provide more info?

--> Tell me what you want beyond those addressed below.

Does IIS hang or really crash?

--> Hangs.

which error does IIS show after restart?

--> The following errors:

1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error: Incorrect function.
2.  SMTP: same as 1
3.  FTP: same as 1

The event recorded before the restart:
Event ID 7031, The IIS Admin service terminated unexpectedly...Run the configured recovery program.

any security tab reports in your eventvwr?

--> No security breaches.

Internal server errors 500? check the iis logs

--> No 500 errors.
0
 
LVL 7

Expert Comment

by:franka
ID: 7066589
3 times same comment?
0
 

Author Comment

by:dustygulleson
ID: 7066619
it's not me that's doing it, it seems like epxerts-exchange did it.
0
 
LVL 7

Expert Comment

by:franka
ID: 7071862
are you running any anti virus software on that server?
0
 
LVL 1

Expert Comment

by:Bird_Dog347
ID: 7990294
not sure if this is solved or not, nut have you turned on the iis debugger? We had that same problem a week ago. Turned on the debugger and it popped up an error every 5 to 10 min. (old asp guy did not know what he was doing) New asp guy has fixed 95 % of the errors and avent hung since. If it is fixed already then please disregard.
0
 

Author Comment

by:dustygulleson
ID: 7999573
Sorry about not replying to everyone, but we discovered the problem!  DO NOT USE ZONEALARM! As soon as we uninstalled ZoneAlarm we never had the problem with IIS hanging.  ZoneAlarm was causing a conflict with IIS causing it to hang.

Pass the word along.
0
 
LVL 7

Accepted Solution

by:
franka earned 500 total points
ID: 7999606
so isn't zone alarm very similar to anti-virus software... ;-)
0
 

Author Comment

by:dustygulleson
ID: 8001222
Franka,

Hehe....no.  ZoneAlarm is not anti-virus software, it's intrusion blocking software.

But, since I have to award someone, I guess it may as well be you!  Thanks!
0
 

Author Comment

by:dustygulleson
ID: 8001224
Closes one to the bulls-eye.
0
 

Expert Comment

by:Wrighteous1
ID: 8525117
I have heard of this problem recently. My company and some of our clients have experienced this problem.
There does not seem to be a fix for this problem. at least not one posted by MS.

As MS is well known for "WORKAROUNDS", I HOPE THIS ONE HELPS.
In the properties of the failed services on the windows 2000 server. you can configure the service to restart after the failure.

Go to-> SERVICES under the... "you know where it is".
GO TO the most important one, "I would select IIS Admin Service.since the services mentioned are dependecies of IIS" and double click or select properties.
GO TO-> the RECOVERY TAB.
The default is set to take no action. for the first, second and subsequent failures.
change it to restart the service. change ALL of them if you have frequent problems and or are paranoid.
select the restart time. 1 minute is the default and should be left as is.

I REALLY HOPE THIS HELPS. IT IS NOT A FIX, BUT IT SHOULD SAVE YOU A TRIP TO THE SERVER @ 2AM TO RESTART THE CEO'S E-MAIL.

Ken Wright

0

Featured Post

Free Tool: Postgres Monitoring System

A PHP and Perl based system to collect and display usage statistics from PostgreSQL databases.

One of a set of tools we are providing to everyone as a way of saying thank you for being a part of the community.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

NTFS file system has been developed by Microsoft that is widely used by Windows NT operating system and its advanced versions. It is the mostly used over FAT file system as it provides superior features like reliability, security, storage, efficienc…
In threads here at EE, each comment has a unique Identifier (ID). It is easy to get the full path for an ID via the right-click context menu. However, we often want to post a short link within a thread rather than the full link. This article shows a…
This video shows how to quickly and easily add an email signature for all users on Exchange 2016. The resulting signature is applied on a server level by Exchange Online. The email signature template has been downloaded from: www.mail-signatures…
The Email Laundry PDF encryption service allows companies to send confidential encrypted  emails to anybody. The PDF document can also contain attachments that are embedded in the encrypted PDF. The password is randomly generated by The Email Laundr…

821 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question