Solved

WIN2k Web Server IIS crashes every day

Posted on 2002-05-08
38
403 Views
Last Modified: 2008-02-26
It's got to be the most frustrating problem I have ever encountered.

Our WIN2k web server IIS is crashing every so often.  Sometimes it goes fine for 3-4 days and then 2-3 times a day.  The server doesn't crash, just IIS.  However, in the services it shows IIS running fine.  When we go to restart it comes up with an error.  The only way to fix the problem is to reboot the server.

We have been troubleshooting this for a whole month and no solution has been found.

PLEASE HELP.  As far as I am concerned this is worth a whooping 500 points to anyone who can figure this wierd anomoly out.
0
Comment
Question by:dustygulleson
  • 18
  • 13
  • 2
  • +5
38 Comments
 
LVL 32

Expert Comment

by:jhance
Comment Utility
IIS and W2K are very reliable.  My guess is that you have a hardware problem:

1) Check your RAM and swap it out for some known good modules.

2) Make sure ALL hardware is W2K SERVER supported by Microsoft in the HCL.  Remove or replace any questionable items.

3) Make sure you have the proper driver versions for all your hardware.

4) Overclocking? Stop it.  Under-cooling?  Fix it.

5) 3rd party software?  Remove/disable it to eliminate it as a cause.

6) W2K SPs and HotFixes up to date?  Perhaps you have a hacker crashing your system.  Got a firewall?  Make sure non-required ports are blocked from public access.
0
 

Author Comment

by:dustygulleson
Comment Utility
JHANCE,

My response:

1) Check your RAM and swap it out for some known good modules.
-- The hardware is a DELL WebApp 1 GiG Mghz, 1 GiG RAM, RAID 5 SCSI 76 Meg

2) Make sure ALL hardware is W2K SERVER supported by Microsoft in the HCL.  Remove or replace any questionable
items.
-- All DELL direct.  Brand new build.

3) Make sure you have the proper driver versions for all your hardware.
-- Have done a thorough check on all drivers.  Latest drivers are installed.

4) Overclocking? Stop it.  Under-cooling?  Fix it.
-- Haven't touched it. Ever.

5) 3rd party software?  Remove/disable it to eliminate it as a cause.
-- Have PCAnywhere, DeepMetrix Livestats 6.0 XSP (Web Traffic Analytics).

6) W2K SPs and HotFixes up to date?  Perhaps you have a hacker crashing your system.  Got a firewall? Make sure non-required ports are blocked from public access.

-- All HotFixes upto date.  Installing WatchGuard 700 this weekend.


Any other thoughts?
0
 
LVL 3

Expert Comment

by:Corvax021899
Comment Utility
Any event log errors?
I would remove or disable any none necessary software.  Just 2k And IIS to see if it's not a 3rd party software that is causing the problem.

0
 

Author Comment

by:dustygulleson
Comment Utility
Response to CORVAX

The event was the first thing we looked at.  It has nothing.  We initially saw a problem with the server having the inability to browse to the PDC.  We then removed the PDC altogther since it wasn't necassary.  I would be happy to zip up the event log to show you if your still interested.

I turned off the Livestats Web Analytics tool tonight and will track it.  I don't think this is the issue because we tested the deployment on our development server with no problems.

I will maintain PCAnywhere on it simply because it has no history of incompatibility with WIN2K servers AND I don't feel like driving the 30 minutes to the datacenter to do a change.
0
 

Author Comment

by:dustygulleson
Comment Utility
JHANCE

How can you tell if you have bad RAM.  Is there a diagnostic tool you know about that can test it?

Also, how could bad RAM impact IIS?
0
 
LVL 2

Expert Comment

by:pssiew
Comment Utility
dusty,

your RAM is most probably ECC so it might not be RAM.

You can check dell's support site for diagnostic tools. They usually have one for servers

Are you running any websites off your IIS ? Rogue asp scripts can cause havoc with IIS.

Since you know its IIS causing the problem, I would suggest you put in either ADPlus or Exception Monitor to get a dump file on IIS, then debug it to see what is the cause if IIS crashing.

For ADPlus - http://support.microsoft.com/directory/article.asp?ID=KB;EN-US;Q286350&

For Exception Monitor - http://www.microsoft.com/technet/treeview/default.asp?url=/TechNet/prodtechnol/iis/downloads/ixcptmon.asp

Hope this helps
0
 
LVL 3

Expert Comment

by:hnminh
Comment Utility
"However, in the services it shows IIS running fine"
Did you mean services control panel had shown Web was running but actually it was not?

"When we go to restart it comes up with an error"
Could you please provide the error message?

Did you check server performance when we crashed? Anything eatup CPU resource?

How many connection at a time are allowed on the server?

Do you use ASP for the site? Do those ASP required other library (for example to connect to oracle DB or anything like that)....

Some idea that hope it help!
0
 
LVL 3

Expert Comment

by:hnminh
Comment Utility
oh, my my english :(. The 3rd question should be read "... performance when the server crashed?..."
0
 
LVL 7

Expert Comment

by:franka
Comment Utility
I don't think it hardware realted.

I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

Can you please provide more info?

Does IIS hang or really crash?

which error does IIS show after restart?
0
 

Author Comment

by:dustygulleson
Comment Utility
HEY ALL:

I actually think we found the problem.  He goes:

Here is the IIS log from one of our web sites:

IIS Log File (CCCU.Org Only):
2002-05-13 16:13:50 209.17.75.51 - W3SVC8 ERWEB1 204.176.38.91 80 GET /pagenotfound.asp 404;http://www.cccu.org/<Rejected-By-UrlScan>?~/docLib 403 5 329 183 0 HTTP/1.1 www.cccu.org Microsoft-WebDAV-MiniRedir/5.1.2505 - -
2002-05-13 16:13:50 209.17.75.51 - W3SVC8 ERWEB1 204.176.38.91 80 GET /pagenotfound.asp 404;http://www.cccu.org/<Rejected-By-UrlScan>?~/docLib 403 5 329 183 0 HTTP/1.1 www.cccu.org Microsoft-WebDAV-MiniRedir/5.1.2505 - -
2002-05-13 16:13:50 209.17.75.51 - W3SVC8 ERWEB1 204.176.38.91 80 GET /pagenotfound.asp 404;http://www.cccu.org/<Rejected-By-UrlScan>?~/ 403 5 329 148 0 HTTP/1.1 www.cccu.org Microsoft-WebDAV-MiniRedir/5.1.2505 - -


The key thing to not is the " Microsoft-WebDAV-MiniRedir/5.1.2505."

WebDAV does not properly handle a particular type of specially malformed request. If a continuous stream of such requests were sent to an affected server, it could degrade the server’s performance to the point where it would be unable to perform useful work.  An attacker could use this vulnerability to temporarily disrupt service on an affected server. During such an attack, the server would be unable to service existing HTTP sessions or accept new ones.  This is strictly a denial of service vulnerability. There is no capability to use this vulnerability to compromise data on the system or to take any kind of administrative action on it.  WebDAV is installed by default on IIS 5 web servers, and has to be installed manually on IIS 4.  

So basically this was a type of DOS attack using the WebDAV vulnerability (only for DOS, no compramise).  

You can find out more about this at Microsoft.com:

http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q241520&ID=KB;EN-US;Q241520

This problem was solved by my senior developer after perusing line after line of logs.

I'll be back if this doesn't actually solve the problem.  However, we did trace the orginating DOS attack to Oakhurst, CA and have the local ISP going through their dial-up log to get the attackers name and address.

Cheers,
Dusty
0
 
LVL 7

Expert Comment

by:franka
Comment Utility
if that WebDAV Denial of Service would work on your server, did you miss the security patches of the last months???
That issue is known and published at least one year.

0
 

Author Comment

by:dustygulleson
Comment Utility
The security patches DO NOT remove WebDAV.  We thought it would have.  We had to HKEY the removal.
0
 
LVL 7

Expert Comment

by:franka
Comment Utility
right, they didn't remove webdav (MS recommend those NTFS deny settings) but they remove that denial of service weakness of it
0
 

Author Comment

by:dustygulleson
Comment Utility
I want to believe that, but we had all the latest hotfixes and security patches installed and this still took place.

Anyway, that doesn't seem to be the problem.  The server crashed just a little bit ago.

I've had it.  We are rebuilding it this weekend.

0
 
LVL 7

Expert Comment

by:franka
Comment Utility
if there are only 3 or some more of these webdav requests that would be no problem.

can you please add some info, especially what I asked in my first comment?

are you using components?
0
 

Author Comment

by:dustygulleson
Comment Utility
That log with the webDAV was an excerpt, not the whole log.

One CGI Component = United Binary's AutoImageEffects
http://www.unitedbinary.com/

Never had a problem until that one site cccu.org went live.  But UBB is being used for another site.
0
 
LVL 7

Expert Comment

by:franka
Comment Utility
that's not not enough info...
please read my first comment.

btw: why don't you deny access to that cccu.org ip to get sure.

0
 
LVL 7

Expert Comment

by:franka
Comment Utility
btw2: why still using pcanywhere instead of terminal services in admin mode?
0
 
LVL 7

Expert Comment

by:franka
Comment Utility
still awainting more info from you!
0
Enabling OSINT in Activity Based Intelligence

Activity based intelligence (ABI) requires access to all available sources of data. Recorded Future allows analysts to observe structured data on the open, deep, and dark web.

 

Author Comment

by:dustygulleson
Comment Utility
sorry for the delay.  please bear with me.  a lot of deadlines in these next two weeks.
0
 
LVL 7

Expert Comment

by:franka
Comment Utility
2 weeks are over
0
 

Author Comment

by:dustygulleson
Comment Utility
thanks franka.  

I AM SOOOOOO FRUSTRATED.

We bought a BRAND NEW dual 1.4 DELL with 1 GIG ram and put the one web site on it.  All the latest updates etc.

AND IT STILL HAS PROBLEMS.

The problem again in synopsis:
Occasionaly the site stops showing up or the timesout because the server dishes it up soooo slowly.  The site can be pinged but can not be browsed to.

When you view the event log you see NO errors.  When you view the performance log you see no problems either.

When we go to restart IIS the server freezes and requires a hard reboot.

What the heck!  Any ideas anyone.  I am raising the point value to 1000.

0
 

Author Comment

by:dustygulleson
Comment Utility
I will add another 1000 points to anyone who can provide a solution.
0
 

Author Comment

by:dustygulleson
Comment Utility
BTW:

This site, CCCU.org, is the ONLY web site on this box.
0
 
LVL 7

Expert Comment

by:franka
Comment Utility
I repeat my first comment:

I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

Is it true? Are you admin or developer or both? please contact your developers!

Can you please provide more info?

Does IIS hang or really crash?

which error does IIS show after restart?

any security tab reports in your eventvwr?

Internal server errors 500? check the iis logs
0
 

Author Comment

by:dustygulleson
Comment Utility
I repeat my first comment:

ANSWERS TO YOUR QUESTIONS:
I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"  Our developer SWEARS it's not the code since this code is running on another web page with 0% problem.

Can you please provide more info?

--> Tell me what you want beyond those addressed below.

Does IIS hang or really crash?

--> Hangs.

which error does IIS show after restart?

--> The following errors:

1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error: Incorrect function.
2.  SMTP: same as 1
3.  FTP: same as 1

The event recorded before the restart:
Event ID 7031, The IIS Admin service terminated unexpectedly...Run the configured recovery program.

any security tab reports in your eventvwr?

--> No security breaches.

Internal server errors 500? check the iis logs

--> No 500 errors.
0
 

Author Comment

by:dustygulleson
Comment Utility
I repeat my first comment:

ANSWERS TO YOUR QUESTIONS:
I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"  Our developer SWEARS it's not the code since this code is running on another web page with 0% problem.

Can you please provide more info?

--> Tell me what you want beyond those addressed below.

Does IIS hang or really crash?

--> Hangs.

which error does IIS show after restart?

--> The following errors:

1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error: Incorrect function.
2.  SMTP: same as 1
3.  FTP: same as 1

The event recorded before the restart:
Event ID 7031, The IIS Admin service terminated unexpectedly...Run the configured recovery program.

any security tab reports in your eventvwr?

--> No security breaches.

Internal server errors 500? check the iis logs

--> No 500 errors.
0
 
LVL 7

Expert Comment

by:franka
Comment Utility
--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"

it's not easy. any external references/includes that may delay?
checked the asp threadpool? increase it in the metabase.

--->it's not the code since this code is running on another web page with 0% problem.

but you! have the problem. is it possible that the traffic increased before the problem occured?

--> Hangs.

can you see any suspicious (asp) counters in perfmon?

--->1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error:
Incorrect function.

only "incorrect function"??? nothing more?

this is the next step you need to examine: a developer may help you. symbols are on SP cd if needed.

http://support.microsoft.com/support/kb/articles/Q286/3/50.ASP
0
 

Author Comment

by:dustygulleson
Comment Utility
I repeat my first comment:

ANSWERS TO YOUR QUESTIONS:
I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"  Our developer SWEARS it's not the code since this code is running on another web page with 0% problem.

Can you please provide more info?

--> Tell me what you want beyond those addressed below.

Does IIS hang or really crash?

--> Hangs.

which error does IIS show after restart?

--> The following errors:

1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error: Incorrect function.
2.  SMTP: same as 1
3.  FTP: same as 1

The event recorded before the restart:
Event ID 7031, The IIS Admin service terminated unexpectedly...Run the configured recovery program.

any security tab reports in your eventvwr?

--> No security breaches.

Internal server errors 500? check the iis logs

--> No 500 errors.
0
 
LVL 7

Expert Comment

by:franka
Comment Utility
3 times same comment?
0
 

Author Comment

by:dustygulleson
Comment Utility
it's not me that's doing it, it seems like epxerts-exchange did it.
0
 
LVL 7

Expert Comment

by:franka
Comment Utility
are you running any anti virus software on that server?
0
 
LVL 1

Expert Comment

by:Bird_Dog347
Comment Utility
not sure if this is solved or not, nut have you turned on the iis debugger? We had that same problem a week ago. Turned on the debugger and it popped up an error every 5 to 10 min. (old asp guy did not know what he was doing) New asp guy has fixed 95 % of the errors and avent hung since. If it is fixed already then please disregard.
0
 

Author Comment

by:dustygulleson
Comment Utility
Sorry about not replying to everyone, but we discovered the problem!  DO NOT USE ZONEALARM! As soon as we uninstalled ZoneAlarm we never had the problem with IIS hanging.  ZoneAlarm was causing a conflict with IIS causing it to hang.

Pass the word along.
0
 
LVL 7

Accepted Solution

by:
franka earned 500 total points
Comment Utility
so isn't zone alarm very similar to anti-virus software... ;-)
0
 

Author Comment

by:dustygulleson
Comment Utility
Franka,

Hehe....no.  ZoneAlarm is not anti-virus software, it's intrusion blocking software.

But, since I have to award someone, I guess it may as well be you!  Thanks!
0
 

Author Comment

by:dustygulleson
Comment Utility
Closes one to the bulls-eye.
0
 

Expert Comment

by:Wrighteous1
Comment Utility
I have heard of this problem recently. My company and some of our clients have experienced this problem.
There does not seem to be a fix for this problem. at least not one posted by MS.

As MS is well known for "WORKAROUNDS", I HOPE THIS ONE HELPS.
In the properties of the failed services on the windows 2000 server. you can configure the service to restart after the failure.

Go to-> SERVICES under the... "you know where it is".
GO TO the most important one, "I would select IIS Admin Service.since the services mentioned are dependecies of IIS" and double click or select properties.
GO TO-> the RECOVERY TAB.
The default is set to take no action. for the first, second and subsequent failures.
change it to restart the service. change ALL of them if you have frequent problems and or are paranoid.
select the restart time. 1 minute is the default and should be left as is.

I REALLY HOPE THIS HELPS. IT IS NOT A FIX, BUT IT SHOULD SAVE YOU A TRIP TO THE SERVER @ 2AM TO RESTART THE CEO'S E-MAIL.

Ken Wright

0

Featured Post

Get up to 2TB FREE CLOUD per backup license!

An exclusive Black Friday offer just for Expert Exchange audience! Buy any of our top-rated backup solutions & get up to 2TB free cloud per system! Perform local & cloud backup in the same step, and restore instantly—anytime, anywhere. Grab this deal now before it disappears!

Join & Write a Comment

NTFS file system has been developed by Microsoft that is widely used by Windows NT operating system and its advanced versions. It is the mostly used over FAT file system as it provides superior features like reliability, security, storage, efficienc…
In this article, you will read about the trends across the human resources departments for the upcoming year. Some of them include improving employee experience, adopting new technologies, using HR software to its full extent, and integrating artifi…
Internet Business Fax to Email Made Easy - With eFax Corporate (http://www.enterprise.efax.com), you'll receive a dedicated online fax number, which is used the same way as a typical analog fax number. You'll receive secure faxes in your email, fr…
In this tutorial you'll learn about bandwidth monitoring with flows and packet sniffing with our network monitoring solution PRTG Network Monitor (https://www.paessler.com/prtg). If you're interested in additional methods for monitoring bandwidt…

762 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

6 Experts available now in Live!

Get 1:1 Help Now