Go Premium for a chance to win a PS4. Enter to Win

x
  • Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 449
  • Last Modified:

WIN2k Web Server IIS crashes every day

It's got to be the most frustrating problem I have ever encountered.

Our WIN2k web server IIS is crashing every so often.  Sometimes it goes fine for 3-4 days and then 2-3 times a day.  The server doesn't crash, just IIS.  However, in the services it shows IIS running fine.  When we go to restart it comes up with an error.  The only way to fix the problem is to reboot the server.

We have been troubleshooting this for a whole month and no solution has been found.

PLEASE HELP.  As far as I am concerned this is worth a whooping 500 points to anyone who can figure this wierd anomoly out.
0
dustygulleson
Asked:
dustygulleson
  • 18
  • 13
  • 2
  • +5
1 Solution
 
jhanceCommented:
IIS and W2K are very reliable.  My guess is that you have a hardware problem:

1) Check your RAM and swap it out for some known good modules.

2) Make sure ALL hardware is W2K SERVER supported by Microsoft in the HCL.  Remove or replace any questionable items.

3) Make sure you have the proper driver versions for all your hardware.

4) Overclocking? Stop it.  Under-cooling?  Fix it.

5) 3rd party software?  Remove/disable it to eliminate it as a cause.

6) W2K SPs and HotFixes up to date?  Perhaps you have a hacker crashing your system.  Got a firewall?  Make sure non-required ports are blocked from public access.
0
 
dustygullesonAuthor Commented:
JHANCE,

My response:

1) Check your RAM and swap it out for some known good modules.
-- The hardware is a DELL WebApp 1 GiG Mghz, 1 GiG RAM, RAID 5 SCSI 76 Meg

2) Make sure ALL hardware is W2K SERVER supported by Microsoft in the HCL.  Remove or replace any questionable
items.
-- All DELL direct.  Brand new build.

3) Make sure you have the proper driver versions for all your hardware.
-- Have done a thorough check on all drivers.  Latest drivers are installed.

4) Overclocking? Stop it.  Under-cooling?  Fix it.
-- Haven't touched it. Ever.

5) 3rd party software?  Remove/disable it to eliminate it as a cause.
-- Have PCAnywhere, DeepMetrix Livestats 6.0 XSP (Web Traffic Analytics).

6) W2K SPs and HotFixes up to date?  Perhaps you have a hacker crashing your system.  Got a firewall? Make sure non-required ports are blocked from public access.

-- All HotFixes upto date.  Installing WatchGuard 700 this weekend.


Any other thoughts?
0
 
Corvax021899Commented:
Any event log errors?
I would remove or disable any none necessary software.  Just 2k And IIS to see if it's not a 3rd party software that is causing the problem.

0
[Webinar] Cloud and Mobile-First Strategy

Maybe you’ve fully adopted the cloud since the beginning. Or maybe you started with on-prem resources but are pursuing a “cloud and mobile first” strategy. Getting to that end state has its challenges. Discover how to build out a 100% cloud and mobile IT strategy in this webinar.

 
dustygullesonAuthor Commented:
Response to CORVAX

The event was the first thing we looked at.  It has nothing.  We initially saw a problem with the server having the inability to browse to the PDC.  We then removed the PDC altogther since it wasn't necassary.  I would be happy to zip up the event log to show you if your still interested.

I turned off the Livestats Web Analytics tool tonight and will track it.  I don't think this is the issue because we tested the deployment on our development server with no problems.

I will maintain PCAnywhere on it simply because it has no history of incompatibility with WIN2K servers AND I don't feel like driving the 30 minutes to the datacenter to do a change.
0
 
dustygullesonAuthor Commented:
JHANCE

How can you tell if you have bad RAM.  Is there a diagnostic tool you know about that can test it?

Also, how could bad RAM impact IIS?
0
 
pssiewCommented:
dusty,

your RAM is most probably ECC so it might not be RAM.

You can check dell's support site for diagnostic tools. They usually have one for servers

Are you running any websites off your IIS ? Rogue asp scripts can cause havoc with IIS.

Since you know its IIS causing the problem, I would suggest you put in either ADPlus or Exception Monitor to get a dump file on IIS, then debug it to see what is the cause if IIS crashing.

For ADPlus - http://support.microsoft.com/directory/article.asp?ID=KB;EN-US;Q286350&

For Exception Monitor - http://www.microsoft.com/technet/treeview/default.asp?url=/TechNet/prodtechnol/iis/downloads/ixcptmon.asp

Hope this helps
0
 
hnminhCommented:
"However, in the services it shows IIS running fine"
Did you mean services control panel had shown Web was running but actually it was not?

"When we go to restart it comes up with an error"
Could you please provide the error message?

Did you check server performance when we crashed? Anything eatup CPU resource?

How many connection at a time are allowed on the server?

Do you use ASP for the site? Do those ASP required other library (for example to connect to oracle DB or anything like that)....

Some idea that hope it help!
0
 
hnminhCommented:
oh, my my english :(. The 3rd question should be read "... performance when the server crashed?..."
0
 
frankaCommented:
I don't think it hardware realted.

I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

Can you please provide more info?

Does IIS hang or really crash?

which error does IIS show after restart?
0
 
dustygullesonAuthor Commented:
HEY ALL:

I actually think we found the problem.  He goes:

Here is the IIS log from one of our web sites:

IIS Log File (CCCU.Org Only):
2002-05-13 16:13:50 209.17.75.51 - W3SVC8 ERWEB1 204.176.38.91 80 GET /pagenotfound.asp 404;http://www.cccu.org/<Rejected-By-UrlScan>?~/docLib 403 5 329 183 0 HTTP/1.1 www.cccu.org Microsoft-WebDAV-MiniRedir/5.1.2505 - -
2002-05-13 16:13:50 209.17.75.51 - W3SVC8 ERWEB1 204.176.38.91 80 GET /pagenotfound.asp 404;http://www.cccu.org/<Rejected-By-UrlScan>?~/docLib 403 5 329 183 0 HTTP/1.1 www.cccu.org Microsoft-WebDAV-MiniRedir/5.1.2505 - -
2002-05-13 16:13:50 209.17.75.51 - W3SVC8 ERWEB1 204.176.38.91 80 GET /pagenotfound.asp 404;http://www.cccu.org/<Rejected-By-UrlScan>?~/ 403 5 329 148 0 HTTP/1.1 www.cccu.org Microsoft-WebDAV-MiniRedir/5.1.2505 - -


The key thing to not is the " Microsoft-WebDAV-MiniRedir/5.1.2505."

WebDAV does not properly handle a particular type of specially malformed request. If a continuous stream of such requests were sent to an affected server, it could degrade the server’s performance to the point where it would be unable to perform useful work.  An attacker could use this vulnerability to temporarily disrupt service on an affected server. During such an attack, the server would be unable to service existing HTTP sessions or accept new ones.  This is strictly a denial of service vulnerability. There is no capability to use this vulnerability to compromise data on the system or to take any kind of administrative action on it.  WebDAV is installed by default on IIS 5 web servers, and has to be installed manually on IIS 4.  

So basically this was a type of DOS attack using the WebDAV vulnerability (only for DOS, no compramise).  

You can find out more about this at Microsoft.com:

http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q241520&ID=KB;EN-US;Q241520

This problem was solved by my senior developer after perusing line after line of logs.

I'll be back if this doesn't actually solve the problem.  However, we did trace the orginating DOS attack to Oakhurst, CA and have the local ISP going through their dial-up log to get the attackers name and address.

Cheers,
Dusty
0
 
frankaCommented:
if that WebDAV Denial of Service would work on your server, did you miss the security patches of the last months???
That issue is known and published at least one year.

0
 
dustygullesonAuthor Commented:
The security patches DO NOT remove WebDAV.  We thought it would have.  We had to HKEY the removal.
0
 
frankaCommented:
right, they didn't remove webdav (MS recommend those NTFS deny settings) but they remove that denial of service weakness of it
0
 
dustygullesonAuthor Commented:
I want to believe that, but we had all the latest hotfixes and security patches installed and this still took place.

Anyway, that doesn't seem to be the problem.  The server crashed just a little bit ago.

I've had it.  We are rebuilding it this weekend.

0
 
frankaCommented:
if there are only 3 or some more of these webdav requests that would be no problem.

can you please add some info, especially what I asked in my first comment?

are you using components?
0
 
dustygullesonAuthor Commented:
That log with the webDAV was an excerpt, not the whole log.

One CGI Component = United Binary's AutoImageEffects
http://www.unitedbinary.com/

Never had a problem until that one site cccu.org went live.  But UBB is being used for another site.
0
 
frankaCommented:
that's not not enough info...
please read my first comment.

btw: why don't you deny access to that cccu.org ip to get sure.

0
 
frankaCommented:
btw2: why still using pcanywhere instead of terminal services in admin mode?
0
 
frankaCommented:
still awainting more info from you!
0
 
dustygullesonAuthor Commented:
sorry for the delay.  please bear with me.  a lot of deadlines in these next two weeks.
0
 
frankaCommented:
2 weeks are over
0
 
dustygullesonAuthor Commented:
thanks franka.  

I AM SOOOOOO FRUSTRATED.

We bought a BRAND NEW dual 1.4 DELL with 1 GIG ram and put the one web site on it.  All the latest updates etc.

AND IT STILL HAS PROBLEMS.

The problem again in synopsis:
Occasionaly the site stops showing up or the timesout because the server dishes it up soooo slowly.  The site can be pinged but can not be browsed to.

When you view the event log you see NO errors.  When you view the performance log you see no problems either.

When we go to restart IIS the server freezes and requires a hard reboot.

What the heck!  Any ideas anyone.  I am raising the point value to 1000.

0
 
dustygullesonAuthor Commented:
I will add another 1000 points to anyone who can provide a solution.
0
 
dustygullesonAuthor Commented:
BTW:

This site, CCCU.org, is the ONLY web site on this box.
0
 
frankaCommented:
I repeat my first comment:

I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

Is it true? Are you admin or developer or both? please contact your developers!

Can you please provide more info?

Does IIS hang or really crash?

which error does IIS show after restart?

any security tab reports in your eventvwr?

Internal server errors 500? check the iis logs
0
 
dustygullesonAuthor Commented:
I repeat my first comment:

ANSWERS TO YOUR QUESTIONS:
I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"  Our developer SWEARS it's not the code since this code is running on another web page with 0% problem.

Can you please provide more info?

--> Tell me what you want beyond those addressed below.

Does IIS hang or really crash?

--> Hangs.

which error does IIS show after restart?

--> The following errors:

1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error: Incorrect function.
2.  SMTP: same as 1
3.  FTP: same as 1

The event recorded before the restart:
Event ID 7031, The IIS Admin service terminated unexpectedly...Run the configured recovery program.

any security tab reports in your eventvwr?

--> No security breaches.

Internal server errors 500? check the iis logs

--> No 500 errors.
0
 
dustygullesonAuthor Commented:
I repeat my first comment:

ANSWERS TO YOUR QUESTIONS:
I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"  Our developer SWEARS it's not the code since this code is running on another web page with 0% problem.

Can you please provide more info?

--> Tell me what you want beyond those addressed below.

Does IIS hang or really crash?

--> Hangs.

which error does IIS show after restart?

--> The following errors:

1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error: Incorrect function.
2.  SMTP: same as 1
3.  FTP: same as 1

The event recorded before the restart:
Event ID 7031, The IIS Admin service terminated unexpectedly...Run the configured recovery program.

any security tab reports in your eventvwr?

--> No security breaches.

Internal server errors 500? check the iis logs

--> No 500 errors.
0
 
frankaCommented:
--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"

it's not easy. any external references/includes that may delay?
checked the asp threadpool? increase it in the metabase.

--->it's not the code since this code is running on another web page with 0% problem.

but you! have the problem. is it possible that the traffic increased before the problem occured?

--> Hangs.

can you see any suspicious (asp) counters in perfmon?

--->1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error:
Incorrect function.

only "incorrect function"??? nothing more?

this is the next step you need to examine: a developer may help you. symbols are on SP cd if needed.

http://support.microsoft.com/support/kb/articles/Q286/3/50.ASP
0
 
dustygullesonAuthor Commented:
I repeat my first comment:

ANSWERS TO YOUR QUESTIONS:
I would bet, it's some self written VB or dirty COM Objects with no error handling used by ASP.

--> No COM object involved.  100% ASP.  Any quick way of finding "dirty code?"  Our developer SWEARS it's not the code since this code is running on another web page with 0% problem.

Can you please provide more info?

--> Tell me what you want beyond those addressed below.

Does IIS hang or really crash?

--> Hangs.

which error does IIS show after restart?

--> The following errors:

1.  Service Control Manager, Event ID 7023, Content: WWW service terminated due to following error: Incorrect function.
2.  SMTP: same as 1
3.  FTP: same as 1

The event recorded before the restart:
Event ID 7031, The IIS Admin service terminated unexpectedly...Run the configured recovery program.

any security tab reports in your eventvwr?

--> No security breaches.

Internal server errors 500? check the iis logs

--> No 500 errors.
0
 
frankaCommented:
3 times same comment?
0
 
dustygullesonAuthor Commented:
it's not me that's doing it, it seems like epxerts-exchange did it.
0
 
frankaCommented:
are you running any anti virus software on that server?
0
 
Bird_Dog347Commented:
not sure if this is solved or not, nut have you turned on the iis debugger? We had that same problem a week ago. Turned on the debugger and it popped up an error every 5 to 10 min. (old asp guy did not know what he was doing) New asp guy has fixed 95 % of the errors and avent hung since. If it is fixed already then please disregard.
0
 
dustygullesonAuthor Commented:
Sorry about not replying to everyone, but we discovered the problem!  DO NOT USE ZONEALARM! As soon as we uninstalled ZoneAlarm we never had the problem with IIS hanging.  ZoneAlarm was causing a conflict with IIS causing it to hang.

Pass the word along.
0
 
frankaCommented:
so isn't zone alarm very similar to anti-virus software... ;-)
0
 
dustygullesonAuthor Commented:
Franka,

Hehe....no.  ZoneAlarm is not anti-virus software, it's intrusion blocking software.

But, since I have to award someone, I guess it may as well be you!  Thanks!
0
 
dustygullesonAuthor Commented:
Closes one to the bulls-eye.
0
 
Wrighteous1Commented:
I have heard of this problem recently. My company and some of our clients have experienced this problem.
There does not seem to be a fix for this problem. at least not one posted by MS.

As MS is well known for "WORKAROUNDS", I HOPE THIS ONE HELPS.
In the properties of the failed services on the windows 2000 server. you can configure the service to restart after the failure.

Go to-> SERVICES under the... "you know where it is".
GO TO the most important one, "I would select IIS Admin Service.since the services mentioned are dependecies of IIS" and double click or select properties.
GO TO-> the RECOVERY TAB.
The default is set to take no action. for the first, second and subsequent failures.
change it to restart the service. change ALL of them if you have frequent problems and or are paranoid.
select the restart time. 1 minute is the default and should be left as is.

I REALLY HOPE THIS HELPS. IT IS NOT A FIX, BUT IT SHOULD SAVE YOU A TRIP TO THE SERVER @ 2AM TO RESTART THE CEO'S E-MAIL.

Ken Wright

0

Featured Post

Hire Technology Freelancers with Gigs

Work with freelancers specializing in everything from database administration to programming, who have proven themselves as experts in their field. Hire the best, collaborate easily, pay securely, and get projects done right.

  • 18
  • 13
  • 2
  • +5
Tackle projects and never again get stuck behind a technical roadblock.
Join Now