IIS Stops displaying web pages even though it appears to be functioning correctly

Hi Jeff.

It may well be network related but I have so far found nothing wrong with DNS, DHCP or any other of the networking components. It is bizare that it suddenly starts after several months of perfect use and occurs every 8 days 2 hours without fail. Here is the full ipconfig (I have removed the domain name for client confidentiality)

C:\Documents and Settings\Administrator>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : server
   Primary Dns Suffix  . . . . . . . : contoso.local
   Node Type . . . . . . . . . . . . : Unknown
   IP Routing Enabled. . . . . . . . : Yes
   WINS Proxy Enabled. . . . . . . . : Yes
   DNS Suffix Search List. . . . . . : contoso.local

PPP adapter RAS Server (Dial In) Interface:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : WAN (PPP/SLIP) Interface
   Physical Address. . . . . . . . . : 00-53-45-00-00-00
   DHCP Enabled. . . . . . . . . . . : No
   IP Address. . . . . . . . . . . . : 192.168.10.27
   Subnet Mask . . . . . . . . . . . : 255.255.255.255
   Default Gateway . . . . . . . . . :
   NetBIOS over Tcpip. . . . . . . . : Disabled

Ethernet adapter Server Local Area Connection:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : HP NC320i PCIe Gigabit Server Adapter
   Physical Address. . . . . . . . . : 00-1C-C4-39-5C-18
   DHCP Enabled. . . . . . . . . . . : No
   IP Address. . . . . . . . . . . . : 192.168.10.10
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 192.168.10.1
   DNS Servers . . . . . . . . . . . : 192.168.10.10
   Primary WINS Server . . . . . . . : 192.168.10.10

C:\Documents and Settings\Administrator>

Have you run the Configure Email and Internet Connection Wizard (CEICW -- which is linked as Connect to the Internet in the Server Management Console > To-Do List) without error?

A visual how-to for the CEICW is here:  http://sbsurl.com/ceicw

Jeff
TechSoEasy

Yes, this runs absolutetely fine on all the servers. We have tried re-running this several times and modirying settings to no avial.

Can you please post the output from running SYSTEMINFO at a command prompt?

Thanks.

Jeff
TechSoEasy

Host Name:                 SERVER
OS Name:                   Microsoft(R) Windows(R) Server 2003 for Small Business Server
OS Version:                5.2.3790 Service Pack 2 Build 3790
OS Manufacturer:           Microsoft Corporation
OS Configuration:          Primary Domain Controller
OS Build Type:             Multiprocessor Free
Registered Owner:          <removed>
Registered Organization:   <removed>
Product ID:                74995-OEM-4213137-50009
Original Install Date:     08/10/2007, 18:17:29
System Up Time:            5 Days, 21 Hours, 12 Minutes, 51 Seconds
System Manufacturer:       HP
System Model:              ProLiant ML310 G4
System Type:               X86-based PC
Processor(s):              2 Processor(s) Installed.
                           [01]: x86 Family 15 Model 6 Stepping 5 GenuineIntel ~3400 Mhz
                           [02]: x86 Family 15 Model 6 Stepping 5 GenuineIntel ~3400 Mhz
BIOS Version:              HP     - 2
Windows Directory:         C:\WINDOWS
System Directory:          C:\WINDOWS\system32
Boot Device:               \Device\HarddiskVolume1
System Locale:             en-gb;English (United Kingdom)
Input Locale:              en-gb;English (United Kingdom)
Time Zone:                 (GMT) Greenwich Mean Time : Dublin, Edinburgh, Lisbon
, London
Total Physical Memory:     2,047 MB
Available Physical Memory: 167 MB
Page File: Max Size:       3,506 MB
Page File: Available:      1,545 MB
Page File: In Use:         1,961 MB
Page File Location(s):     C:\pagefile.sys
Domain:                    <removed>
Logon Server:              \\SERVER
Hotfix(s):                 73 Hotfix(s) Installed.
                           [01]: File 1
                           [02]: File 1
                           [03]: File 1
                           [04]: File 1
                           [05]: File 1
                           [06]: File 1
                           [07]: File 1
                           [08]: File 1
                           [09]: File 1
                           [10]: File 1
                           [11]: File 1
                           [12]: File 1
                           [13]: File 1
                           [14]: File 1
                           [15]: File 1
                           [16]: File 1
                           [17]: File 1
                           [18]: File 1
                           [19]: File 1
                           [20]: File 1
                           [21]: File 1
                           [22]: File 1
                           [23]: File 1
                           [24]: File 1
                           [25]: File 1
                           [26]: File 1
                           [27]: File 1
                           [28]: File 1
                           [29]: File 1
                           [30]: File 1
                           [31]: File 1
                           [32]: Q147222
                           [33]: KB933854 - QFE
                           [34]: KB912442 - Update
                           [35]: KB916803 - Update
                           [36]: KB924334 - Update
                           [37]: KB926666 - Update
                           [38]: KB931832 - Update
                           [39]: KB928365 - Update
                           [40]: Q936181
                           [41]: KB925398_WMP64
                           [42]: KB921503 - Update
                           [43]: KB924667-v2 - Update
                           [44]: KB925876 - Update
                           [45]: KB925902 - Update
                           [46]: KB926122 - Update
                           [47]: KB927891 - Update
                           [48]: KB929123 - Update
                           [49]: KB930178 - Update
                           [50]: KB931784 - Update
                           [51]: KB932168 - Update
                           [52]: KB933360 - Update
                           [53]: KB933714 - Update
                           [54]: KB933729 - Update
                           [55]: KB933854 - Update
                           [56]: KB935839 - Update
                           [57]: KB935840 - Update
                           [58]: KB935966 - Update
                           [59]: KB936021 - Update
                           [60]: KB936357 - Update
                           [61]: KB936594 - Update
                           [62]: KB936782 - Update
                           [63]: KB938127 - Update
                           [64]: KB939653 - Update
                           [65]: KB941202 - Update
                           [66]: KB941568 - Update
                           [67]: KB941569 - Update
                           [68]: KB941672 - Update
                           [69]: KB942615 - Update
                           [70]: KB942840 - Update
                           [71]: KB943460 - Update
                           [72]: KB944653 - Update
                           [73]: XpsEPSC
Network Card(s):           1 NIC(s) Installed.
                           [01]: HP NC320i PCIe Gigabit Server Adapter
                                 Connection Name: Server Local Area Connection
                                 DHCP Enabled:    No
                                 IP address(es)
                                 [01]: 192.168.10.10

C:\Documents and Settings\Administrator>

It's interesting that this particular machine had SP2 integrated into it's installation media... which I thought may help you.  

But what's even more interesting is the 8day/2hr interval.  This is screaming DHCP to me... so I would have to ask if DHCP is running on the SBS or is it running on a separate device (router)?

Also, if you look in the default web site properties > Web Site Tab > Advanced, is there an entry for both "Default" and "127.0.0.1" for port 80?

Jeff
TechSoEasy

Hi Jeff,

Thanks for the replies! Last night I had exactly the same thought about DHCP and have modified the lease time to 4 days. It will be interseting to see if this makes any difference and obviously I will let you know if it does (But I will need to wait a few days to check that one)

I checked the advanced tab and there was an entry for default but not 127.0.0.1. This I have now added.

As you can see, this one is a bit of a nightmare! I really don't want to have to go to two clients and tell them we need to format their servers but so far that's the only solution. I will keep you up to date on those changes.

Where is your DHCP Server running from though?

Jeff
TechSoEasy

Ah, sorry! It's the inbuilt SBS agent. The router DHCP has been disabled.

Just in case that's not the fix... I had forgotten to make sure you reviewed all of the networking issues post-SP2 (some of which are also post-SP1):  http://sbsurl.com/sp2

Specifically this one:
http://blogs.technet.com/sbs/archive/2007/04/24/common-networking-issues-after-applying-windows-server-2003-sp2-on-sbs.aspx

Jeff
TechSoEasy

The DHCP modification seems to have made no difference to the 8 days. I am assuming this means it's just a coincidence and is not nercesserily related to DHCP.

In regards to the networking issues. I have read through that link and most of the stuff we have already tried. The NIC has the latest drivers, I have made the modification to the registry to disable RSS and TaskOffLoad and even applied a hotfix that was supposed to resolve those issues (936594). I have not tried making any changes to the NIC properties but i will try that next and report back in 8 days!

By chance is Limewire installed anywhere on your network?  Be thorough in looking for it.

Jeff
TechSoEasy

I have looked hard for limewire and don't believe it is running on the network. I can also guarantee it wasn't on our network we we had the issue.

There has however been an interesting turn of events. Every other occasion this has occured we have logged in a rebooted the server within 24 hours at the most. On this occasion I decided to leave it to see what happened. IIS went down as usual after 8 days on Wednesday 16th at 1am. Today, Friday 18th, at 2pm we were notified by our monitoring software that the server was not responding to SMTP requests. We looged in to the server and loads of services had crashes or stopped and we were greated with thousands of errors such as these:

Event 2019: The server was unable to allocate from the system nonpaged pool because the pool was empty.

Event 5: The shadow copy of volume F: could not be created due to insufficient non-paged memory pool for a bitmap structure.

Event 8026: LDAP Bind was unsuccessful on directory server.Ninex.local for distinguished name ''. Directory returned error:[0x51] Server Down.    

We restarted the server and all was good again, the events stopped and all services came back up fine. I'm not 100% sure but this strikes me as a memory leak somewhere? I always thought the periodic nature of the problem seemed like some form of memory leak or memeory problem of some description but this seems to confirm it in my mind. Just wondered if anyone had any thoughts with this new info?

I am having this issue with a clients server.  The IIs stops repsonding, there is nothing in the event viewer.  I have tried restarting IIs, IIs Admin etc, but to no avail.  The only thing that appears to work is a reboot of the server.

John.

John,

Could I ask what backup system you are using?

Tim

The client is using a Dell LTO 2 tape drive and Veritas BackupExec 10d.
We applied all the updates from Dell (their server is a PowerEdge 2800 with 6Gb RAM) last night, except for the BIOS, this failed with a lack of memory, so something is causing a memory leak.  But is still strange I cannot find anything in the event log.

He tells me that if he reboots within 14 days then is does not get to the stage of services failing.

p.s. the reason for 6Gb is purley due to warrenty replacement - he had 4Gb, they sent 4Gb replacement but only 2 modules were failing so he left the remaining 2 in.

John.

I am having this same issue as well. I have multiple IIS servers, but one in particular seems to experience a problem at random intervals where IIS  just blatantly stops running and the ONLY solution thus far is a complete reboot. I looked in Windows Event Viewer but there is no information. I looked in the IIS logs (system32/LogFiles) but nothing in those files indicate any failures or warnings.

No recent changes have been made to this IIS website recently and no programs have been installed on the server itself either.

I am using Symantec BackupExec 11d and this server is 2003 SP1 Std Edition. I would get SP2 and other related patches/hotfixes, but from what everyone else has said, that has not fixed the problem.

I do not believe this to be a network issue either, because my server has a static IP address, it is not assigned through DHCP. I also checked and my DNS server has the appropriate Host (A) and PTR records. I can ping forward and reverse and receive requests just fine.

Anymore advice or suggestions on this would help a lot of us out.

Thanks!

The reason I asked about backup is because we seem to have found our culpret. On all the problem servers we were using an online backup system that sends backups to an external server. This is a popular system so we didn't think it was worth removing but upon removal last week the problem seems to have stopped. The reason we left it so long before trying to remove this software is because we have obviously left this server with no backup system for the least 11 days and that is not something we wanted to do unless as a last resort.

It does seem that the program must have been causing some kind of memory problem but this was difficult to detect. The errors we got in the event log when the server was not restarted would seem to support this. I would suggest that perhaps IIS is very suceptable to this problem and is consequently the first to fail but when we left it for several more days the entire server ground to a halt.

I would suggest two things. If at all possible, try leaving the servers running once IIS has stopped to see if anything else fails if you don't reboot. I know that this is probably impossible in most scenarios but might be worth a try if possible. I would then remove all third party sofware (including the backup system) to see if that makes any difference. Obviously ensure you have another way to make sure data is protected during that duration.

Apart from that I'm all out of answers. I will certainly keep an eye on this post and will respond if anything else comes to light.

Thanks for the quick reply GGUser.

It's interesting you mention leaving the server alone after it fails. To be honest, we actually have no choice but to leave it alone for most of the time. Our server with the issue also contains some important ERP system files so we cannot rbeoot this server unless it's after hours or on select weekends. Thus, having an IIS problem with a few websites being inaccessible is quite a pain.

With that said, the server has not been working since Monday of this week at least (the problem probably occurred over the weekend) and since then over the past 3 days there hasn't been much of a change at all. All other systems on the server seem to be functioning just fine, and there are no other strange errors in the Event Viewer.

You said your server came to a grinding halt after you left it alone for a while? You think the problem is BackupExec? Hmm...interesting.

The eventual result is the TCP/IP failing and no one can connect to the server, however we were able to RDP in on the WAN side and shutdown the server gracefully.  However, it did take some considerable time to shutdown.

John.

John - That sounds very similar to our problem. With ours we were able to RDP but the store, file server and a few other services had already failed. I would suspect if left the entire stack would have failed also but we got to it in time.

I'm now pretty sure the problem must be caused by a memory leak but usually these are easy to detect by looking in task manager. The only thing that seemed to be taking up more memory than would be expected was the store service.

Again, I am guessing, but perhaps our backup tool was doing something strange when it backed up the information store and causing a memory leak? The next thing for me to try is just using NT backup on its own. The online tool we use uses NT backup as the underlying basis for its exchange backup. It creates a backup file using NT backup and then sends that to the online server. There is a chance that NT backup is causing the problem so that is what I will check next.

I would still suggest the best course of action would be to remove all third party applications and see if that makes any difference.

GGUser,
Thanks.

I will check what else is in installed, but I am not very keen on removing the Veritas BackupExec 10d.  The SBS box uses SQL2000 as the core of their business (the backbone to the software we write).

Could you tell me the best way to find the application that is causing the leak - you seem to have an idea about how to go about it.

Some time ago I did setup some performance monitoring, but I was not sure how to interpret it.

John.

I fully understand why you wouldn't want to remove the backup!! You could however try uninstalling the backup software completely and then reinstall it. BackupExec is a very popular package so I have no doubt about it as a backup tool but your specific installation could have become corrupt I guess?

Here are some tips for finding memory leaks:

1 - Run perfmon and monitor all the memory counters etc, this can help identify abnormal memory usage.

2 - Use task manager - You can add new counters by clicking view->select column. Page Faults, Page Faults Delta, and Virtual Memory Size may be of help.

It is definately possible for the page file to bloat due to memory leaks. This could potentially cause all manner of problems including those we've identified. It may be very hard to track down though!

Well, not sure if this is related, but with my server still having IIS in a non-functional state, I got to work this morning at about 7:30am, and shortly after that, finally something out of the ordinary popped up in my event viewer:

Source: Srv
Event ID: 2022
Type: Warning

The server was unable to find a free connection 63 times in the last 60 seconds.  This indicates a spike in network traffic.  If this is happening frequently, you should consider increasing the minimum number of free connections to add headroom.  To do that, modify the MinFreeConnections and MaxFreeConnections for the LanmanServer in the registry.

We have a folder called "Applications" that is shared as the G: drive which is mapped automatically via a login.bat script for all users, and just after we got this error, the mapped network driver stopped working and the connection to the server was no longer available.

I took the recommendation from Microsoft and created the Min and Max connections values in the registry, then I rebooted.

We'll see if the problem with IIS comes back or not.

Thanks for the help guys.

We have just had the problem re-occur on our main server. The problem definately appears to be related to resources and the exchange backup. When we turn of exchange backups and let the log files collect the problem does not occur. When we backup the exchange database (Using any program) the error occurs at the same interval. We have now seen this occur with the online backup utility, NT backup and Backup Exec so it is not related to the backup program itself.

When the problem occurs we get other strange issues related to resources but we can't seem to locate what specific resource is causing the problem. Page file usage looks fine, as does memory usage. Increasing physcial memory makes no difference as we have tried that.

I'm having this problem on my Windows SBS 2003 server and I'm going through the advice given above to make sure I've not overlooked something.

But I believe my problem may be related to the SBS 2003 server version requiring that it be a domain controller - which is not necessary in my environment but I let it entertain itself and be one anyway.

The server has been in use for about two-years and shortly after it went online (not sure but eight-days sounds about right but in the middle of the night) it promoted itself to a domain controller and caused IIS 6.0 to stop.

I left the server alone regarding being a domain controller and got IIS to work but and have been rebooting the server about every eight-days for the last two-months as a work-around to this problem.

Now rebooting does not seem to bring IIS back online and I'm stumpted.

For the record we have got to the bottom of our problem within the last 2 weeks.  Spoolsv.exe was eating up all the non-paged pool free memory which in turn started a cascade failure of the services starting with printing, iis, and evenually tcp/ip.

If we restarted the spooler service (net stop spooler/ net start spooler) then the resources were freed and all the other services came back online.

Of course spoolsv.exe was not cause but the mediary, the cause was a HP Printer Driver for LJ9050n.  We have tried installing the current version, but it still causes spoolersv.exe to eat up the non-paged pool.

As a workaround we have put in a scheduled cmd file that restarts the spooler server every sunday when no one is in. Not perfect but at least their server is no longer ceasing to dunction.

I will be also reporting it to HP - don't live in much hope I will get anywhere.

Regards
John.

P.S.

for those that don't know, you can view the non=page pool through task manger.

In task manager, select Tools/Select Columns.  Put a tick in non-page pool in the right column.

Non-Paged pool shows as np-pool in the main screen.

Hope this may help someone.