Solved

IIS7.5 Shutting Down Randomly

Posted on 2011-09-30
19
1,184 Views
Last Modified: 2012-05-12
I'm in desperate need of help. We recently got a new server (2008) running IIS 7.5. Before of all our websites ran on a 2003 server using IIS 6. The sites are all written in .NET 2.

When I moved everything over to the new server, I had to set up the App Pools using Classic Pipline mode for them to run properly. Our hosting company ultimately switched the IP address from the old 2003 server to the new 2008 server. And, after they did, IIS7.5 would periodically shut down. The first day I had them make the switch, I had to restart IIS probably a dozen times because IIS kept shutting down. NOTE that the app pools weren't shutting down. ALSO NOTE that ALL sites would shut down.

I had the hosting company switch the IP address back to the old server and I went and converted all of our Websites to .NET 4 and modified the code that .NET 4 didn't like so that I could run all the sites in Integrated Pipeline Mode.

I got all the updated code on the server, updated all the App Pools to 4.0 and Integrated mode and had the hosting company switch the IP back this morning. For 2.5 hours, everything ran great...

But then all the sites shut down again.

This is BEYOND frustrating.

I'll post some warning messages coming up in the application log, but note that NONE of these warnings are new. All of these warnings showed up in IIS6 logs.

I have no idea why this keeps happening, but I've wasted an inordinate amount of time trying to figure this out and would appreciate any help anyone can offer.

Again, these warnings NEVER shut down IIS6.

Also, I have not been able to repoduce these warnings on the Websites.

Event code: 3003 
Event message: A validation error has occurred. 
Event time: 9/30/2011 9:59:43 AM 
Event time (UTC): 9/30/2011 4:59:43 PM 
Event ID: e79470f2fcc44c21a6f3e98b70c0d61b 
Event sequence: 1208 
Event occurrence: 1 
Event detail code: 0 
Exception information: 
    Exception type: HttpRequestValidationException 
    Exception message: A potentially dangerous Request.Form value was detected from the client (hiresTxt="<a href='http://imag...").
   at System.Web.HttpRequest.ValidateString(String value, String collectionKey, RequestValidationSource requestCollection)
   at System.Web.HttpRequest.ValidateNameValueCollection(NameValueCollection nvc, RequestValidationSource requestCollection)
   at System.Web.HttpRequest.get_Form()
   at System.Web.HttpRequest.get_HasForm()
   at System.Web.UI.Page.GetCollectionBasedOnMethod(Boolean dontReturnNull)
   at System.Web.UI.Page.DeterminePostBackMode()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)
   at System.Web.UI.Page.ProcessRequest()
   at System.Web.UI.Page.ProcessRequestWithNoAssert(HttpContext context)
   at System.Web.UI.Page.ProcessRequest(HttpContext context)
   at ASP.gallery_aspx.ProcessRequest(HttpContext context)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Open in new window


Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 9/30/2011 9:49:43 AM 
Event time (UTC): 9/30/2011 4:49:43 PM 
Event ID: 2d6f793e6a4a4dc7a73dd67d5df3ee63 
Event sequence: 339 
Event occurrence: 3 
Event detail code: 0 
Exception information: 
    Exception type: FormatException 
    Exception message: Input string was not in a correct format.
   at System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   at System.Number.ParseInt32(String s, NumberStyles style, NumberFormatInfo info)
   at System.Convert.ToInt32(String value)
   at HowToBuy.Page_Load(Object sender, EventArgs e)
   at System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e)
   at System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e)
   at System.Web.UI.Control.OnLoad(EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Open in new window


Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 9/30/2011 9:47:31 AM 
Event time (UTC): 9/30/2011 4:47:31 PM 
Event ID: 07a1f5019c014b1f9f3ea64b57e5390b 
Event sequence: 20 
Event occurrence: 1 
Event detail code: 0 
Exception information: 
    Exception type: ArgumentOutOfRangeException 
    Exception message: Index and length must refer to a location within the string.
Parameter name: length
   at System.String.InternalSubStringWithChecks(Int32 startIndex, Int32 length, Boolean fAlwaysCopy)
   at Search.Page_Load(Object sender, EventArgs e)
   at System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e)
   at System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e)
   at System.Web.UI.Control.OnLoad(EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Open in new window


I honestly don't think these warnings have anything to do with IIS7.5 shutting down, but who knows.

Oh, and while I was writing this, it HAPPENED AGAIN. Here's another warning that showed up:

Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 9/30/2011 10:29:48 AM 
Event time (UTC): 9/30/2011 5:29:48 PM 
Event ID: 92346f20dfe144c0aca0f713b1fe3e6b 
Event sequence: 235 
Event occurrence: 1 
Event detail code: 0 
Exception information: 
    Exception type: WebException 
    Exception message: The underlying connection was closed: An unexpected error occurred on a receive.
   at System.Net.HttpWebRequest.GetResponse()
   at GenerateSpecSheet.Page_Load(Object sender, EventArgs e)
   at System.Web.Util.CalliHelper.EventArgFunctionCaller(IntPtr fp, Object o, Object t, EventArgs e)
   at System.Web.Util.CalliEventHandlerDelegateProxy.Callback(Object sender, EventArgs e)
   at System.Web.UI.Control.OnLoad(EventArgs e)
   at System.Web.UI.Control.LoadRecursive()
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)

An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

Open in new window


Please help.

Thanks!
0
Comment
Question by:infotechelg
  • 14
  • 4
19 Comments
 
LVL 14

Expert Comment

by:binaryevo
ID: 36893665
is the 2003 server still up and running?  if so i would say use that as production for now and run some tests on the 2008 ( IIS 7.5 ) box.  I would first start by placing a simple hello world application ( written in same environment as the other applications - .net 2.0 i think ).  Let that hello world site run some and have users bang on it a little bit to see if you can reproduce the issue.  If so then chances are you will  upgrade all your apps to .net 3.5+.  If that doesnt work then an IIS reinstall may need to be done.
0
 

Author Comment

by:infotechelg
ID: 36893763
Hi binaryevo. Yes, the 2003 server is still up and running, but I have already converted all my apps to 4.0 on the new Server.
0
 

Author Comment

by:infotechelg
ID: 36899617
Anyone else?
0
 
LVL 14

Expert Comment

by:binaryevo
ID: 36907228
And your sure there are no memory leaks between the IIS app pools?  Ive never seen this before either... Very bizarre behavior.  My recommendation would be to reinstall IIS.
0
 

Author Comment

by:infotechelg
ID: 36910659
Yeah, we just reinstalled IIS. We'll see how that goes.

How do you check for memory leaks in the app pools?
0
 

Author Comment

by:infotechelg
ID: 36912214
Restarting IIS did not help. This is still happening, and it seems to be happening more frequently than it was before.
0
 

Author Comment

by:infotechelg
ID: 36912304
sorry, i meant reinstalling IIS, not restarting IIS.
0
 
LVL 14

Expert Comment

by:binaryevo
ID: 36917936
Wow... Then i would think this is more than likely not an IIS issue but a code issue and possibly how IIS ( new version ) is handling it.  Take a look at this article, as it outlines pretty well how to check for memory leaks in .net  http://stackoverflow.com/questions/5274566/detecting-memory-leaks-in-asp-net.

0
 

Author Comment

by:infotechelg
ID: 36921207
would this still apply even if I'm not getting "Out of Memory" exceptions in my .NET Websites?
0
How your wiki can always stay up-to-date

Quip doubles as a “living” wiki and a project management tool that evolves with your organization. As you finish projects in Quip, the work remains, easily accessible to all team members, new and old.
- Increase transparency
- Onboard new hires faster
- Access from mobile/offline

 

Author Comment

by:infotechelg
ID: 36933476
I think I narrowed down the Website that is causing IIS to shut down. I saw this warning in the application log. Do you have any thoughts? Do you think this could be causing IIS to shut down?

Event code: 3005 
Event message: An unhandled exception has occurred. 
Event time: 10/7/2011 12:49:36 PM 
Event time (UTC): 10/7/2011 7:49:36 PM 
Event ID: 010a9366cbc94001b3acff15007ac44d 
Event sequence: 3460 
Event occurrence: 14 
Event detail code: 0 
 
Application information: 
    Application domain: /LM/W3SVC/5/ROOT-1-129624766261336158 
    Trust level: Full 
    Application Virtual Path: / 
    Application Path: D:\webpub\LBL_2009\ 
    Machine name: CFD183 
 
Process information: 
    Process ID: 7624 
    Process name: w3wp.exe 
    Account name: CFD183\ForASPNET 
 
Exception information: 
    Exception type: FormatException 
    Exception message: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or a non-white space character among the padding characters. 
   at System.Convert.FromBase64String(String s)
   at Encrypt.Decrypt(String cipherText, String Password)
   at CommonCode.GetQueryString()
   at Search.GetResults()
   at ASP.search_aspx.__RenderContent1(HtmlTextWriter __w, Control parameterContainer)
   at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
   at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
   at System.Web.UI.Control.Render(HtmlTextWriter writer)
   at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Control.RenderControl(HtmlTextWriter writer)
   at ASP.headerfooter_master.__Renderform1(HtmlTextWriter __w, Control parameterContainer)
   at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
   at System.Web.UI.HtmlControls.HtmlForm.RenderChildren(HtmlTextWriter writer)
   at System.Web.UI.HtmlControls.HtmlContainerControl.Render(HtmlTextWriter writer)
   at System.Web.UI.HtmlControls.HtmlForm.Render(HtmlTextWriter output)
   at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.HtmlControls.HtmlForm.RenderControl(HtmlTextWriter writer)
   at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
   at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
   at System.Web.UI.Control.Render(HtmlTextWriter writer)
   at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Control.RenderControl(HtmlTextWriter writer)
   at System.Web.UI.Control.RenderChildrenInternal(HtmlTextWriter writer, ICollection children)
   at System.Web.UI.Control.RenderChildren(HtmlTextWriter writer)
   at System.Web.UI.Page.Render(HtmlTextWriter writer)
   at System.Web.UI.Control.RenderControlInternal(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Control.RenderControl(HtmlTextWriter writer, ControlAdapter adapter)
   at System.Web.UI.Control.RenderControl(HtmlTextWriter writer)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint)

Open in new window

0
 

Author Comment

by:infotechelg
ID: 36945706
I honestly don't think anyone can figure this out. I paid $259 for microsoft support and they're clueless (so far).

This is what appears to be happening:

Of the 10 sites I have on the new IIS 7.5 Webserver, one of them appears to be causing the issue. Whatever is happening, it seems that it's causing IIS to prevent broadcasting of the host name to IP bindings to the world because it appears that DNS no longer exists for any of the Websites on the server. But, if I restart IIS, everything comes back up (that is, until it shuts down again).

The Microsoft guy said this is a DNS issue, not an IIS issue. I disagree. If it were a DNS issue, restarting IIS wouldn't make a difference.

Hope this information helps. I'm a little frustrated by the lack of response I've received on Experts Exchange. Help would be greatly appreciated.
0
 
LVL 14

Expert Comment

by:binaryevo
ID: 36949059
I agree with you.  If this exception causes a leak, it could cause IIS issues but, it doesnt seem like this would at first glance.  Try just shutting down that site and running the rest...  I know you said you isolated it but, im curious to see how the other sites work as a result of this one being shut off.  Sorry for your frustration, im trying to help but, this is a bizarre issue ive never personally run into...  
0
 

Author Comment

by:infotechelg
ID: 36950386
I appreciate the help. I had actually moved the DNS of the offending site to a different server (same as shutting it off, I suppose) for about a week and the issue never happened. I moved the DNS of the offending site back to the new server and in about 20 minutes, everything shut down.
0
 

Author Comment

by:infotechelg
ID: 36968480
Microsoft seems to think this is a 3rd party software issue, they think it may specifically be related to antivirus software on the server. When we figure it out, i'll post the solution here in case this happens to someone else.
0
 
LVL 10

Expert Comment

by:ddiazp
ID: 36968517
Have you checked the IIS logs for that site? Maybe this happens whenever a specific object/page/script runs on that specific site you mentioned.

Is it feasible to turn your firewall ON (to block all incoming connections), restart IIS and see if the problem is still there?
0
 

Author Comment

by:infotechelg
ID: 36981485
Yeah, we've extensively looked through almost every log file imaginable.

Right now, the hosting company has turned off the software that Microsoft believes is the offending software, Trend Micro OfficeScan. We're going to leave it off for a few weeks and see if the sites stay up.
0
 

Author Comment

by:infotechelg
ID: 37031405
So, it turns out the Trend Micro OfficeScan wasn't the issue. Back to square one.
0
 

Accepted Solution

by:
infotechelg earned 0 total points
ID: 37156298
The hosting company completely wiped and rebuilt the server with no 3rd party software on it and so far it seems to be working. I guess we just had a bad install?
0
 

Author Closing Comment

by:infotechelg
ID: 37175775
done
0

Featured Post

Highfive + Dolby Voice = No More Audio Complaints!

Poor audio quality is one of the top reasons people don’t use video conferencing. Get the crispest, clearest audio powered by Dolby Voice in every meeting. Highfive and Dolby Voice deliver the best video conferencing and audio experience for every meeting and every room.

Join & Write a Comment

A quick way to get a menu to work on our website, is using the Menu control and assign it to a web.sitemap using SiteMapDataSource. Example of web.sitemap file: (CODE) Sample code to add to the page menu: (CODE) Running the application, we wi…
Running classic asp applications under Windows Server 2008 R2 (x64) and IIS 7 is not as easy as one may think. It took me a while to figure it out while getting error 8002801d a few times. After you install the OS you will need to install the fol…
Sending a Secure fax is easy with eFax Corporate (http://www.enterprise.efax.com). First, Just open a new email message.  In the To field, type your recipient's fax number @efaxsend.com. You can even send a secure international fax — just include t…
This video demonstrates how to create an example email signature rule for a department in a company using CodeTwo Exchange Rules. The signature will be inserted beneath users' latest emails in conversations and will be displayed in users' Sent Items…

705 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now