Server reboots every hour or few minutes

I have Kaspersky on it and it found nothing. Results from the symantec scan; "W32.blaster.worm has not been found on your computer"

What are you using IIS for, WSUS services??

IIS is for OWA
Here is the error that precedes the system reboot:
8/28/2009      8:43:44 AM      Application Error      Error      -100      1000      N/A      MH-MSX      Faulting application lsass.exe, version 5.2.3790.1830, faulting module msv1_0.dll, version 5.2.3790.4530, fault address 0x0000000000016df1.

Googled it and it was largely useless

I removed Kaspersky and stopped RDP-ing to the server and it didn't errantly reboot until this morning. so I can rule out Kaspersky and RDP.

I'm testing a hypothesis, though it's a long shot. I have APC software installed on the server and it would shut it down in case of little battery remaining on the UPS. It runs on Java. I've uninstalled the APC software and Java. It hasn't rebooted yet, but it's only been about an hour.

I hate computers. I'm gonna go be a hobo.

Well, let's break down the error into something more tangible:

LSASS stands for Local Security Authority Subsystem Service> We also know this pertains to IIS  (Internet Information Service).

So, you are having problems with authentication on IIS. Is it the subsystem's routine?? HMM

http://support.microsoft.com/kb/893712

http://support.microsoft.com/kb/838656

I checked kb-893712and neither switch is enabled. kb-838656 may be accurate.
I must say however that my system has not rebooted since I removed java. I've been advised to spend money only as a last resort.

I'll get back in a few days with an update.

Well, the java uninstall didn't fix it. I'm going to follow the advice of kb 838656, and hope it's the cure.

Having the same problem. Windows Server 2003 Enterprise with SP2, Exchange 2003 Enterprise SP2. Installed KB968389 on 8/25/09 (along with several others), which updated msv1_0.dll (indicated as the faulting module in the event log errors). This fault started occuring last night, 9/4/09.

My question is: can patch KB968389 be removed safely?

@Rohk - Post in a new question. I've actually paid for my account. You cannot hijack this thread.

OK, so I tried to install that patch, but it won't install. I think it is the x86 version and my OS is x64. I'm going to try to re-install SP2. Any other suggestions?

SFC /scannow should varify versions of OS files. I think I would go to the command prompt and type:

SFC /scannow

Have your install disc handy. It may ask for it.

Okaaaay. Let me put my comment another way: have you installed patch KB968389? If yes, when did you install it? Did your rebooting problem start before, or after (and how long after) you installed that patch? Have you tried uninstalling that patch? What were the results?

The error message in your event log (as posted by you, above) indicates msv1_0.dll is the "faulting module." This DLL is replaced by the recently released patch KB968389, and may be related to the problem you are having.

Good luck.

@Rohk - ahh, ok. Since you put it that way, I have not installed that patch. I will look into it.

I'm still trying to find the OS disk to do a SFC.

Mondays are always so freaking busy and I am the everyman IT here at this company.

I found and uninstalled kb968389 and rebooted. Original error still occurring.

I ran the SFC scan, but it didn't find anything. The dialog box disappeared without any queries or errors which leads me to believe that everything was nominal.

What service pack are you on??

If on SP1, then install SP2.

The version of LSSAS you are using is SP1. SP1 has many problems that were mostly corrected with SP2.

I'm on SP2

Not trying to hijack, but my Exchange server is doing the same thing - Exchange 2007, x64, SP2, started rebooting on 9/4 just like yours.  We have to find an answer to this.

"The version of LSSAS you are using is SP1"

Should I re-run SP2 install?

I think I would.

I've re-installed SP2. Let's see if it reboots autonomously today.

I wanted to provide an update of some things I've been investigating with this issue.  Again, not trying to hijack but provide further info for others to troubleshoot.

My server rebooted last night at 2:33am and again this morning at 8:33am.  Both times in the security logs is a failed logon/logoff attempt by a user that was recently terminated from our company.  Her account still exsists in AD but has been disabled now for a couple of weeks.  These security log events happen exactly 1 second before the NTLM exception happens.  The NTLM exception causes lsass.exe to crash which cause the reboot.  The failed logon attempt is coming from 208.54.83.83, which is a Tmobile IP.  She had a Tmobile Android phone that was hooked up to Exchange, that I'm guessing she hasn't removed the Exchange profile from her phone yet, so its still trying to check email.

Im wondering if with the latest Windows update changes to the msv1_0.dll file, various phones trying to use OWA's address is somehow exploiting a bug that causes the NTLM package to fail while trying to authenticate?

That may be a long shot, but could others having this issue check the security logs at the same time at the first error message is generated and see if you have failed logon attempts by a user's phone, a disabled user, etc....

SP2 fix failed. Still rebooting.

I'm going to examine my logs for your findings above. Have you tried removing the user?

Not yet.  That would be my next step.

Well, let's prevent the server from rebooting so we can get a blue screen of death>

Right click "my computer" icon>>select "properties">>go to the "advanced" tab>>Under the startup and recovery section, choose the "settings" button. Then, deselect auto sartup upon error.

Once you reboot. It should provide you with a blue screen. Please writed down the 0x... code and application that is problematic.

As far as I can tell, the server reboots cleanly.  That is, it does not 'blue screen' and just reboot itself.  The winlogon.exe process actually shuts it down cleanly.

I agree. Every time it reboots, there is a countdown and no BSOD.

The reboot, will make it appear to be a clean reboot if the computer is selected to auto restart upon error.

This is why we might unselect that option to see the BSOD.

I've disabled reboot upon BSOD

Let us know what happens upon the next reboot.

Write down the application and 0x... error code.

Second auto-reboot following the "restart after error". No BSOD. Rebooted just fine tho.

OK, had to revert to research:

EventID.NET:
http://www.eventid.net/display.asp?eventid=5000&eventno=1313&source=LsaSrv&phase=1

"Ray Fernandez (Last update 7/27/2005):
I just installed Windows 2003 Enterprise Server running Exchange 2003 and IIS 6 for OWA, and it was giving me this error. After calling Microsoft, they said the reason for that was the frequent Bot attacks to IIS 6, and pointed me to install MS04-011 and MS04-007. They also suggested MS05-019 if it applied to my system. They said these patches are not been pushed as Windows updates because of some issues and they must be downloaded and installed manually. After installing those patches and rebooting the system, the event disappeared."

This posting was submitted in 2005. So, we should make sure the patches apply to your system.  Links to the patches are at the bottom of the EventID.Net web page.

I have seen several articles like this too, dated from 2004-2006.  It seems like these issues should be patched by now though?  Also it doesn't seem like a coincidence that there are three of us who just started having this issue around the same couple of days.  Not discounting the research, just trying to apply normal troubleshooting logic/variables into the mix.

Just auto-rebooted again.  Events are time stampted at the same time.  Disabled user tries to logon, NTLM throws an exception:

From the security log:
-----------------------------------

Event Type:      Failure Audit
Event Source:      Security
Event Category:      Logon/Logoff
Event ID:      531
Date:            9/16/2009
Time:            3:29:19 PM
User:            NT AUTHORITY\SYSTEM
Computer:      DAXBHMEX01
Description:
Logon Failure:
      Reason:            Account currently disabled
      User Name:      lwilliams
      Domain:            DCC001
      Logon Type:      8
      Logon Process:      Advapi  
      Authentication Package:      Negotiate
      Workstation Name:      DAXBHMEX01
      Caller User Name:      DAXBHMEX01$
      Caller Domain:      DCC001
      Caller Logon ID:      (0x0,0x3E7)
      Caller Process ID:      4384
      Transited Services:      -
      Source Network Address:      208.54.83.63
      Source Port:      57618

From the System log:
-----------------------

Event Type:      Error
Event Source:      LsaSrv
Event Category:      Security Package Manager
Event ID:      5000
Date:            9/16/2009
Time:            3:29:19 PM
User:            N/A
Computer:      DAXBHMEX01
Description:
The security package NTLM generated an exception.  The exception information is the data.

I'm on the phone with MS right now. Unlike others, I promise to post the resolution so future users find a corrective action.

A while back, I use to see clients that were trying to authenticate with a Kerberos Authentication server, using LMHash authentication. From that, we saw clients with the inability to logon.

So, we prevented the clients from authenticating using LMhash. Here is the posting:
http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/Windows_2003_Active_Directory/Q_23132123.html

Now, from what I am seeing, you have clients that are trying to authenticate using NTLMhash. I wonder if new updates are not permitting 2003 server to be compatible with NTLMhash authentication through IIS6??

This thread explains how to authenticate using NTLM and kerberos at the same time using IIS6.
http://support.microsoft.com/kb/215383

DANGER: NTLM hash is a very easy hack and clients should not be authenticating in NTLM if at all possible, (especially using a web based api).

I am thinking the same group policy for preventing LMhash would not only secure your network, but might prevent the server from trying to cough up a NTLM authentication. To find out how to prevent LMhash authentication see the Experts Exchange link. If nothing else, this helps secure your network a bit better.

Also perform this on one test machine:

Backup prior to testing.

Playing with LSASS registry keys can cause BSOD's. So be very precise.

The test machine I would try out would be.
DAXBHMEX01

Furthermore, LM and NTLM viruses may try to infultrate your server. I would check that machine out for Malware.

@sullimd - we had four reboots yesterday, and the first seemed to follow directly after a failed Android phone logon (not a disabled account, however, just a bad password). However, the subsequent reboots did not show any Android logon just prior to the reboot, so unfortunately that doesn't appear to be the cause (at least for us). Also, the subsequent reboots did not appear to directly follow any sort of failed logon. I'm curious to know if this pattern (failed Android logon followed by lsass failure and reboot) is still holding true for you?

@ChiefIT - the auto-reboot after failure of lsass is by design, and is not typically accompanied by a BSOD (replying to one of your earlier messages).

@MH-Administrator - was Microsoft able to help you?

My pattern was holding true every time.  Failed login from this one users phone - then the next log at the same timestamp to the second (as seen above) was the NTLM package throwing the exception.  

I ended up blocking 208.54.0.0/16 yesterday at my firewall to prevent any Tmobile phone from accessing OWA.  I also emailed the user at a personal email address HR had on file to tell her to delete her Exchange account.  Haven't heard back from her, but all day yesterday while her phone was blocked we did NOT have a reboot.  No reboots today either.  

The fact that both of us had failed Android logins is too much of a coincidence to be ignored.  Not that the Android is the problem, but in some way it seems like its exploiting some type of Windows bug that may be undiscovered at this point.

I'm interested to see what MS said too.

@sullimd - we just had another reboot, directly preceded by a failed Android phone logon (same user as yesterday). I'm going to go back for a closer look at the logs from the other three reboots yesterday...

You can try to block the IP range on the failed security log attempt, but just know that it will block all phones on Tmobile from accessing Exchange.  It is somewhat of a drastic move depending on how many phones rely on email, but it can help in troubleshooting to see if thats the issue.

I've examined my logs and there is no correlation between user logins and server reboots. Of four reboots, various users successfully authenticated seconds before lsass crashed. Some of these users don't have smartphones. I have very few failed authentications.

I'm on my third escalation with MS support. So far, they've had me try to install KB868356 (http://support.microsoft.com/kb/838656). It wouldn't install on my machine, but I thought I'd add it JIC it resolves your issue.

I transposed the KB. It's KB 838656; http://support.microsoft.com/kb/838656

I saw that fix several days ago, but the fix is for x64 SP1 and I'm running SP2.  It would not install for me.

This phone and these remote stations that are authenticating through NTLMhash might be upgraded to a kerberos authentication scheme.

Phones, these days, are mini handheld computers with their own OS. So, I am thinking the OS version of the phone can be upgraded to support Kerberos and get off the NTLM authentication you are seeing.

Microsoft updates to service pack 2, disables the server's ability to be backwards compatible with LMhash Authentication as a security update. This may render clients unable to authenticate with the server because the client is on the WRONG authentication protocol. This is why things might work for a while, then a few computers fail to logon after a windows update. By default, MS computers, including the server are suppose to use kerberos. But NTLM was used to be backwards compatible with Legacy machines. This is why most of your articles pertaining to NTLM authentication are dated way back.

I did find an article that effects the servers running EXCHANGE and SHAREPOINT. It disables kerberose on the Exhcange server. This effects the front and backend servers. The inability to negotiate Kerberos from the clients, I COULD see causing the server to reboot repetetively.

Guys:

Tell me what you are looking at:

Are these a couple clients that are trying to authenticate via NTLM?

Or is the Server stuck trying to provide authentication via NTLM over IIS6?

If it is the clients, we may need to get in touch with these clients phones and machines to make sure they authenticate via Kerberos. For a phone, that may require a OS update.


If this is the server, It looks like there are things to check that pertain to the way OWA clients interact with your authentication over IIS.

http://www.microsoft.com/exchange/2007/support/e2k3owa.mspx

Here are the two logs that happens before/at the time of the crash.

DAXBHMEX01 is my Exchange server - I do not have a front/back.  All services are on the same box.  WINDROID is the Google Android phone.  208.54.x.x is Tmobile's IP range.  Looks like the phone is trying NTLM.

---------

Event Type:      Failure Audit
Event Source:      Security
Event Category:      Logon/Logoff
Event ID:      537
Date:            9/16/2009
Time:            7:29:16 AM
User:            NT AUTHORITY\SYSTEM
Computer:      DAXBHMEX01
Description:
Logon Failure:
       Reason:            An error occurred during logon
       User Name:      LWILLIAMS
       Domain:            DCC001
       Logon Type:      3
       Logon Process:      NtLmSsp
       Authentication Package:      NTLM
       Workstation Name:      WINDROID
       Status code:      0xC0000225
       Substatus code:      0x0
       Caller User Name:      -
       Caller Domain:      -
       Caller Logon ID:      -
       Caller Process ID:      -
       Transited Services:      -
       Source Network Address:      208.54.83.79
       Source Port:      6908
---------

Event Type:      Failure Audit
Event Source:      Security
Event Category:      Logon/Logoff
Event ID:      531
Date:            9/16/2009
Time:            7:29:15 AM
User:            NT AUTHORITY\SYSTEM
Computer:      DAXBHMEX01
Description:
Logon Failure:
       Reason:            Account currently disabled
       User Name:      lwilliams
       Domain:            DCC001
       Logon Type:      8
       Logon Process:      Advapi  
       Authentication Package:      Negotiate
       Workstation Name:      DAXBHMEX01
       Caller User Name:      DAXBHMEX01$
       Caller Domain:      DCC001
       Caller Logon ID:      (0x0,0x3E7)
       Caller Process ID:      4056
       Transited Services:      -
       Source Network Address:      208.54.83.63
       Source Port:      33957

@MH-Administrator: please report the version of msv1_0.dll in your \Windows\System32 directory (right-click, properties, version tab). I'd like to confirm that the uninstall of KB968389 successfully backed out that DLL. Thanks.

I found 968389 installed again. I thought I removed it but I guess I was wrong. It's gone now and the ver is 5.2.3790.3959.

No errors yet. We'll see whats what.

I went into WSUS and declined the update. Would that be enough to keep it from installing?

Should be.  If you have any GPO's controlling it make sure those apply to the server in question.

Ah, okay. So I think my original suspicion may be correct then. I have uninstalled the bundle of patches that I installed on 8/25, one at a time. I built a dependency matrix and uninstalled them in the following order:

KB956744
KB973540
KB971032 (reboot)
KB960859
KB971557
KB973869
KB973507 (reboot)
KB973354
KB968389 (reboot)

You should build your own matrix if there is any question of what order your own installed patches should be uninstalled in.

I'll reinstall all of these patches except KB968389 and we'll see what happens. My reboots have not been daily, so I'll report back in a week if there are no reboots (or sooner if there are).

Update: I asked my Android user to try connecting again with a bad password. Logon errors logged, but no reboot. So far, so good...

Still no reboot yet. Mine were at eight a day or none for several days. I'll check back in a few days to see if everything remains stable.

Howdy All::

Looks like we have a fourth person with this LSASS problem. Anyone willing to make sure this person is having the same problem and work with him on fixing it????

John

http://www.experts-exchange.com/OS/Microsoft_Operating_Systems/Server/2003_Server/Q__24748877.html

I've gone a week without a reboot, so I think we solved this.

Lets stop here please. Whilst sharing information, symptoms, resolution attempts across a number of systems/outfits is great, it also makes troubleshooting and diagnosis a nightmare by changing too many parameters within the question set.

MH - You stated you had been on the phone to Microsoft - are you in a position to advise on Microsoft's advice regarding the issue you have given to them?

In your initial post you stated 'my server' - is this the only server you have in your workgroup or domain? Do you have any others? Are they also experiencing the same issue?
What services is this server running? Is it just file and print? Other? DC?
What role is the server playing within your company/organisation?
What applications are installed? SQL? MSDE? Anything?

Thanks
Keith

Lets stop here please. Whilst sharing information, symptoms, resolution attempts across a number of systems/outfits is great, it also makes troubleshooting and diagnosis a nightmare by changing too many parameters within the question set.

MH - You stated you had been on the phone to Microsoft - are you in a position to advise on Microsoft's advice regarding the issue you have given to them?

In your initial post you stated 'my server' - is this the only server you have in your workgroup or domain? Do you have any others? Are they also experiencing the same issue?
What services is this server running? Is it just file and print? Other? DC?
What role is the server playing within your company/organisation?
What applications are installed? SQL? MSDE? Anything?

Thanks
Keith

@keith: If you're a Microsoft engineer, you have the problem you describe: what exactly is the trigger, what exact configurations are affected, etc. For our purposes, however, we've found the culprit: Microsoft released a patch which is causing our various systems to "spontaneously" (from our point of view) reboot. There have been at least three other threads on this issue started in the past couple of weeks just on Experts Exchange alone... and who knows how many on other message boards.

Let's look at the commonalities (all of which have already been spelled out upthread): Microsoft Windows Server 2003, running various versions of Exchange (2003 and 2007) with OWA enabled. We all installed patch KB968389 shortly before the reboots started. The error messages in the logs point to a DLL which was updated by that patch (msv1_0.dll). When we uninstalled that patch, the reboots went away. As far as I'm concerned, we're done.

Now, if you're Microsoft, you've got a headache to deal with, but that's not our responsibility - it's theirs. I'm looking forward to an updated patch that doesn't cause my Exchange server to reboot, and I won't reinstall KB968389 on that server until they've proven it will be stable.

Rohk - I am no longer Microsoft.

MH-Administrator, in addition to my questions above, can you confirm that since the removal of the patch on the 18th, you have still not had any issues?

I am not aware of any 'replacement' patch being issued currently but the issue is documented on a number of forums with PSS recommending the removal if this form of issue is seen, as mentioned by previous commentors. Here is just one of them.

http://messagexchange.blogspot.com/2009/09/kb968389-lsass-reboots.html

@ keith: I wasn't trying to say that you were with Microsoft. I was simply suggesting that Microsoft has more work to do on this issue. Also, thanks for confirming my solution.