Services fail to start with Error 5, Access Denied

Hi,

Probably a Microsoft Support ticket, but thought I'd try EE first.

On a Windows Server 2008 with SQL 2005 which is a member of a SBS2008 domain, within the past days, several services fail to start with Access Denied.  Webroot is the security and shows no compromise.

The services that fail are ADFS 2.0 and all Microsoft Dynamics CRM services.  SQL services start and are running with the exception of SQL Server VSS Writer which fails as a Local System account.  SQL Management Studio is working.  There is a mix of user accounts for the MSCRM services and ADFS is Network Service.

A couple recent changes might be possible clues, but there is no evidence to support them being the cause.  I changed a domain admin password a couple weeks ago.  I did MSCRM solution updates a couple days ago.  My first attempt to repair this was to restore a SQL database backup of the MSCRM data prior to the solution updates.  That did not solve the problem, the services still fail to start.  I changed one of the CRM services from its user account to Local System but that did not make any difference, still fails with Access Denied.

Application Event viewer shows only IIS-W3SVC-WP errors with several ID's 2214, 2268.  System Event viewer shows ID 7000, Access Denied for the associated services that fail.  ADFS Events show ID 364 from 5-7-2015 8:48am through the present time.  Administrative Events show MSCRMMonitoring errors ID 18690 starting 5-6-2015 6:16 pm.

Reboots do not help, system is fully up to date.  I'll try a DSKCHK as well.

There does not seem to be a problem with domain membership, I can login, even remotely, without trouble.  The problem seems to have happened overnight, and a day after the solution updates were done.  But MSCRM was working after the updates.

Thank you for any help.
Eugene_PalmerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Randy DownsOWNERCommented:
Did you try changing the domain admin password back?
CoralonCommented:
One possibility would be to disable all the services that are throwing the error, and then try bringing them up one by one, after a reboot & waiting for a few minutes for the server to 'settle down'.  It's possible that one of them may be dependent on the other and won't necessarily start correctly unless they are up.  

Try this a few times, varying the order you bring them up in.  

Coralon
Eugene_PalmerAuthor Commented:
I did change the domain admin password back.

None of the services will start, regardless of dependencies.
Active Protection takes the fight to cryptojacking

While there were several headline-grabbing ransomware attacks during in 2017, another big threat started appearing at the same time that didn’t get the same coverage – illicit cryptomining.

Randy DownsOWNERCommented:
(Is it possible to back out the MSCRM solution updates?
EugeneZ ZhitomirskySQL SERVER EXPERTCommented:
please post your sql 2005 edition and sp lvl

<SQL Server VSS Writer which fails as a Local System account.>
if you have rights  on sql server and using this service "SQL Server VSS Writer' : add  Local System to sql logins..

-----------------------------
<several services fail to start with Access Denied.>
what exactly? can you post dependencies?  can you check dependencies of these servers? do they need to access sql server?
check sql server error logs to see what accounts are looking for access?


also is Builtin\administrators account login with sa right still in your sql 2005?

can you try to use "Local System account' instead of AD logins for your unlucky services ( just make sure to add it as sql login with sa rights ) to see if it is fixing?
compdigit44Commented:
Do you have a good full backup of the serve before the change?
What was the version/build of MSCRM before and after the update?
Are any of your service account getting locked out in AD?
Eugene_PalmerAuthor Commented:
I did 'backout' of the MSCRM solution updates by restoring a SQL database backup prior to the solution updates  I always make a backup prior to any solution changes.  This did not help, and MSCRM was working after the updates.

I do have a full server backup and that will likely be the next step, one that I am not looking forward to as it is far older than I realized (1 month, bad person) and it might not go well due to MSCRM database changes that I will have to push forward to the present by restoring the latest backup, etc.  Not confident the computer account will be stable at that age either.

The services that fail are listed in the OP.  There is an additional service I did not mention that seems superfluous, A DoubleTake.  I don't have all the dependencies in front of me, but the fact that ADFS fails indicates it is not related to SQL logins.  Several of these services have no dependencies.  There is no problem getting to SQL.  SQL is fully up to date.
Randy DownsOWNERCommented:
You might try changing the user for some of your services to see if you can start them. Maybe use the Administrator account or a newly created account that is a member of the appropriate groups.
EugeneZ ZhitomirskySQL SERVER EXPERTCommented:
could you post details about your Microsoft Dynamics CRM version and currently installed updates?
you may need more than DB restore... Did you do this already?
also check these below articles for possible solution:

Error: Can’t Login to CRM After Windows Update to ADFS 2.0
http://www.powerobjects.com/2013/08/15/error-login-crm-after-windows-update-adfs-20/


Prepare your solutions for the update
https://msdn.microsoft.com/en-us/library/dn481577%28v=crm.6%29.aspx
Eugene_PalmerAuthor Commented:
Well I had a great post in there with all the services and their respective accounts and failures which was somehow lost.  This is CRM 2013 SP1.  That ADFS problem I hit 2 years ago (and I am in that thread)  This is different.

This is what I know-
This system was working all day Wednesday AFTER the CRM solution updates, which were done Tuesday.  So the updates are not the problem.  There are no listed Windows updates during the evening of Wednesday night-Thursday morning.  The first notable errors start with MSCRM Monitoring at 6am Thursday morning.

The last thing I've done is reset the computer account in AD, disjoin and rejoin.  Nothing changed.
EugeneZ ZhitomirskySQL SERVER EXPERTCommented:
ok:
what changed between:  Wednesday PM and  6am Thursday morning?
can you check EV system\apps logs?
do you see there anything odd?
compdigit44Commented:
Are you able to log onto the server as the service account?
Did you check the service accounts membership to make sure no one by mistake changed anything?
By any chance did you install windows updates with the CRM updates?
Also by any chance is the server a virtual server? If so you could clone it ..take it offline then do a full system restore to the online copy and if it failed revert back to the cloned offine copy
Eugene_PalmerAuthor Commented:
The server has been restored to its last good full backup, then rolled forward with both Windows and MSCRM solution updates, with SQL backups restored in sequence with the other stuff pushed forward.  So far everything is working.  It was not that fun and involved all kinds of shenanigans including cold images, native bu, bootrec, rebuilding ADFS (stale namespaces), MSDTC connections and other things.  Time for something easier than native BU, and make sure Doubletake is doing its job.

I still don't know what happened.  Nothing changed between Wednesday pm and Thursday am. Intrusion is always possible, but I don't see how.  A VSS related corruption of accounts?  

Thanks for the help, I'm going to close.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Eugene_PalmerAuthor Commented:
Restoring a backup was the only solution.  Nothing that I hadn't tried already was suggested.  No reason for the failure was put forth.
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2008

From novice to tech pro — start learning today.