MSDE Install Fails at InstallSQLAgentSecurity

I've seen this issue described here and in many other forums, but when the basic solutions don't help, the threads always die off.

I'm trying to install MSDE release A (other releases fail also) and I'm getting:

InstallSQLAgentSecurity failed (EDI-SERVER2,LocalSystem,87).
Action ended 13:55:22: InstallFinalize. Return value 3.

I'm installing on:  Windows Server 2003 Standard, as domain controller (not primary), with terminal services in application mode, with file and print sharing turned on

Primary domain controller is a separate computer running Windows Server 2003 for Small Business Server, communication between the two is working fine, no errors in logs, and AD updates replicate just fine.

The pdc and the install target both have 2 nics, with nic 1 on both connected to the general network and nic 2 on both connected to each other with a cross-link cable.  Most replication traffic occurs on nic 2 as it should.

The PDC is also running BackupExec (which is what I'm trying to install on the target), as well as SQL 2000 Server and MSDE, as well as Exchange.

Terminal Services is running in Application Mode
I'm installing at the console, (which shows up as a console session in TS manager)

Other applications are already installed, so I don't want to uninstall Terminal Server, and can't seem to make it just switch to admin mode (I remember this being easy to switch in 2000, but 2003 only gives me the option to uninstall it - just clicking next on components doesn't bring up a TS dialog)

I've also tried changing the instance name from bkupexec to bkupexec2, and I've tried using a blank SA password as well.

File and print sharing are definitely enabled (ie server service is started and in automatic mode and I can access shares from a workstation).

Here's the batch file I've been using to install:
change user /install
notepad "MSDE_BKUPEXEC.log"
change user /execute

have also tried the following in place of above setup line:

Here are snippets of the log file:

Error in log file before rollback:

Starting custom action CallSQLAdHlp
C:\Program Files\Microsoft SQL Server\80\Tools\Binn\sqladhlp.exe : 0
MSI (s) (E4:3C): Executing op: ActionStart(Name=RollbackAgentSecurity.2D02443E_7002_4C0B_ABC9_EAB2C064397B,,)
MSI (s) (E4:3C): Executing op: CustomActionSchedule Action=RollbackAgentSecurity.2D02443E_7002_4C0B_ABC9_EAB2C064397B,ActionType=3329,Source=BinaryData,Target=UnInstSQLAgentSecurity,)
MSI (s) (E4:3C): Executing op: ActionStart(Name=InstallAgentSecurity.2D02443E_7002_4C0B_ABC9_EAB2C064397B,,)
MSI (s) (E4:3C): Executing op: CustomActionSchedule Action=InstallAgentSecurity.2D02443E_7002_4C0B_ABC9_EAB2C064397B,ActionType=3073,Source=BinaryData,Target=InstallSQLAgentSecurity,)
MSI (s) (E4:D4): Invoking remote custom action. DLL: C:\WINDOWS\Installer\MSI153.tmp, Entrypoint: InstallSQLAgentSecurity
SOFTWARE\Microsoft\Microsoft SQL Server Setup.{E09B48B5-E141-427A-AB0C-D3605127224A}
TempFolder is C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\{E09B48B5-E141-427A-AB0C-D3605127224A}\
Loading extended custom action library C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\{E09B48B5-E141-427A-AB0C-D3605127224A}\sqlcax.dll
SOFTWARE\Microsoft\Microsoft SQL Server Setup.{E09B48B5-E141-427A-AB0C-D3605127224A}
Starting custom action InstallSQLAgentSecurity
InstallSQLAgentSecurity failed (EDI-SERVER2,LocalSystem,87).
Action ended 11:38:49: InstallFinalize. Return value 3.

Error at end of log file:

MSI (s) (9C:78): Product: Microsoft SQL Server Desktop Engine -- Installation operation failed.

MSI (s) (9C:78): Cleaning up uninstalled install packages, if any exist
MSI (s) (9C:78): MainEngineThread is returning 1603
MSI (s) (9C:6C): Destroying RemoteAPI object.
MSI (s) (9C:90): Custom Action Manager thread ending.
MSI (c) (94:C4): Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied.  Counter after decrement: -1
MSI (c) (94:C4): MainEngineThread is returning 1603

Come on experts...HELP!
Who is Participating?
SjoerdVerweijConnect With a Mentor Commented:
File and Print Sharing are not installed and/or enabled.
(Which they must be for MSDE to install).
djharrisAuthor Commented:
What makes you say that?  I go into services and look for the Server service.  It's status is "Started" and startup type is "Automatic."  I run net start Server and it tells me the service has already been started.  I am sharing files and printers and both are accessible from other workstations.  Seems to me the service is BOTH installed AND enabled.
Keep up with what's happening at Experts Exchange!

Sign up to receive Decoded, a new monthly digest with product updates, feature release info, continuing education opportunities, and more.

Sorry, that was a bit of a stock answer.

Actually, I've had this happen myself. If SQL Server is installed, bkupexec will not install properly. The only thing that worked for me was to completely uninstall any and all MSDE instances, then completely uninstall SQL Server, reboot, install bkupexec, reboot, reinstall SQL Server.
djharrisAuthor Commented:
The SQL I referred to is on another server.  The server I'm trying to install on is a different Windows 2003 install with just Terminal Server and some applications.  The applications installed are Quickbooks and Centricity, which is a medical practice manager and which accesses the SQL on the other server.
Odd, odd, odd. Maybe you should try the hotfix anyway...
djharrisAuthor Commented:'re not going to believe this.  I've been restarting my server over and over trying different things, all to no avail.  I've been double- and triple-checking that services are running, etc.

This time, however, I manually stopped the Server service (as opposed to restarting the server).  After answering yes to all the warnings (service in use, dependencies) the service finally stopped.  I then restarted the service and !!!!! MSDE installed!  

Now I don't know if the issue was because one of the peripheral services was causing a problem or because the service itself was in a funky state, but I've been restarting the server a number of times and never had success, but manually restarting the service seemed to do the trick.
[ insert Twilight zone theme here ]

As long as it works now... :-)
djharrisAuthor Commented:
Thanks for at least trying...may this "technique" can be added to your stock answers :)

I'd like to offer partial points (100 sound fair?)...can the admin do that for me?
Whatever you like. I don't know if partial points can be done, post a question in the Community Support forum with a link to this question, and what you want done.
djharris was right

I was getting the same error when installing MSDE on my Windows 2003 Server.

Starting custom action InstallSQLAgentSecurity
InstallSQLAgentSecurity failed (EDI-SERVER2,LocalSystem,87).
Action ended 11:38:49: InstallFinalize. Return value 3.

By Stopping the Server Service, which stops the Computer Browser, Net Logon, and DFS services, then start the Server Service, which doesn't start the other services, it places the Windows 2003 server in a state that allow MSDE to install.

Thanks djharris.
djharrisAuthor Commented:
Wow, thanks for the thanks!  I feel slightly less crazy now that someone else confirmed my solution.
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.