Error while installing cumulative update pack 12 on SQL Server 2008 SP3

Hi,

Please advise on my problem:
I installed cumulative update 12 for sp3 on sql 2008 version on one of my production servers ,and after rebooting the server i was unable to connect to the instance.

Below is the error i see in the sql error log:

09/08/2013 02:09:16,spid9s,Unknown,Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup<c/> repair it<c/> or rebuild it. For more information about how to rebuild the master database<c/> see SQL Server Books Online.
09/08/2013 02:09:16,spid9s,Unknown,Error: 3417<c/> Severity: 21<c/> State: 3.
09/08/2013 02:09:16,spid9s,Unknown,Script level upgrade for database 'master' failed because upgrade step 'katmai_cumulative_undo.sql' encountered error 200<c/> state 7<c/> severity 25. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the 'master' database<c/> it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors<c/> take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.
09/08/2013 02:09:16,spid9s,Unknown,Error: 912<c/> Severity: 21<c/> State: 1.
09/08/2013 02:09:16,spid9s,Unknown,Invalid object name 'msdb.dbo.sysutility_mi_configuration_internal'.
09/08/2013 02:09:16,spid9s,Unknown,Error: 208<c/> Severity: 16<c/> State: 1.
09/08/2013 02:09:16,spid9s,Unknown,------------------------------------------------
09/08/2013 02:09:16,spid9s,Unknown,Starting execution of Katmai_Cumulative_undo.sql
09/08/2013 02:09:16,spid9s,Unknown,------------------------------------------------
09/08/2013 02:09:16,spid9s,Unknown,Database 'master' is upgrading script 'katmai_cumulative_undo.sql' from level 167778004 to level 2.

I tried restoring with an earlier backup with single user mode but it did not allow me and it threw an error when i tried opening a windows command prompt and type
sqlcmd.

The only thing i did was adding a trace flag -T902 to startup parameters in the properties of sql services and it connected fine.

I have left it that way but not sure how to proceed from here since if i remove the trace flag it refuses to connect.


Please help.

Thanks,
Sonali
SP_2018 .IT ConsultantAsked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Daniel_PLDB Expert/ArchitectCommented:
Hi,
Since your SQL Server is up and running with bypass script upgrade mode trace flag turned on (-T902) please connect to it and issue following command:

ALTER INDEX pk_sysutility_mi_configuration_internal_configuration_id
ON [msdb].[dbo].[sysutility_mi_configuration_internal]
REBUILD

Open in new window


Then ensure your SQL Server Agent xp's are enabled:

EXEC sp_configure 'show advanced', 1;
RECONFIGURE;
EXEC sp_configure 'allow updates', 0;
RECONFIGURE;
EXEC sp_configure 'Agent XPs', 1;
RECONFIGURE;
GO

Open in new window


Next remove trace flag and then try to reinstall CU12. If SQL Server still fails then you need to restore master database from recent backup.

You need to start it in single user mode (as you stated above), use -m option in SQL Server configuration manager. Ensure anyone will not connect to it before you - in single user only one connection is possible.

Please get back with the results.
0
SP_2018 .IT ConsultantAuthor Commented:
Hi,

Sorry for the delay in replying back.

I tried executing the Alter index statement as advised , however when i executed the command it gave me an error that the table was missing. I check for this table
in the msdb database and found a few of tables missing.
I also tried bring the server in single user mode and restore the master database
but i was not able to do it.
I then stopped the sql server agent and restored the msdb database with a previous backup
and tried restarted the services without the trace flag -T902 and it did start the service successfully this time.
I have not reapplied the SQL updates after making the change yet since i was not sure
of the result and i did not have enough downtime in the event it stopped working again.
 Please advise if i need to make any further changes.

Thanks
0
Daniel_PLDB Expert/ArchitectCommented:
Hi,

After removing 902 trace flag you let SQL Server finish scipt update mode. To be sure please verify your SQL Server errorlog. If possible please attach/post SQL Server errorlog here (please be sure to remove any sensitive data before).

Regards,
Daniel
0
SP_2018 .IT ConsultantAuthor Commented:
Hi Daniel,

Attached is the sql error log after the msdb database was restored.
Please advise.

Thanks
SQL-ERROR-log-092613.txt
0
Daniel_PLDB Expert/ArchitectCommented:
Hi,

Your SQL Server build is 10.00.5835 which is SQL Server 2008 SP3 CU10, you need 10.00.5844. Since you installed CU12 build number does not reflect this, so I think you need to consider reinstall of CU12 for this instance. Additionally in you error log there are entries for Katmai_Cumulative_undo.sql scripts being executed, it seems installation has been undone.

In your error log there is a message stating that service broker is disabled on msdb, please enable it before reinstall:

--Stop SQL Server Agent

USE master;
GO
ALTER DATABASE msdb SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
GO
USE master;
GO
ALTER DATABASE msdb SET ENABLE_BROKER ;
GO
USE master;
GO
ALTER DATABASE msdb SET MULTI_USER;
GO

--Start SQL Server Agent

Open in new window


Regards,
Daniel
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft Development

From novice to tech pro — start learning today.