SQL Server 2008 R2 cannot open MSDB, Marked SUSPECT. Error 926.

I have a client who is a legal firm running Amicus client management software which is using SQl Server 2008 R as a backend on Windows 7 PRO 64 Bit.
They started noticing data missing from the application such as current event history.  Looking to the backend database, I tried to access the database from the Management Studio, and received the following error:
***************
TITLE: Microsoft SQL Server Management Studio
------------------------------

An exception occurred while executing a Transact-SQL statement or batch. (Microsoft.SqlServer.ConnectionInfo)

------------------------------
ADDITIONAL INFORMATION:

Database 'msdb' cannot be opened. It has been marked SUSPECT by recovery. See the SQL Server errorlog for more information. (Microsoft SQL Server, Error: 926)

For help, click: http://go.microsoft.com/fwlink?ProdName=Microsoft+SQL+Server&ProdVer=10.50.1617&EvtSrc=MSSQLServer&EvtID=926&LinkId=20476
***********************

Researching this error led me to the errorlog, which is attached.   I noticed in it a number of errors opening the associated.dayabases.

I am unaware of any backup that is taken of this database.  Microsoft Backup and Restore is used for file and image backups.

I'm wondering what i can do to solve their problem.

Any insight would be very welcome.

 
Thanks!
ERRORLOG.txt
Mark LitinOwnerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mandeep SinghDatabase AdministratorCommented:

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
David Johnson, CD, MVPOwnerCommented:
Did you have a power outage recently and have write-caching enabled on the drives?

Start SQL Server with trace flag 3608 (to allow you to detach msdb). Shut down the SQL Server service and then going to the 'C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Binn' directory and from a command prompt doing 'start sqlservr.exe -c -T3608'
Use the master database and detach msdb using "sp_detach_db 'msdb'"
Rename the damaged msdb files (msdbdata.mdf and msdblog.mdf in the 'C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Data' directory)
Run the instmsdb.sql script from the 'C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Install' directory
Shutdown and restart SQL Server normally

https://social.msdn.microsoft.com/Forums/sqlserver/en-US/d9a8bd3b-017c-4ff9-b26c-71402c70bfe6/database-msdb-cannot-be-opened?forum=sqldisasterrecovery
Mark LitinOwnerAuthor Commented:
Thanks, David

Yes on both questions.

I'll perform these steps and report back.

Much obliged.

Mark
Active Protection takes the fight to cryptojacking

While there were several headline-grabbing ransomware attacks during in 2017, another big threat started appearing at the same time that didn’t get the same coverage – illicit cryptomining.

Mark LitinOwnerAuthor Commented:
Hi David

OK.  Sql Server seems to have been installed specifically to support the Amicus CRM system, and the only semblance of the path you documented (which I remember from past experiences)  is actually at
c:\apps\Microsoft Sql Server\MSSQL10_50.AMICUS\MSSQL\Binn.  I have confirmed that is the only place it installed.

So I navigated to that directory and ran "start sqlservr.exe -c -T3608", and received a Sql Server dialog with red X and OK. I've attached an image.

What do you suggest?

Thanks again.

Mark
sql05-715.PNG
Marten RuneSQL Expert/Infrastructure ArchitectCommented:
There are errorlogs Before this one. All named errorlog. Can you please attach them also.

I Believe we're looking at a failing I/O Subsystem. I would like to look at these to determine if any backups were taken.

You need to investigate this, I't can be snapshotbackups that SQL isnt aware of.

Currently, youre at a position were SQL did not start, is this correct?

Salvage what can be salvaged!
Start with copying all LDF, MDF and NDF files from this subsystem disk to Another disk. Later well investigate them to see what can be done. You may very well be at a Point were dataloss is at risk.

Please respond, and I'll try to help you.

In the mean time. set up Another SQL on a healthy system (i e OS and disk subsystem). And well use it to investigate the damnaged databases. specifically 'AmicusMedia'  and 'Amicus'.

Some system do a "backup" when the klient pushes all data out to a text file on some client. I dont know Amicus, but it might be a good idea to investigate this possibility.

Is SQL Server a speciality of yours? If not, I would suggest you hire help. Don't use the first responce google gives you, you might make things a lot worse.

AND if this is a failing subsystem, you should STOP using it after the MDF, LDF and NDF files are copied.
Before shuting it down, export all windows logs so that you can read them from Another system. I e Application, security and so forth. Also make sure you take all logs from the directory containing the errorlog you provided. In there is also 4 (I belive) files that has the extension .trn (20MB each). Those are from a trace (Quote from your errorlog: '2015-05-15 17:39:11.88 spid7s      SQL Trace ID 1 was started by login "sa".') that I can see has been running. They might have valid information in them reveiling whats been going on.

Regards Marten
PS Dont jump the gun here, this is the time to keep calm, and Think Before you act.
Vitor MontalvãoMSSQL Senior EngineerCommented:
Check if you have access to the msdb files.
Mark LitinOwnerAuthor Commented:
Thanks so much for your fantastic  input, Martin.

I handed the case off to the software company whose database was corrupted.  I really do appreciate what you're offered and will gt back to them for a status of what was done and where they currently stand.

Thanks again.  I'll keep this case as good guidance of what can be done for such SQL Server events.

Mark
Marten RuneSQL Expert/Infrastructure ArchitectCommented:
Youre welcome.

Regards Marten
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Microsoft SQL Server 2008

From novice to tech pro — start learning today.