Catostrophic Failure on Backup Exec 12.5

VSS service is started and all writer show as stable. I had already checked that. It happens very sporadically (not always on the same job either) but always related to the sql backup jobs on the same sql server.

Could an sql job running on the sql server cause this issue?

In looking at the link JDettman provided, the only thing I could not find is the vsp.sys driver under the System Information - Software Environment - System Drivers. Everything else checks out but I do not understand what this is referring to: Verify the Backup to Disk folders being used are not selected for backup.

Also, Advanced Open File is not checked on either of the three backup jobs that pertain to that sql server. I initially had it checked but unchecked it to test before I posted this question. Not sure what the advantage is of having it checked.

Oh, gotcha. No, I am backing up to tape.

Any other ideas here?

What is the difference between having the open file option checked and not? What does that do?

Also, could an sql job running on the sql server at the same time as the backup cause this issue?

The problem started after our busy time of the year began (2 months ago roughly). It was backing up fine prior to that. And it is very random. I will take a look at the windows event viewer for the SQL server and see if there is anything out of the ordinary.

After taking a look at the event viewer on the SQL server, I don't see any errors around the time that the backup occurs and the catastrophic failures occurs for VSS. All I see are informational events in the application section of the event viewer like this:

Event Type:      Information
Event Source:      MSSQLSERVER
Event Category:      (6)
Event ID:      18265
Date:            2/1/2013
Time:            8:11:18 AM
User:            DCDOMAIN\besa
Computer:      MS7
Description:
Log was backed up. Database: SEGDB12, creation date(time): 2010/05/24(11:18:12), first LSN: 2345:445:1, last LSN: 2345:447:1, number of dump devices: 1, device information: (FILE=1, TYPE=VIRTUAL_DEVICE: {'SEGDB12_00__2a36d06e_7913_433e_8be6_d0228b9e9d7a_'}). This is an informational message only. No user action is required.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 59 47 00 00 0a 00 00 00   YG......
0008: 06 00 00 00 4d 00 53 00   ....M.S.
0010: 49 00 4e 00 37 00 00 00   ....7...
0018: 07 00 00 00 6d 00 61 00   ....m.a.
0020: 73 00 74 00 65 00 72 00   s.t.e.r.
0028: 00 00                     ..      

Is there any other ideas on what to check to get to the bottom of this?

Here you go:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001 Microsoft Corp.

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {5eeff731-cd9f-4a0d-992c-ea99adac0de2}
   State: [1] Stable
   Last error: No error

Writer name: 'MSDEWriter'
   Writer Id: {f8544ac1-0611-4fa5-b04b-f7ee00b03277}
   Writer Instance Id: {dd2b5a5c-33fc-4147-ab53-2c9d3c823ca3}
   State: [1] Stable
   Last error: No error

Writer name: 'Event Log Writer'
   Writer Id: {eee8c692-67ed-4250-8d86-390603070d00}
   Writer Instance Id: {449b6076-3334-40a7-8ff5-fbf7ca351c6d}
   State: [1] Stable
   Last error: No error

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {da22869d-b16b-4b56-b47f-364d7441f88f}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {71b92d71-f4bc-4828-a076-a488484f274d}
   State: [1] Stable
   Last error: No error

Writer name: 'TermServLicensing'
   Writer Id: {5382579c-98df-47a7-ac6c-98a6d7106e09}
   Writer Instance Id: {67e37aa1-4a61-488f-ad0d-0964e15210eb}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {a16c514c-ad98-4ebc-b7eb-f8f0381c5e42}
   State: [1] Stable
   Last error: No error

Writer name: 'IIS Metabase Writer'
   Writer Id: {59b1f0cf-90ef-465f-9609-6ca8b2938366}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {309e2d02-62b0-46fe-b21c-060cbd2952ac}
   State: [1] Stable
   Last error: No error

a...are you referring to the System events logs on the SQL Server? If yes, nothing shows.

b...I'm not sure how to perform this?

The Application event log only seems to be recording the SQLServer Success and Failure Audits as nothing else appears no matter how far back I go. I see a few Information logs appear after each database is finished backing up similar to the one I posted in ID: 38844404 above.

I tested the Resource Credentials and all came back successful (system state and shadow copy components stated that server credentials were used).

I ran the log backup with the System State and I recenved the same error. I then ran it without backing up the system state and it completed fine with the exception of the prompt I explain below which appears after every backup job. I also want to mention that I am backing up Full, Differential, and log and all three jobs use the same selection list.


I also notice that if I run a backup while logged in to the server, I receive a prompt stating "IDR Full Backup Success" and it mentions that it is recommended that I rerun the Intelligent Disaster Recovery Preparation Wizard and select "Copy - Disaster recovery information (.dr) files" option, to back up your disaster recovery information (.dr) file. I don't think I even choose this option??

I went into the Properties of the backup jobs in question I don't see the option to turn off IDR. Any idea how to turn it off?

There is no serial numbers and installations. Attached is a screenshot of what is available under options.
BackupExecOptions.jpg

So I tried that but attached are the only options I see to remove, etc. IDR is not on the list.
BackupExecOptions2.jpg

So I took a look at the SQL Server error logs and it is showing errors around the same time of the backup jobs are running. Below are the errors it shows:

[165] ODBC Error: 0, Unable to complete login process due to delay in opening server connection [SQLSTATE 08001]

[382] Logon to server '(local)' failed (ConnUpdateStartExecutionDate)

[382] Logon to server '(local)' failed (SaveAllSchedules)

These errors are occurring almost at every backup time. Any ideas?

Well, I reviewed my backups for the SQL Server and I am still getting the Catastrophic Failure errors on Backup Exec and the errors mentioned above on the SQL Server. Any other ideas?

I did an nslookup and it does resolve to the correct IP address for the SQL  Server from the Backup Exec Server.

Yes, Windows 2003 server has Service Pack 2. So, this issue could be correlated to other SQL server jobs being performed at the same time? The reason I ask is because we have multiple jobs running at any given point in time throughout the day.

<<Yes, Windows 2003 server has Service Pack 2. So, this issue could be correlated to other SQL server jobs being performed at the same time? >>

 If there on that server yes.  What the error is saying is that SQL is too busy to complete the login in the connection timeout alloted.

 This causes the VSSWriter to signal an error to the backup.

Jim.

Is there a way to allow either a longer timeout or allow more memory for this function?

<<Is there a way to allow either a longer timeout or allow more memory for this function? >>

 I haven't found an actual setting anywhere as yet.  My fear is it will be something like an  undocumented registry key.  But I haven't had time to dig.

Jim.

Following up to see if there is any further assistance in this.

I've looked, but have found nothing to date on getting the timeout change.

One thing we should double check though; make sure that whatever protocol the client is using (use SQL Server Configuration Manager) to see what order the client has the protocols listed in (tcpip, named pipes, etc).  Then check SQL server.  If they are in a different order, change the client to match the server.

That ensures that the client won't waste time with SQL trying to get a connection with the right protocol.

Jim.

The client being the backup server?

<<The client being the backup server? >>

 Yes.  The SQL Client configuration should match the SQL Network configuration on the SQL server.

Jim.

OK, so I accessed my SQL Server and this information is only from my SQL server:

Protocols for MSSQLSERVER (under SQL Server 2005 Network Configuration) lists Shared Memory (enabled), Named Pipes (disabled), TCP/IP (enabled), VIA (disabled).

Client Protocols (under SQL Native Client Configuration) lists Shared Memory (enabled), TCP/IP (enabled), Named Pipes (enabled) , VIA (disabled).

All of these are from top to bottom. Is this what you were referring to or are you also referring to the configurations of the SQL Server Configuration Manager on the backup Exec server?

<<All of these are from top to bottom. Is this what you were referring to or are you also referring to the configurations of the SQL Server Configuration Manager on the backup Exec server? >>

  Yes.   But with what you listed and the current enabled/disabled settings, change the order on the server side putting TCP/IP in front of Named pipes (order number will be 2 for TCPIP and 3 for named pipes).

  Do a right click on a protocol and select order to change it.  This will bring up a dialog box.  Selected TCPIP and click the up arrow.

  This change won't impact anything on the client side (the backup exec server) if by chance it talks to another SQL server box using named pipes and nothing on the server side since named pipes is disabled anyway.

 The orders will then match between the two.

Jim.

I notice that the order is default for MS SQL  server. I looked at another test SQL Server we have and the order is the same. Should I still change it?

<<I notice that the order is default for MS SQL  server. I looked at another test SQL Server we have and the order is the same. Should I still change it? >>

 Yes.  But normally you would want named pipes before TCPIP on a local lan.

 However in the case, I don't know what else is configured in your environment.  Making the change this way, nothing will be impacted except what were trying to change.

Jim.

The best solution was not presented. It was as simple as restarting the server.