Solved

Poletergeist in System State of server...

Posted on 2011-09-09
23
893 Views
Last Modified: 2013-12-01
Environ:
VMWare Virtual 4.0
Backup Exec (BE) 12.5 SP4 (all hotfixes/updates current)
BE is on a win2k3 r2 phys server
target comp (ServT) is win2k3 r2 virtual server

Problem:
BE job will hang with status "Snapshot Processing" on ServT.  In order to stop job I have to kill 'bengine.exe' in Task Mgr(TM) & then cancel the job & restart the BE services.  I believe that there may be something going on with the System State(SS)/Shadow Copy Components(SCC) of ServT for the following reasons:  When trying to select either the SS or SCC for ServT from the Selection List in BE, the application will become unresponsive and hang.  This requires a forceful close via TM and re-opening for BE to become responsive again, however the same thing will happen if you try to select the SS or the SCC of ServT.

Things I've tried:
1) Restarting ServT until able to select the SS & SSC components in BE.  This worked for a short period of time, but it was like patching the Hoover Dam with chewing gum (was only going to work for a little bit).  This does not work now, I can select the components but the BE job will hang at Snapshot Processing.

2) Contacted Symantec Support who informed that this behavior was "queer" and the two techs I was in contact with had not seen this particular behavior from BE before..

3) The third tech I spoke with suggested I apply a windows patch on ServT that targeted the VSS Writers. That patch was from :: http://support.microsoft.com/kb/940349 ::  This fix did not work either


Note:
I was able to deselect the SS & the SCC from ServT and the BE job seems to be running fine.  However, I don't know if that clearly identifies an issue with those components or not..

Questions:
What should I do at this point?  
Is there a way to repair (or check to see if there is an actual issue with SS or SCC) these components?  
Or should I be looking in a different direction?

Any help or insight you guys can provide will be greatly appreciated

Regards,
ASB_Tech
0
Comment
Question by:ASB_Tech
  • 14
  • 8
23 Comments
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36511588
Do you have ANY other backup jobs?  For example do you employ any other backup services (VEEAM, UltraBac, Barracuda, et. al)?  I know some of them will modify the VSS to operate differently to conform to their agent(s) specific needs.
0
 

Author Comment

by:ASB_Tech
ID: 36511611
Yes we also use Veeam B&R, but that backup runs without problems (usually). There are three other servers in the same job & those are backed up with Veeam as well.
0
 

Author Comment

by:ASB_Tech
ID: 36511818
We have been using this combination of backup software (BE 12.5 & Veeam B&R) since Sept '09 and have been fine up until the last 2 1/2 months.  
0
VMware Disaster Recovery and Data Protection

In this expert guide, you’ll learn about the components of a Modern Data Center. You will use cases for the value-added capabilities of Veeam®, including combining backup and replication for VMware disaster recovery and using replication for data center migration.

 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36511882
we recently ran into an issue with our barracuda backup because VEEAM modified the VSS on a server.
0
 

Author Comment

by:ASB_Tech
ID: 36511892
The backup solutions that we are using are Backup Exec 12.5 & Veeam Backup & Replication 5.0.2.  We aren't doing any sort of baracuda backup..

btw ServT is a win2k3 r2 server.
0
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36511933
my point was, that veeam modified the VSS service, which could explain the system state issue.  on your ServT, what services are running?  
0
 

Author Comment

by:ASB_Tech
ID: 36512010
These are the services that are running on Servt
kyle1.bmp
kyle2.bmp
0
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36512056
could veeam be running at the same time Backupexec is?
0
 

Author Comment

by:ASB_Tech
ID: 36512084
It does not appear to be,, the only time they look like they overlap is on Mondays when ServT is doing its full in Veeam.. All of the other nights the incremental finishes at least an hour before the BE job runs.

note:
I am taking the overlap on Mondays out of the equation this weekend.  I will be setting the jobs to do the Full (in Veeam) starting Saturday afternoon and incremental backups every other night to avoid potential conflicts between the backup software
0
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36512542
does veeam backup cleanly?  i know that veeam creates and locks snapshots when it tries to backup VMs
0
 

Author Comment

by:ASB_Tech
ID: 36512979
If by cleanly you mean Veeam creates a snapshot on the target server & then removes that snapshot once the job is completed (and the elements of the snapshot in the datastore) then yes it does.
0
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36523912
after you removed the overlap, any difference?
0
 

Author Comment

by:ASB_Tech
ID: 36524449
Won't know until this evening.  BE does not run on Sat/Sun evenings.  Will update tomorrow morning.
0
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36531161
any changes?
0
 

Author Comment

by:ASB_Tech
ID: 36531250
Ran into a whole different set of issues last night..  The HD on our BE-Serv was completely full (0bytes free space) so all of our backup jobs failed.. I've cleaned up some space and we should be rolling tonight.  Will update in the morning.
0
 

Author Comment

by:ASB_Tech
ID: 36539034
Unfortunately I do not have any results to post on this topic.. we had a communications hiccup last night during the backup process causing the backups to miss their windows to run... it would seem that there may be an actual poltergeist messing with me..

Will post with results/findings in the am, hopefully... *fingers crossed* that it runs to get some sort of feedback
0
 

Author Comment

by:ASB_Tech
ID: 36544665
we recently ran into an issue with our barracuda backup because VEEAM modified the VSS on a server.

What was the issue that you were referencing in this?  and did you find a fix for it? if so, what were your remediation steps?
0
 
LVL 4

Accepted Solution

by:
Andy Keeney earned 500 total points
ID: 36545491
0
 

Author Comment

by:ASB_Tech
ID: 36546043
Interesting.. don't know if it'll solve all of our issues or not but it can't hurt to give it a go!

Also, even without the SS & SCC selected on ServT the job was hung at 'Snapshot Processing'. I'll schedule a maintenance period tomorrow or early next week & see what effects (if any) that particular fix has.

Thanks,
ASB_Tech
0
 
LVL 23

Expert Comment

by:Rahul Misra
ID: 36547545
HD filling up on BEWS Media server ?? Please ensure that you do not have debug logging enabled for BEWS services else it will fill up again. Have you attempted to backup the SYSTEM STATE, SCC on the concerned server using NTBACKUP ??
If re-registering or repairing the VSS does not help...
reinstall the latest MS SP on the concerned server.
0
 

Author Comment

by:ASB_Tech
ID: 36580638
@AndyK167
I applied that fix to servT last night & we are running a backup on it this evening, so we will see if that had a positive effect on it or not.

@iamthecreator
HD filling up on BEWS  -  No, there is sufficient space to run our backups.
NTBackup Attempted - No, not as of yet. waiting to see how the results come back from the most recent fix
MS SP reinstall -  just curious but what effect do you think this will have? (currently on servT:  win2k3 r2 se sp2)
0
 

Assisted Solution

by:ASB_Tech
ASB_Tech earned 0 total points
ID: 36599567
No dice on the backup running successfully after the doing the fix you posted @AndyK167..

I pulled ServT out of the backup job in question and created a new job solely for ServT.  Upon creating that job and having it run the first night it never got past the queued status.  After spending Friday morning on the phone with Symantec support they came to the conclusion that the VSS writers were at fault (go figure) & were quick to pass blame to Microsoft.  While that makes sense that it's a Microsoft problem, I don't think I'll be contacting them for this issue. Customer support with contracted vendors is hard enough I don't ever relish my experiences with Microsoft CS...

After a brief departmental meeting on the issue we have decided to remove ServT from BE altogether.  We are backing up ServT in a slew of different ways & with Veeam B&R 5.0.2 we can do file level restores, which was the main purpose of Symantec for this server & we couldn't do on previous versions of Veeam B&R.  This is a 'white-flag' solution & not one that I am proud to result to, but it's not the end of the world for us to not have BE backing up this server.

I appreciate the comments provided,
ASB_Tech
0
 

Author Closing Comment

by:ASB_Tech
ID: 36895836
Issue was not resolved.

Threw in the towel on this one, as provided fixes did not work & Symantec support was more willing to pass blame then help solve the issue.
0

Featured Post

Use Case: Protecting a Hybrid Cloud Infrastructure

Microsoft Azure is rapidly becoming the norm in dynamic IT environments. This document describes the challenges that organizations face when protecting data in a hybrid cloud IT environment and presents a use case to demonstrate how Acronis Backup protects all data.

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

Veeam Backup & Replication has added a new integration – Veeam Backup for Microsoft Office 365.  In this blog, we will discuss how you can benefit from Office 365 email backup with the Veeam’s new product and try to shed some light on the needs and …
In this article, I will show you HOW TO: Suppress Configuration Issues and Warnings Alert displayed in Summary status for ESXi 6.5 after enabling SSH or ESXi Shell.
Advanced tutorial on how to run the esxtop command to capture a batch file in csv format in order to export the file and use it for performance analysis. He demonstrates how to download the file using a vSphere web client (or vSphere client) and exp…
This tutorial will walk an individual through the steps necessary to enable the VMware\Hyper-V licensed feature of Backup Exec 2012. In addition, how to add a VMware server and configure a backup job. The first step is to acquire the necessary licen…

789 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question