• Status: Solved
  • Priority: Medium
  • Security: Public
  • Views: 912
  • Last Modified:

Poletergeist in System State of server...

Environ:
VMWare Virtual 4.0
Backup Exec (BE) 12.5 SP4 (all hotfixes/updates current)
BE is on a win2k3 r2 phys server
target comp (ServT) is win2k3 r2 virtual server

Problem:
BE job will hang with status "Snapshot Processing" on ServT.  In order to stop job I have to kill 'bengine.exe' in Task Mgr(TM) & then cancel the job & restart the BE services.  I believe that there may be something going on with the System State(SS)/Shadow Copy Components(SCC) of ServT for the following reasons:  When trying to select either the SS or SCC for ServT from the Selection List in BE, the application will become unresponsive and hang.  This requires a forceful close via TM and re-opening for BE to become responsive again, however the same thing will happen if you try to select the SS or the SCC of ServT.

Things I've tried:
1) Restarting ServT until able to select the SS & SSC components in BE.  This worked for a short period of time, but it was like patching the Hoover Dam with chewing gum (was only going to work for a little bit).  This does not work now, I can select the components but the BE job will hang at Snapshot Processing.

2) Contacted Symantec Support who informed that this behavior was "queer" and the two techs I was in contact with had not seen this particular behavior from BE before..

3) The third tech I spoke with suggested I apply a windows patch on ServT that targeted the VSS Writers. That patch was from :: http://support.microsoft.com/kb/940349 ::  This fix did not work either


Note:
I was able to deselect the SS & the SCC from ServT and the BE job seems to be running fine.  However, I don't know if that clearly identifies an issue with those components or not..

Questions:
What should I do at this point?  
Is there a way to repair (or check to see if there is an actual issue with SS or SCC) these components?  
Or should I be looking in a different direction?

Any help or insight you guys can provide will be greatly appreciated

Regards,
ASB_Tech
0
ASB_Tech
Asked:
ASB_Tech
  • 14
  • 8
2 Solutions
 
Andy KeeneyCommented:
Do you have ANY other backup jobs?  For example do you employ any other backup services (VEEAM, UltraBac, Barracuda, et. al)?  I know some of them will modify the VSS to operate differently to conform to their agent(s) specific needs.
0
 
ASB_TechAuthor Commented:
Yes we also use Veeam B&R, but that backup runs without problems (usually). There are three other servers in the same job & those are backed up with Veeam as well.
0
 
ASB_TechAuthor Commented:
We have been using this combination of backup software (BE 12.5 & Veeam B&R) since Sept '09 and have been fine up until the last 2 1/2 months.  
0
NEW Veeam Backup for Microsoft Office 365 1.5

With Office 365, it’s your data and your responsibility to protect it. NEW Veeam Backup for Microsoft Office 365 eliminates the risk of losing access to your Office 365 data.

 
Andy KeeneyCommented:
we recently ran into an issue with our barracuda backup because VEEAM modified the VSS on a server.
0
 
ASB_TechAuthor Commented:
The backup solutions that we are using are Backup Exec 12.5 & Veeam Backup & Replication 5.0.2.  We aren't doing any sort of baracuda backup..

btw ServT is a win2k3 r2 server.
0
 
Andy KeeneyCommented:
my point was, that veeam modified the VSS service, which could explain the system state issue.  on your ServT, what services are running?  
0
 
ASB_TechAuthor Commented:
These are the services that are running on Servt
kyle1.bmp
kyle2.bmp
0
 
Andy KeeneyCommented:
could veeam be running at the same time Backupexec is?
0
 
ASB_TechAuthor Commented:
It does not appear to be,, the only time they look like they overlap is on Mondays when ServT is doing its full in Veeam.. All of the other nights the incremental finishes at least an hour before the BE job runs.

note:
I am taking the overlap on Mondays out of the equation this weekend.  I will be setting the jobs to do the Full (in Veeam) starting Saturday afternoon and incremental backups every other night to avoid potential conflicts between the backup software
0
 
Andy KeeneyCommented:
does veeam backup cleanly?  i know that veeam creates and locks snapshots when it tries to backup VMs
0
 
ASB_TechAuthor Commented:
If by cleanly you mean Veeam creates a snapshot on the target server & then removes that snapshot once the job is completed (and the elements of the snapshot in the datastore) then yes it does.
0
 
Andy KeeneyCommented:
after you removed the overlap, any difference?
0
 
ASB_TechAuthor Commented:
Won't know until this evening.  BE does not run on Sat/Sun evenings.  Will update tomorrow morning.
0
 
Andy KeeneyCommented:
any changes?
0
 
ASB_TechAuthor Commented:
Ran into a whole different set of issues last night..  The HD on our BE-Serv was completely full (0bytes free space) so all of our backup jobs failed.. I've cleaned up some space and we should be rolling tonight.  Will update in the morning.
0
 
ASB_TechAuthor Commented:
Unfortunately I do not have any results to post on this topic.. we had a communications hiccup last night during the backup process causing the backups to miss their windows to run... it would seem that there may be an actual poltergeist messing with me..

Will post with results/findings in the am, hopefully... *fingers crossed* that it runs to get some sort of feedback
0
 
ASB_TechAuthor Commented:
we recently ran into an issue with our barracuda backup because VEEAM modified the VSS on a server.

What was the issue that you were referencing in this?  and did you find a fix for it? if so, what were your remediation steps?
0
 
ASB_TechAuthor Commented:
Interesting.. don't know if it'll solve all of our issues or not but it can't hurt to give it a go!

Also, even without the SS & SCC selected on ServT the job was hung at 'Snapshot Processing'. I'll schedule a maintenance period tomorrow or early next week & see what effects (if any) that particular fix has.

Thanks,
ASB_Tech
0
 
Iamthecreator OMAdministrateur Systeme et ReseauxCommented:
HD filling up on BEWS Media server ?? Please ensure that you do not have debug logging enabled for BEWS services else it will fill up again. Have you attempted to backup the SYSTEM STATE, SCC on the concerned server using NTBACKUP ??
If re-registering or repairing the VSS does not help...
reinstall the latest MS SP on the concerned server.
0
 
ASB_TechAuthor Commented:
@AndyK167
I applied that fix to servT last night & we are running a backup on it this evening, so we will see if that had a positive effect on it or not.

@iamthecreator
HD filling up on BEWS  -  No, there is sufficient space to run our backups.
NTBackup Attempted - No, not as of yet. waiting to see how the results come back from the most recent fix
MS SP reinstall -  just curious but what effect do you think this will have? (currently on servT:  win2k3 r2 se sp2)
0
 
ASB_TechAuthor Commented:
No dice on the backup running successfully after the doing the fix you posted @AndyK167..

I pulled ServT out of the backup job in question and created a new job solely for ServT.  Upon creating that job and having it run the first night it never got past the queued status.  After spending Friday morning on the phone with Symantec support they came to the conclusion that the VSS writers were at fault (go figure) & were quick to pass blame to Microsoft.  While that makes sense that it's a Microsoft problem, I don't think I'll be contacting them for this issue. Customer support with contracted vendors is hard enough I don't ever relish my experiences with Microsoft CS...

After a brief departmental meeting on the issue we have decided to remove ServT from BE altogether.  We are backing up ServT in a slew of different ways & with Veeam B&R 5.0.2 we can do file level restores, which was the main purpose of Symantec for this server & we couldn't do on previous versions of Veeam B&R.  This is a 'white-flag' solution & not one that I am proud to result to, but it's not the end of the world for us to not have BE backing up this server.

I appreciate the comments provided,
ASB_Tech
0
 
ASB_TechAuthor Commented:
Issue was not resolved.

Threw in the towel on this one, as provided fixes did not work & Symantec support was more willing to pass blame then help solve the issue.
0

Featured Post

New feature and membership benefit!

New feature! Upgrade and increase expert visibility of your issues with Priority Questions.

  • 14
  • 8
Tackle projects and never again get stuck behind a technical roadblock.
Join Now