?
Solved

Poletergeist in System State of server...

Posted on 2011-09-09
23
Medium Priority
?
903 Views
Last Modified: 2013-12-01
Environ:
VMWare Virtual 4.0
Backup Exec (BE) 12.5 SP4 (all hotfixes/updates current)
BE is on a win2k3 r2 phys server
target comp (ServT) is win2k3 r2 virtual server

Problem:
BE job will hang with status "Snapshot Processing" on ServT.  In order to stop job I have to kill 'bengine.exe' in Task Mgr(TM) & then cancel the job & restart the BE services.  I believe that there may be something going on with the System State(SS)/Shadow Copy Components(SCC) of ServT for the following reasons:  When trying to select either the SS or SCC for ServT from the Selection List in BE, the application will become unresponsive and hang.  This requires a forceful close via TM and re-opening for BE to become responsive again, however the same thing will happen if you try to select the SS or the SCC of ServT.

Things I've tried:
1) Restarting ServT until able to select the SS & SSC components in BE.  This worked for a short period of time, but it was like patching the Hoover Dam with chewing gum (was only going to work for a little bit).  This does not work now, I can select the components but the BE job will hang at Snapshot Processing.

2) Contacted Symantec Support who informed that this behavior was "queer" and the two techs I was in contact with had not seen this particular behavior from BE before..

3) The third tech I spoke with suggested I apply a windows patch on ServT that targeted the VSS Writers. That patch was from :: http://support.microsoft.com/kb/940349 ::  This fix did not work either


Note:
I was able to deselect the SS & the SCC from ServT and the BE job seems to be running fine.  However, I don't know if that clearly identifies an issue with those components or not..

Questions:
What should I do at this point?  
Is there a way to repair (or check to see if there is an actual issue with SS or SCC) these components?  
Or should I be looking in a different direction?

Any help or insight you guys can provide will be greatly appreciated

Regards,
ASB_Tech
0
Comment
Question by:ASB_Tech
[X]
Welcome to Experts Exchange

Add your voice to the tech community where 5M+ people just like you are talking about what matters.

  • Help others & share knowledge
  • Earn cash & points
  • Learn & ask questions
  • 14
  • 8
23 Comments
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36511588
Do you have ANY other backup jobs?  For example do you employ any other backup services (VEEAM, UltraBac, Barracuda, et. al)?  I know some of them will modify the VSS to operate differently to conform to their agent(s) specific needs.
0
 

Author Comment

by:ASB_Tech
ID: 36511611
Yes we also use Veeam B&R, but that backup runs without problems (usually). There are three other servers in the same job & those are backed up with Veeam as well.
0
 

Author Comment

by:ASB_Tech
ID: 36511818
We have been using this combination of backup software (BE 12.5 & Veeam B&R) since Sept '09 and have been fine up until the last 2 1/2 months.  
0
Independent Software Vendors: We Want Your Opinion

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36511882
we recently ran into an issue with our barracuda backup because VEEAM modified the VSS on a server.
0
 

Author Comment

by:ASB_Tech
ID: 36511892
The backup solutions that we are using are Backup Exec 12.5 & Veeam Backup & Replication 5.0.2.  We aren't doing any sort of baracuda backup..

btw ServT is a win2k3 r2 server.
0
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36511933
my point was, that veeam modified the VSS service, which could explain the system state issue.  on your ServT, what services are running?  
0
 

Author Comment

by:ASB_Tech
ID: 36512010
These are the services that are running on Servt
kyle1.bmp
kyle2.bmp
0
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36512056
could veeam be running at the same time Backupexec is?
0
 

Author Comment

by:ASB_Tech
ID: 36512084
It does not appear to be,, the only time they look like they overlap is on Mondays when ServT is doing its full in Veeam.. All of the other nights the incremental finishes at least an hour before the BE job runs.

note:
I am taking the overlap on Mondays out of the equation this weekend.  I will be setting the jobs to do the Full (in Veeam) starting Saturday afternoon and incremental backups every other night to avoid potential conflicts between the backup software
0
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36512542
does veeam backup cleanly?  i know that veeam creates and locks snapshots when it tries to backup VMs
0
 

Author Comment

by:ASB_Tech
ID: 36512979
If by cleanly you mean Veeam creates a snapshot on the target server & then removes that snapshot once the job is completed (and the elements of the snapshot in the datastore) then yes it does.
0
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36523912
after you removed the overlap, any difference?
0
 

Author Comment

by:ASB_Tech
ID: 36524449
Won't know until this evening.  BE does not run on Sat/Sun evenings.  Will update tomorrow morning.
0
 
LVL 4

Expert Comment

by:Andy Keeney
ID: 36531161
any changes?
0
 

Author Comment

by:ASB_Tech
ID: 36531250
Ran into a whole different set of issues last night..  The HD on our BE-Serv was completely full (0bytes free space) so all of our backup jobs failed.. I've cleaned up some space and we should be rolling tonight.  Will update in the morning.
0
 

Author Comment

by:ASB_Tech
ID: 36539034
Unfortunately I do not have any results to post on this topic.. we had a communications hiccup last night during the backup process causing the backups to miss their windows to run... it would seem that there may be an actual poltergeist messing with me..

Will post with results/findings in the am, hopefully... *fingers crossed* that it runs to get some sort of feedback
0
 

Author Comment

by:ASB_Tech
ID: 36544665
we recently ran into an issue with our barracuda backup because VEEAM modified the VSS on a server.

What was the issue that you were referencing in this?  and did you find a fix for it? if so, what were your remediation steps?
0
 
LVL 4

Accepted Solution

by:
Andy Keeney earned 2000 total points
ID: 36545491
0
 

Author Comment

by:ASB_Tech
ID: 36546043
Interesting.. don't know if it'll solve all of our issues or not but it can't hurt to give it a go!

Also, even without the SS & SCC selected on ServT the job was hung at 'Snapshot Processing'. I'll schedule a maintenance period tomorrow or early next week & see what effects (if any) that particular fix has.

Thanks,
ASB_Tech
0
 
LVL 23

Expert Comment

by:Rahul Misra
ID: 36547545
HD filling up on BEWS Media server ?? Please ensure that you do not have debug logging enabled for BEWS services else it will fill up again. Have you attempted to backup the SYSTEM STATE, SCC on the concerned server using NTBACKUP ??
If re-registering or repairing the VSS does not help...
reinstall the latest MS SP on the concerned server.
0
 

Author Comment

by:ASB_Tech
ID: 36580638
@AndyK167
I applied that fix to servT last night & we are running a backup on it this evening, so we will see if that had a positive effect on it or not.

@iamthecreator
HD filling up on BEWS  -  No, there is sufficient space to run our backups.
NTBackup Attempted - No, not as of yet. waiting to see how the results come back from the most recent fix
MS SP reinstall -  just curious but what effect do you think this will have? (currently on servT:  win2k3 r2 se sp2)
0
 

Assisted Solution

by:ASB_Tech
ASB_Tech earned 0 total points
ID: 36599567
No dice on the backup running successfully after the doing the fix you posted @AndyK167..

I pulled ServT out of the backup job in question and created a new job solely for ServT.  Upon creating that job and having it run the first night it never got past the queued status.  After spending Friday morning on the phone with Symantec support they came to the conclusion that the VSS writers were at fault (go figure) & were quick to pass blame to Microsoft.  While that makes sense that it's a Microsoft problem, I don't think I'll be contacting them for this issue. Customer support with contracted vendors is hard enough I don't ever relish my experiences with Microsoft CS...

After a brief departmental meeting on the issue we have decided to remove ServT from BE altogether.  We are backing up ServT in a slew of different ways & with Veeam B&R 5.0.2 we can do file level restores, which was the main purpose of Symantec for this server & we couldn't do on previous versions of Veeam B&R.  This is a 'white-flag' solution & not one that I am proud to result to, but it's not the end of the world for us to not have BE backing up this server.

I appreciate the comments provided,
ASB_Tech
0
 

Author Closing Comment

by:ASB_Tech
ID: 36895836
Issue was not resolved.

Threw in the towel on this one, as provided fixes did not work & Symantec support was more willing to pass blame then help solve the issue.
0

Featured Post

Industry Leaders: We Want Your Opinion!

We value your feedback.

Take our survey and automatically be enter to win anyone of the following:
Yeti Cooler, Amazon eGift Card, and Movie eGift Card!

Question has a verified solution.

If you are experiencing a similar issue, please ask a related question

When converting a physical machine to a virtual machine using VMware vCenter Converter Standalone or vCenter Converter Enterprise, if an adapter type is not selected during the initial customization the resulting virtual machine may contain an IDE d…
Your data is at risk. Probably more today that at any other time in history. There are simply more people with more access to the Web with bad intentions.
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…
This Micro Tutorial walks you through using a remote console to access a server and install ESXi 5.1. This example is showing remote access and installation using a Dell server. The hypervisor is the very first component of your virtual infrastructu…

770 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question