Solved

Poletergeist in System State of server...

Posted on 2011-09-09
23
887 Views
Last Modified: 2013-12-01
Environ:
VMWare Virtual 4.0
Backup Exec (BE) 12.5 SP4 (all hotfixes/updates current)
BE is on a win2k3 r2 phys server
target comp (ServT) is win2k3 r2 virtual server

Problem:
BE job will hang with status "Snapshot Processing" on ServT.  In order to stop job I have to kill 'bengine.exe' in Task Mgr(TM) & then cancel the job & restart the BE services.  I believe that there may be something going on with the System State(SS)/Shadow Copy Components(SCC) of ServT for the following reasons:  When trying to select either the SS or SCC for ServT from the Selection List in BE, the application will become unresponsive and hang.  This requires a forceful close via TM and re-opening for BE to become responsive again, however the same thing will happen if you try to select the SS or the SCC of ServT.

Things I've tried:
1) Restarting ServT until able to select the SS & SSC components in BE.  This worked for a short period of time, but it was like patching the Hoover Dam with chewing gum (was only going to work for a little bit).  This does not work now, I can select the components but the BE job will hang at Snapshot Processing.

2) Contacted Symantec Support who informed that this behavior was "queer" and the two techs I was in contact with had not seen this particular behavior from BE before..

3) The third tech I spoke with suggested I apply a windows patch on ServT that targeted the VSS Writers. That patch was from :: http://support.microsoft.com/kb/940349 ::  This fix did not work either


Note:
I was able to deselect the SS & the SCC from ServT and the BE job seems to be running fine.  However, I don't know if that clearly identifies an issue with those components or not..

Questions:
What should I do at this point?  
Is there a way to repair (or check to see if there is an actual issue with SS or SCC) these components?  
Or should I be looking in a different direction?

Any help or insight you guys can provide will be greatly appreciated

Regards,
ASB_Tech
0
Comment
Question by:ASB_Tech
  • 14
  • 8
23 Comments
 
LVL 4

Expert Comment

by:Andy Keeney
Comment Utility
Do you have ANY other backup jobs?  For example do you employ any other backup services (VEEAM, UltraBac, Barracuda, et. al)?  I know some of them will modify the VSS to operate differently to conform to their agent(s) specific needs.
0
 

Author Comment

by:ASB_Tech
Comment Utility
Yes we also use Veeam B&R, but that backup runs without problems (usually). There are three other servers in the same job & those are backed up with Veeam as well.
0
 

Author Comment

by:ASB_Tech
Comment Utility
We have been using this combination of backup software (BE 12.5 & Veeam B&R) since Sept '09 and have been fine up until the last 2 1/2 months.  
0
 
LVL 4

Expert Comment

by:Andy Keeney
Comment Utility
we recently ran into an issue with our barracuda backup because VEEAM modified the VSS on a server.
0
 

Author Comment

by:ASB_Tech
Comment Utility
The backup solutions that we are using are Backup Exec 12.5 & Veeam Backup & Replication 5.0.2.  We aren't doing any sort of baracuda backup..

btw ServT is a win2k3 r2 server.
0
 
LVL 4

Expert Comment

by:Andy Keeney
Comment Utility
my point was, that veeam modified the VSS service, which could explain the system state issue.  on your ServT, what services are running?  
0
 

Author Comment

by:ASB_Tech
Comment Utility
These are the services that are running on Servt
kyle1.bmp
kyle2.bmp
0
 
LVL 4

Expert Comment

by:Andy Keeney
Comment Utility
could veeam be running at the same time Backupexec is?
0
 

Author Comment

by:ASB_Tech
Comment Utility
It does not appear to be,, the only time they look like they overlap is on Mondays when ServT is doing its full in Veeam.. All of the other nights the incremental finishes at least an hour before the BE job runs.

note:
I am taking the overlap on Mondays out of the equation this weekend.  I will be setting the jobs to do the Full (in Veeam) starting Saturday afternoon and incremental backups every other night to avoid potential conflicts between the backup software
0
 
LVL 4

Expert Comment

by:Andy Keeney
Comment Utility
does veeam backup cleanly?  i know that veeam creates and locks snapshots when it tries to backup VMs
0
 

Author Comment

by:ASB_Tech
Comment Utility
If by cleanly you mean Veeam creates a snapshot on the target server & then removes that snapshot once the job is completed (and the elements of the snapshot in the datastore) then yes it does.
0
Highfive Gives IT Their Time Back

Highfive is so simple that setting up every meeting room takes just minutes and every employee will be able to start or join a call from any room with ease. Never be called into a meeting just to get it started again. This is how video conferencing should work!

 
LVL 4

Expert Comment

by:Andy Keeney
Comment Utility
after you removed the overlap, any difference?
0
 

Author Comment

by:ASB_Tech
Comment Utility
Won't know until this evening.  BE does not run on Sat/Sun evenings.  Will update tomorrow morning.
0
 
LVL 4

Expert Comment

by:Andy Keeney
Comment Utility
any changes?
0
 

Author Comment

by:ASB_Tech
Comment Utility
Ran into a whole different set of issues last night..  The HD on our BE-Serv was completely full (0bytes free space) so all of our backup jobs failed.. I've cleaned up some space and we should be rolling tonight.  Will update in the morning.
0
 

Author Comment

by:ASB_Tech
Comment Utility
Unfortunately I do not have any results to post on this topic.. we had a communications hiccup last night during the backup process causing the backups to miss their windows to run... it would seem that there may be an actual poltergeist messing with me..

Will post with results/findings in the am, hopefully... *fingers crossed* that it runs to get some sort of feedback
0
 

Author Comment

by:ASB_Tech
Comment Utility
we recently ran into an issue with our barracuda backup because VEEAM modified the VSS on a server.

What was the issue that you were referencing in this?  and did you find a fix for it? if so, what were your remediation steps?
0
 
LVL 4

Accepted Solution

by:
Andy Keeney earned 500 total points
Comment Utility
0
 

Author Comment

by:ASB_Tech
Comment Utility
Interesting.. don't know if it'll solve all of our issues or not but it can't hurt to give it a go!

Also, even without the SS & SCC selected on ServT the job was hung at 'Snapshot Processing'. I'll schedule a maintenance period tomorrow or early next week & see what effects (if any) that particular fix has.

Thanks,
ASB_Tech
0
 
LVL 23

Expert Comment

by:Iamthecreator
Comment Utility
HD filling up on BEWS Media server ?? Please ensure that you do not have debug logging enabled for BEWS services else it will fill up again. Have you attempted to backup the SYSTEM STATE, SCC on the concerned server using NTBACKUP ??
If re-registering or repairing the VSS does not help...
reinstall the latest MS SP on the concerned server.
0
 

Author Comment

by:ASB_Tech
Comment Utility
@AndyK167
I applied that fix to servT last night & we are running a backup on it this evening, so we will see if that had a positive effect on it or not.

@iamthecreator
HD filling up on BEWS  -  No, there is sufficient space to run our backups.
NTBackup Attempted - No, not as of yet. waiting to see how the results come back from the most recent fix
MS SP reinstall -  just curious but what effect do you think this will have? (currently on servT:  win2k3 r2 se sp2)
0
 

Assisted Solution

by:ASB_Tech
ASB_Tech earned 0 total points
Comment Utility
No dice on the backup running successfully after the doing the fix you posted @AndyK167..

I pulled ServT out of the backup job in question and created a new job solely for ServT.  Upon creating that job and having it run the first night it never got past the queued status.  After spending Friday morning on the phone with Symantec support they came to the conclusion that the VSS writers were at fault (go figure) & were quick to pass blame to Microsoft.  While that makes sense that it's a Microsoft problem, I don't think I'll be contacting them for this issue. Customer support with contracted vendors is hard enough I don't ever relish my experiences with Microsoft CS...

After a brief departmental meeting on the issue we have decided to remove ServT from BE altogether.  We are backing up ServT in a slew of different ways & with Veeam B&R 5.0.2 we can do file level restores, which was the main purpose of Symantec for this server & we couldn't do on previous versions of Veeam B&R.  This is a 'white-flag' solution & not one that I am proud to result to, but it's not the end of the world for us to not have BE backing up this server.

I appreciate the comments provided,
ASB_Tech
0
 

Author Closing Comment

by:ASB_Tech
Comment Utility
Issue was not resolved.

Threw in the towel on this one, as provided fixes did not work & Symantec support was more willing to pass blame then help solve the issue.
0

Featured Post

NetScaler Deployment Guides and Resources

Citrix NetScaler is certified to support many of the most commonly deployed enterprise applications. Deployment guides provide in-depth recommendations on configuring NetScaler to meet specific application requirements.

Join & Write a Comment

Are you looking to recover an email message or a contact you just deleted mistakenly? Or you are searching for a contact that you erased from your MS Outlook ‘Contacts’ folder and now realized that it was important.
In this step by step tutorial with screenshots, we will show you HOW TO: Enable SSH Remote Access on a VMware vSphere Hypervisor 6.5 (ESXi 6.5). This is important if you need to enable SSH remote access for additional troubleshooting of the ESXi hos…
Teach the user how to join ESXi hosts to Active Directory domains Open vSphere Client: Join ESXi host to AD domain: Verify ESXi computer account in AD: Configure permissions for domain user in ESXi: Test domain user login to ESXi host:
This tutorial will walk an individual through configuring a drive on a Windows Server 2008 to perform shadow copies in order to quickly recover deleted files and folders. Click on Start and then select Computer to view the available drives on the se…

771 members asked questions and received personalized solutions in the past 7 days.

Join the community of 500,000 technology professionals and ask your questions.

Join & Ask a Question

Need Help in Real-Time?

Connect with top rated Experts

11 Experts available now in Live!

Get 1:1 Help Now