Backup Exec 12.5, Win2k8 Server DC, Backup Failure error V-79-57344-65245

Unfortunately no, I do not have Trend Micro installed. I have Kaspersky, I removed it once in an attempt to troubleshoot the problem but the error still occurred. Here is a more detailed error report.

Job Completion Status
Job ended: Wednesday, February 25, 2009 at 11:39:55 AM
Completed status: Failed
Final error: 0xe000fedd - A failure occurred accessing the object list.
Final error category: Resource Errors

For additional information regarding this error refer to link V-79-57344-65245

 

Errors
Click an error below to locate it in the job log
Backup- System?State V-79-57344-65245 - A failure occurred accessing the object list.

 

Exceptions
Click an exception below to locate it in the job log
Backup- System?State The item System?State\System Files\System Files in use - skipped.

 

I've also tried a windows backup and this fails when trying to back up the system state as well. Unfortunately the logs are completely empty. The event manager has the following error.

Log Name:      Application
Source:        Microsoft-Windows-Backup
Date:          3/2/2009 1:53:30 PM
Event ID:      517
Task Category: None
Level:         Error
Keywords:      
User:          SYSTEM
Computer:      File-Server.admin.pathlabs.org
Description:
Backup started at '3/2/2009 6:51:52 PM' failed with following error code '2155348237' (Enumeration of the files failed.). Please rerun backup once issue is resolved.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Backup" Guid="{1db28f2e-8f80-4027-8c5a-a11f7f10f62d}" />
    <EventID>517</EventID>
    <Version>0</Version>
    <Level>2</Level>
    <Task>0</Task>
    <Opcode>0</Opcode>
    <Keywords>0x8000000000000000</Keywords>
    <TimeCreated SystemTime="2009-03-02T18:53:30.576Z" />
    <EventRecordID>7830</EventRecordID>
    <Correlation />
    <Execution ProcessID="181900" ThreadID="179728" />
    <Channel>Application</Channel>
    <Computer>File-Server.admin.pathlabs.org</Computer>
    <Security UserID="S-1-5-18" />
  </System>
  <EventData>
    <Data Name="BackupTime">2009-03-02T18:51:52.827Z</Data>
    <Data Name="ErrorCode">2155348237</Data>
    <Data Name="ErrorMessage">%%2155348237</Data>
  </EventData>
</Event>

                                              

1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:

Are you able to back up WMI? From the server with the system state problem: Start > Run > wmimgmt.msc

Right click on the WMI Control (Local) and click properties. Choose the Backup/Restore tab and try a backup.

I'm having the same problem as yourself with one of my customer's 2008 DCs and 12.5 which I'm currently working through with Symantec. I'll let you know when I have it resolved, hopefully you'll be able to benefit also.

HTH.

The WMI backup seemed to work with no problems.

Symantec themselves are still baffled with this error. I believe it's now gone back to development to help with resolution. I'll come back to you when I have more information.

Yes I'm not sure it's a backup exec problem as windows backup fails to back up the system state as well. Symantec closed the issue on my end. They did tell me it was a problem with system files being in use while trying to be backed up. Unfortunately they couldn't tell me which system files were the culprit. I'm going to start uninstalling programs one at a time to figure out if one of them is causing the backup issue.

Ok. Have you tried running Vssadmin from the command line to make sure that volume shadow copy is working?

From the command prompt:

vssadmin list writers

Everything should come back as stable and no errors. I've just spent another hour on the phone to Symantec about this and its been escalated again. :)

Yeah Symantec and I did the whole vssadmin littany, but I think I may have discovered what's going on. (at least on my end.) I have WSUS installed on my server, and when I removed it, the system state backed up just fine. If you have WSUS installed on the server, you might be able to get away with disabling it.

Strike that.. I spoke too soon. removing wsus didn't help

Ah ok. Unfortunately my WSUS server isn't the problematic one, hope you've got yours sorted anyway. :)

Ok so I've sort of narrowed it down to another application causing the issue. I ran SGMon.exe (it's in the Program Files\Symantec\Backup Exec  folder.) And ran another backup of the system state to a local hard drive. After about 30 minutes of reading the log, I discovered the following.

BEREMOTE: [03/10/09 15:27:00] [1207] Status 0x0000007B returned calling FindFirstFile for \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5\program files\watchguard\wsm10.2\wmserver\..\apache\bin\apache.exe in brUtil::GenerateFileList
BEREMOTE: [03/10/09 15:27:00] [1207] brUtil::SetObjectsList( ): Failed to apply the component exclusion list BEREMOTE: [03/10/09 15:27:00] [1207] Status CODE (0x0000007B) generating Component System Files file list in SHADOW::OpenComponent
BEREMOTE: [03/10/09 15:27:00] [1207] Informational: calling IVssBackupComponents::SetBackupSucceeded with status FALSE for component System Files in SHADOW::OpenComponent
BEREMOTE: [03/10/09 15:27:00] [1207] AD:Status FS_OBJECT_LIST_FAILURE (0xE000FEDD) calling FS_OpenObj in SystemState::OpenObj:172

This section leads me to believe that the issue is similar to the issue with Backup Exec and Trend Micro. (http://seer.entsupport.symantec.com/docs/310751.htm) I don't believe Volume Shadow Copy Service can recognize the path of \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy5\program files\watchguard\wsm10.2\wmserver\..\apache\bin\apache.exe

Uninstalling Watchguard Management Server allows me to back up my system state, but it's not an acceptable solution to the problem as I need to install both programs here.

ajwuk: I'd run sgmon and run a log, then search the log file for "failed" and see what you come up with. I suspect it might be something similar.

Unbelievably I've literally found the same thing myself today in the SGMon logs, I also have the WatchGuard server services running. Have just reported this back to Symantec. Will have a word with WatchGuard tomorrow too as my company is a WG partner, might be able to get something out of their tech guys.

Heh, I hope you get something out of it because I'm trying to fight someone on the WG forums and explain to them that it is indeed something to do with the WMS. Calling into support for me is a 4 hour wait, then another 4 hour wait.. It's generally painful for me to deal with it, but I think I will call them both tomorrow as well and hope that we both can get them to figure out how to fix it.

No problem, I'll report back as soon as I have anything further. Bizzare though, I'd had the WatchGuard software installed before this problem occured, it just suddenly stopped working. Have you tried reinstalling the reporting and log server components since removing them and see if the system state still backs up? I'll have to try this in the morning now as an overnight job is already running.

Very odd indeed. This is a brand new server install for me and . I wonder if maybe a BE update somehow hosed it?

Could be. I've just removed WatchGuard server services and the backup has worked for me too. Reinstalled and reconfigured log, reports, webblocker DB and quarantine server. Tried the backup again and guess what... it failed.

I've sent the report back to Symantec Enterprise support for analysis, hopefully they will come up with something useful.

Just an fyi.. This is the thread on the WatchGuard forums that I created.. General synopsis is that it's probably a windows issue, but maybe WatchGuard can help isolate the reg keys needed to change the /../ which I think is the problem.

http://www.watchguard.com/forum/default.asp?action=9&boardid=2&read=23307&fid=82

I just got off the phone with Symantec Enterprise support. They (in a nutshell) said that they have no solution for this problem, and they consider it to be WatchGuard's problem.

Another update, I provided WatchGuard with all my information at hand. The tech said he was going to forward it up to an engineer. I'm not sure where it is going to go from there. I also informed him that I know of another person who is having the same issue.

Unfortuantely I'm on site at the moment, will check tonight to see if I'd had any response from them.

I'm still trying to see if WatchGuard can help with this but Symantec are really starting to get on my nerves!

This is their response:

"It sounds as though you will need to contact the makers of the Watchguard application to find out how it can be configured to allow our application access to the system state. It would not be within our scope to override a piece of security software.
 
Please let me know if you have any questions."

I've sent them a very short response back. :)

The other part is that this is not a problem for the same scenario on Windows 2003, only Windows 2008.

Well in my experience so far both Symantec and WatchGuard are pointing fingers at Microsoft. Which may be true, but Microsoft isn't going to help in this instance, I think it's WatchGuard who needs to look into it myself. They've assigned an engineer in my instance, but I haven't heard anything back. You might want to give them a call also.

Will do. Symantec are looking into it again for me now that I've threatened to stop selling their software. :-)

Latest update... after many hours of installing/uninstalling and further testing I've now narrowed it down to the quarantine Postgre SQL server and/or reports server configurations which cause this error (either will cause the problem if the other is not present also), the log server and web blocker database have no effect.

Symantec are still being useless, although I have someone new to talk to about it now.

any news on your end ajwuk? I've sort of hit a brick wall on my end.

I also ended up installing the WSM on a different server. Seems that Watchguard nor Symantec are interested in coming up with a solution for this.

I'm just going to give the points to ajwuk since there's no real resolution here.

Thanks again for giving it a try ajwuk.

I had not seen the particular issue posted until now and had been working with Symantec one and off on this issue for the past two weeks.  We resolved this by going to the registry under:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\wmserver_service

In ImagePath correcting the path as the entry appeared incorrect.  The entry on our server contained "\..\" which was causing the backup to fail..  

We changed the path from:

"C:\Program Files\WatchGuard\wsm10.2\wmserver\..\apache\bin\apache.exe" -k runservice -f "C:\ProgramData\WatchGuard\wmserver\conf\httpd.conf"

To:

"C:\Program Files\WatchGuard\wsm10.2\apache\bin\apache.exe" -k runservice -f "C:\ProgramData\WatchGuard\wmserver\conf\httpd.conf"

After this the System State backup ran correctly and we are not noticing any issues with Watchguard Management Server either.

Hope this helps!