Link to home
Create AccountLog in
Avatar of Ian Taylor
Ian TaylorFlag for United Kingdom of Great Britain and Northern Ireland

asked on

dpm serious issues

Hi,

Are end of month backup to tape job started last week and on pretty much all 9 DPM Servers I am seeing the following error:

"This operation failed because of a protection agent faulire. (ID 998 Details. The device is not connected (0X8007048F) - Retry the Operation"

When looking into event viewer I can see  the following entry:

The back up to tape job failed for the following reason: (ID: 3311)
The operation failed because of a protection agent failure. (ID: 998)

Backup job for datasource: Z:\ on production server: FILE.domain.internal failed.
 Backup job failed at: 31/12/2015 02:51:53.
 Backup Type: tape backup.

there is a entry for the tape library:

hplto: The device, \Device\TapeDrive0, is not ready for access yet.

I've spent a number of hours on this and getting no closer.............I'm pretty stumped due to this happening on MOST of are DPM Servers.........nothing has changed, monthlys have been running perfectly fine all year!

I've updated to the latest firmware and drivers for the tape drive and library
Avatar of Ian Taylor
Ian Taylor
Flag of United Kingdom of Great Britain and Northern Ireland image

ASKER

Looking at the DPM logs I can see the following entry:

1184      206C      12/28      11:53:35.594      27      CommonErrorHandler.cs(173)            D9D0289A-56C3-4FCF-BC47-1FFA9559D16A      WARNING      AgentStatus[MTAForWrite] - (CommandID=MTAPerformIO, StatusReason=Error) failed with HRESULT 0x8007048F, error -2147023729.
Avatar of Dan McFadden
How do the DPM servers access the tape device?  IP?  Is the tape device functioning properly?

The error message, simply put, is saying that the server(s) cannot communicate with the tape device.

I would verify that the device is online and functioning properly.

Dan
Hi Dan,

Each DPM Server has there own tape library (MSL2024) and each server is connected via SAS.

Tape Libraries are online, I've checked the web GUI and no reported issues........
Can you verify that the server hardware and the tape device are all running the latest BIOS/firmware and drivers?

Its an issue that has been seen before.

link:  http://www.networksteve.com/enterprise/topic.php/Protection_Agent_failure_error_during_backup_to_tape_with_DPM_20/?TopicId=75676&Posts=0

Dan
Thanks but I've already come across that thread, all tape libraries have the latest firmware and drivers.

Tape libraries have been rebooted, dpm servers rebooted.............

The strange thing is its all started at the same time even though everything has been working fine all year
Any new windows update that replaced the hp tape driver? Change in the driver to the HBA or raid card the tape is attached to? I've seen that in the Symantec world. The latest doesn't mean it necessarily works.
None - Windows Updates is controlled by SCCM and we havent deployed any updated recently.

I'm stumped!
Any updates to DPM? Sccm service pack or anything?
None, is it worth opening a case with Microsoft on this?

As its the holiday period no system changes have been made
Checked again, backup to tape jobs failing all over the place with the same error
Just to update all on this, I have done the following:

1. Updated tape library to latest firmware and drives
2. Updated DPM 2012 R2 to latest Update Rollup
3. Latest Windows Update (for 2012 R2)
4. Reboot DPM Server
5. Rebooted Tape Library
6. Cleared DPM error logs
7. Done this: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_kc-0100617
8. Check Server, Tape GUI for any hardware issues, none
9. HP ProLiant Server is fully updated with latest ProLiant Pack
10. Deployed latest DPM Agent to servers
11. No reported issues via PRTG, no drops on any servers or tape libraries

After all this the problem still remains
Have you tried completely deinstalling the device and its drivers?  Verifying that the device is no present for any of the local components (DPM, etc.), disconnecting the tape library, bouncing the server and doing a re-installation of the hardware?

Dan
This article may be worth a read.  Mentions running a query to reset the info DPM stores on the tape libraries.

Link:  http://scug.be/scdpm/2009/11/25/system-center-data-protection-manager-2007-tape-problems/

Dan
Thanks Dan, will give it a read.
Cant see the query.txt in that post! :(
ASKER CERTIFIED SOLUTION
Avatar of Ian Taylor
Ian Taylor
Flag of United Kingdom of Great Britain and Northern Ireland image

Link to home
membership
Create an account to see this answer
Signing up is free. No credit card required.
Create Account
it fixed my issue