dpm serious issues

Ian Taylor
Ian Taylor used Ask the Experts™
on
Hi,

Are end of month backup to tape job started last week and on pretty much all 9 DPM Servers I am seeing the following error:

"This operation failed because of a protection agent faulire. (ID 998 Details. The device is not connected (0X8007048F) - Retry the Operation"

When looking into event viewer I can see  the following entry:

The back up to tape job failed for the following reason: (ID: 3311)
The operation failed because of a protection agent failure. (ID: 998)

Backup job for datasource: Z:\ on production server: FILE.domain.internal failed.
 Backup job failed at: 31/12/2015 02:51:53.
 Backup Type: tape backup.

there is a entry for the tape library:

hplto: The device, \Device\TapeDrive0, is not ready for access yet.

I've spent a number of hours on this and getting no closer.............I'm pretty stumped due to this happening on MOST of are DPM Servers.........nothing has changed, monthlys have been running perfectly fine all year!

I've updated to the latest firmware and drivers for the tape drive and library
Comment
Watch Question

Do more with

Expert Office
EXPERT OFFICE® is a registered trademark of EXPERTS EXCHANGE®
Ian TaylorIT Infrastructure Architect .:|:.:|:.

Author

Commented:
Looking at the DPM logs I can see the following entry:

1184      206C      12/28      11:53:35.594      27      CommonErrorHandler.cs(173)            D9D0289A-56C3-4FCF-BC47-1FFA9559D16A      WARNING      AgentStatus[MTAForWrite] - (CommandID=MTAPerformIO, StatusReason=Error) failed with HRESULT 0x8007048F, error -2147023729.
Dan McFaddenSystems Engineer

Commented:
How do the DPM servers access the tape device?  IP?  Is the tape device functioning properly?

The error message, simply put, is saying that the server(s) cannot communicate with the tape device.

I would verify that the device is online and functioning properly.

Dan
Ian TaylorIT Infrastructure Architect .:|:.:|:.

Author

Commented:
Hi Dan,

Each DPM Server has there own tape library (MSL2024) and each server is connected via SAS.

Tape Libraries are online, I've checked the web GUI and no reported issues........
Ensure you’re charging the right price for your IT

Do you wonder if your IT business is truly profitable or if you should raise your prices? Learn how to calculate your overhead burden using our free interactive tool and use it to determine the right price for your IT services. Start calculating Now!

Dan McFaddenSystems Engineer

Commented:
Can you verify that the server hardware and the tape device are all running the latest BIOS/firmware and drivers?

Its an issue that has been seen before.

link:  http://www.networksteve.com/enterprise/topic.php/Protection_Agent_failure_error_during_backup_to_tape_with_DPM_20/?TopicId=75676&Posts=0

Dan
Ian TaylorIT Infrastructure Architect .:|:.:|:.

Author

Commented:
Thanks but I've already come across that thread, all tape libraries have the latest firmware and drivers.

Tape libraries have been rebooted, dpm servers rebooted.............

The strange thing is its all started at the same time even though everything has been working fine all year
Casey WeaverManaged Services Windows Engineer III

Commented:
Any new windows update that replaced the hp tape driver? Change in the driver to the HBA or raid card the tape is attached to? I've seen that in the Symantec world. The latest doesn't mean it necessarily works.
Ian TaylorIT Infrastructure Architect .:|:.:|:.

Author

Commented:
None - Windows Updates is controlled by SCCM and we havent deployed any updated recently.

I'm stumped!
Casey WeaverManaged Services Windows Engineer III

Commented:
Any updates to DPM? Sccm service pack or anything?
Ian TaylorIT Infrastructure Architect .:|:.:|:.

Author

Commented:
None, is it worth opening a case with Microsoft on this?

As its the holiday period no system changes have been made
Ian TaylorIT Infrastructure Architect .:|:.:|:.

Author

Commented:
Checked again, backup to tape jobs failing all over the place with the same error
Ian TaylorIT Infrastructure Architect .:|:.:|:.

Author

Commented:
Just to update all on this, I have done the following:

1. Updated tape library to latest firmware and drives
2. Updated DPM 2012 R2 to latest Update Rollup
3. Latest Windows Update (for 2012 R2)
4. Reboot DPM Server
5. Rebooted Tape Library
6. Cleared DPM error logs
7. Done this: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_kc-0100617
8. Check Server, Tape GUI for any hardware issues, none
9. HP ProLiant Server is fully updated with latest ProLiant Pack
10. Deployed latest DPM Agent to servers
11. No reported issues via PRTG, no drops on any servers or tape libraries

After all this the problem still remains
Dan McFaddenSystems Engineer

Commented:
Have you tried completely deinstalling the device and its drivers?  Verifying that the device is no present for any of the local components (DPM, etc.), disconnecting the tape library, bouncing the server and doing a re-installation of the hardware?

Dan
Dan McFaddenSystems Engineer

Commented:
This article may be worth a read.  Mentions running a query to reset the info DPM stores on the tape libraries.

Link:  http://scug.be/scdpm/2009/11/25/system-center-data-protection-manager-2007-tape-problems/

Dan
Ian TaylorIT Infrastructure Architect .:|:.:|:.

Author

Commented:
Thanks Dan, will give it a read.
Ian TaylorIT Infrastructure Architect .:|:.:|:.

Author

Commented:
Cant see the query.txt in that post! :(
IT Infrastructure Architect .:|:.:|:.
Commented:
Ian TaylorIT Infrastructure Architect .:|:.:|:.

Author

Commented:
it fixed my issue

Do more with

Expert Office
Submit tech questions to Ask the Experts™ at any time to receive solutions, advice, and new ideas from leading industry professionals.

Start 7-Day Free Trial