dpm serious issues


Are end of month backup to tape job started last week and on pretty much all 9 DPM Servers I am seeing the following error:

"This operation failed because of a protection agent faulire. (ID 998 Details. The device is not connected (0X8007048F) - Retry the Operation"

When looking into event viewer I can see  the following entry:

The back up to tape job failed for the following reason: (ID: 3311)
The operation failed because of a protection agent failure. (ID: 998)

Backup job for datasource: Z:\ on production server: FILE.domain.internal failed.
 Backup job failed at: 31/12/2015 02:51:53.
 Backup Type: tape backup.

there is a entry for the tape library:

hplto: The device, \Device\TapeDrive0, is not ready for access yet.

I've spent a number of hours on this and getting no closer.............I'm pretty stumped due to this happening on MOST of are DPM Servers.........nothing has changed, monthlys have been running perfectly fine all year!

I've updated to the latest firmware and drivers for the tape drive and library
Ian TaylorIT Infrastructure Architect .:|:.:|:.Asked:
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Ian TaylorIT Infrastructure Architect .:|:.:|:.Author Commented:
Looking at the DPM logs I can see the following entry:

1184      206C      12/28      11:53:35.594      27      CommonErrorHandler.cs(173)            D9D0289A-56C3-4FCF-BC47-1FFA9559D16A      WARNING      AgentStatus[MTAForWrite] - (CommandID=MTAPerformIO, StatusReason=Error) failed with HRESULT 0x8007048F, error -2147023729.
Dan McFaddenSystems EngineerCommented:
How do the DPM servers access the tape device?  IP?  Is the tape device functioning properly?

The error message, simply put, is saying that the server(s) cannot communicate with the tape device.

I would verify that the device is online and functioning properly.

Ian TaylorIT Infrastructure Architect .:|:.:|:.Author Commented:
Hi Dan,

Each DPM Server has there own tape library (MSL2024) and each server is connected via SAS.

Tape Libraries are online, I've checked the web GUI and no reported issues........
Your Guide to Achieving IT Business Success

The IT Service Excellence Tool Kit has best practices to keep your clients happy and business booming. Inside, you’ll find everything you need to increase client satisfaction and retention, become more competitive, and increase your overall success.

Dan McFaddenSystems EngineerCommented:
Can you verify that the server hardware and the tape device are all running the latest BIOS/firmware and drivers?

Its an issue that has been seen before.

link:  http://www.networksteve.com/enterprise/topic.php/Protection_Agent_failure_error_during_backup_to_tape_with_DPM_20/?TopicId=75676&Posts=0

Ian TaylorIT Infrastructure Architect .:|:.:|:.Author Commented:
Thanks but I've already come across that thread, all tape libraries have the latest firmware and drivers.

Tape libraries have been rebooted, dpm servers rebooted.............

The strange thing is its all started at the same time even though everything has been working fine all year
Casey WeaverManaged Services Windows Engineer IIICommented:
Any new windows update that replaced the hp tape driver? Change in the driver to the HBA or raid card the tape is attached to? I've seen that in the Symantec world. The latest doesn't mean it necessarily works.
Ian TaylorIT Infrastructure Architect .:|:.:|:.Author Commented:
None - Windows Updates is controlled by SCCM and we havent deployed any updated recently.

I'm stumped!
Casey WeaverManaged Services Windows Engineer IIICommented:
Any updates to DPM? Sccm service pack or anything?
Ian TaylorIT Infrastructure Architect .:|:.:|:.Author Commented:
None, is it worth opening a case with Microsoft on this?

As its the holiday period no system changes have been made
Ian TaylorIT Infrastructure Architect .:|:.:|:.Author Commented:
Checked again, backup to tape jobs failing all over the place with the same error
Ian TaylorIT Infrastructure Architect .:|:.:|:.Author Commented:
Just to update all on this, I have done the following:

1. Updated tape library to latest firmware and drives
2. Updated DPM 2012 R2 to latest Update Rollup
3. Latest Windows Update (for 2012 R2)
4. Reboot DPM Server
5. Rebooted Tape Library
6. Cleared DPM error logs
7. Done this: http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_kc-0100617
8. Check Server, Tape GUI for any hardware issues, none
9. HP ProLiant Server is fully updated with latest ProLiant Pack
10. Deployed latest DPM Agent to servers
11. No reported issues via PRTG, no drops on any servers or tape libraries

After all this the problem still remains
Dan McFaddenSystems EngineerCommented:
Have you tried completely deinstalling the device and its drivers?  Verifying that the device is no present for any of the local components (DPM, etc.), disconnecting the tape library, bouncing the server and doing a re-installation of the hardware?

Dan McFaddenSystems EngineerCommented:
This article may be worth a read.  Mentions running a query to reset the info DPM stores on the tape libraries.

Link:  http://scug.be/scdpm/2009/11/25/system-center-data-protection-manager-2007-tape-problems/

Ian TaylorIT Infrastructure Architect .:|:.:|:.Author Commented:
Thanks Dan, will give it a read.
Ian TaylorIT Infrastructure Architect .:|:.:|:.Author Commented:
Cant see the query.txt in that post! :(
Ian TaylorIT Infrastructure Architect .:|:.:|:.Author Commented:

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Ian TaylorIT Infrastructure Architect .:|:.:|:.Author Commented:
it fixed my issue
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Windows Server 2012

From novice to tech pro — start learning today.