Backup Exec tape backups failing. "An unknown error has occurred"

Our backups have been failing for the last month.

It often fails after backing up a random amount of data.
Now it fails instantly at a byte count of 0.

We are running windows 2003 standard with a HP LTO Ultrium 448 tape drive and Backup Exec 12.

I can :
Backup to disk.
Restore from tape.
Run a successful "Test run"

I have tried many solutions, namely:
Rebooting the server
Recreating the backup job
Updating the tape drivers (Both HP and Symantec, rebooting each time)
Enabling Disk caching
Disabling all HP Insight Agents
Using the cleaning tape 10+ times in a row
Using different tapes.
Update BE from 12 SP3 to 12.5 SP2
Again updating the tape drivers
More reboots

Event Type:      Error
Event Source:      Backup Exec
Event Category:      None
Event ID:      34113
Date:            25-05-09
Time:            18:21:21
User:            N/A
Computer:      SERVER2
Backup Exec Alert: Job Failed
(Server: "SERVER2") (Job: "Backup 00010") Backup 00010 -- The job failed with the following error: An unknown error has occurred.

Event Type:      Error
Event Source:      Backup Exec
Event Category:      None
Event ID:      57600
Date:            25-05-09
Time:            18:21:20
User:            N/A
Computer:      SERVER2
An unknown error occurred on device "HP 1".

0000: f4 84 00 e0               ô„.à    

I'm guessing this is a hardware issue, but don't know how to confirm that.
I have noticed that "Computer Management" / "Storage" / "Removable Storage" / "Libaries"
shows the HP tape drive with a red X. Don't know if this is relevant.

Any help would be greatly appreciated.
Who is Participating?
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Does it work with NTBackup?

Is the tape drive external, if so can you move it to a different computer to test?

Do you have another tape drive or backup device you can connect to the SCSI card?

The red X does NOT sound good.

Go into MyComputer/Properties/DeviceManager and see how the tape drive is shown in there.

DennisPostAuthor Commented:
Thanks for commenting guys!
I have already tried that twice with both the HP and Symantec drivers.
I tried it with NTBackup. No success, but that might be due to Symantec BE interferring.
I hoping the red X is only there BECAUSE Backup Exec is installed and there not necessary there and therfore is disable to save resources..... How can I confirm this?
The tape drive is internal, but I have an identical server. If need be, i'll rip out the other tape drive and try that.
All is fine in Device manager.
Newly released Acronis True Image 2019

In announcing the release of the 15th Anniversary Edition of Acronis True Image 2019, the company revealed that its artificial intelligence-based anti-ransomware technology – stopped more than 200,000 ransomware attacks on 150,000 customers last year.

Thomas RushCommented:
Go to  Find the link under "value added software" for "Library and Tape Tools", click that link, download the software.  L&TT is a free diagnostic tool for HP tape drives and libraries.

In L&TT you'll find a lot of tests for both the tape drive and media.  I'd use a scratch piece of media and run some of those tests to try to identify the problems with the tape drive (don't use media with 'real' backup data on it, as most tests will overwrite any data).

If nothing obvious jumps out at you, generate a support ticket and post it here.

One other possibility -- BackupExec may have gotten confused about the physical drive hardware/serial number/etc.   Check to make sure your backup jobs are looking for the tape drive as BackupExec is currently seeing it -- go in to a job, remove the tape drive, add the drive as a target back in to the job.   If that doesn't work, in BackupExec, try removing the tape drive and then rediscovering it.  You'll have to change your backup jobs to point to the newly discovered device.
DennisPostAuthor Commented:
Hi SelfGovern,
Thanks for commenting.

I have downloaded and installed L&TT. No devices have been discovered. "Device claimed by NT"
All Backup Exec services and the Removable Storage service were disable prior to running L&TT.
>>in BackupExec, try removing the tape drive and then rediscovering it
Could you please explain to my how I can do this.

Thomas RushCommented:
Well, if L&TT doesn't see the drive and you're sure that BE services were stopped, it sounds like there's a physical problem somewhere.  But it's really puzzling that you can successfully restore.

So this is a tough one; I'll provide these troubleshooting steps that I'd perform were I sitting in your computer room with you:

1) Without any intent to insult your intelligence or your wife's beauty or cooking skills, I'd again use the BE icon to stop all services, and then I would go to the Windows Services applet and make sure they showed "stopped".  I'd then start L&TT (I should have exited before stopping services) and perform discovery.  Does it still not see the drive?  What controllers does it see?  If it sees the drive now, get that scratch tape and run appropriate diagnostics.  Post the support ticket here.

(restart BE services)

2) I'd check cables and controller/HBA.  (This will require powering down the server; if that's not convenient, skip to #3 and check this later if needed)  If we can do a restore, this should not be the problem, but remember step one of troubleshooting electronics: "Is it plugged in?"  So verify that all cables are fully inserted, square, and tight.   Also, verify that this controller is a supported one.  I believe you said firmware was the latest, good.   Does this drive work with a different controller, or a different controller work with this drive?

3) The device-in-BE thing.  In the specifications for your backup job, you can tell it to use a particular drive (such as "HP 1"), or "All devices".  I normally recommend people identify a particular drive, but if that drive's name/ID changes for some reason, the backup jobs will fail.  Perhaps that is what happened?  If so, look at the backup job properties for the target device, and change it to "all", at least temporarily.  If a backup succeeds now, you can leave it as "all" (at least, until you add another tape device), or change it back to a single named device.

4) Background information -- this drive is inside a server, correct?  Would you describe the physical location of the server --  floor, counter, or rack; in a data center or office area; near any printers?  What lights are on or flashing on the drive when no tape is inserted?   You said you have run a cleaning tape... how old is that tape, and is it possible it's used up?

If one of the steps above works, Yippee!   If not, I'll have to know the results when you ran 1 and 3, and what specific controller you're using.  And... I like blackberry or blueberry pies, if anyone feels like cooking.  :)
DennisPostAuthor Commented:
Hi SelfGovrn,
1) I r not insulted, me is used to it :P
Originally I stopped all services in services.msc. I tried again after disabling the services. Same result no device found.
Following your advice I stopped the services via BE "Stop all services". Also no devices found.
Removavle Storage Service is also stopped.
See TLL Results.jpg.
2) I'll leave this as a last option. Unpiad overtime :-(    
3) Changing to "All" didn't work either.
Don't know if it's relevant but... When I try to backup to a blank tape I get the "unknown error", but when I try to append 28MB to a tape with 7GB free I get "e000810d - Physical Volume Library Media not found." or "e0000f16 - The operation failed to acquire the minimum number of drives and media needed."

 4) It is indeed an internal tape drive. The tower server is on the floor. No printers in the room. Whether a tape is inserted or not, only the "Ready" light is on. (No flashing).
The other lights "Clean", "Tape" and "Drive" don't burn.
Can't remember how old the tape is. I'll order a couple of new ones just in case. The times a Cleaning tape has been used up, I get a warning from BE.

Cannot catalog tapes anymore. (Maybe this is an upgrade to 12.5 issue?)
Can run an inventory.
Can (Defintitely) restore from tape using HP1. (Not just "All")
Cannot backup with NTBackup
Thanks a lot for all your time and effort you are putting in to help me resolve this!!!!!!

DennisPostAuthor Commented:
After opening as support case with HP, they sent me new Tape Drive.
My backup from yesterday is still running. It's been running for 11 1/2 hours now, "Verifying", but the job is set to cancel after 6 hours.

If I cancel the job now, will I still be able to get to the data?
btw, L&TT gives the same data as the above mentioned jpg.
DennisPostAuthor Commented:
Canceling the job didn't work.
I had to restart the services.
Final byte 1.366.592.809 byte, instead of 76,000,000,000+ bytes.
Anyone have any idea's?
DennisPostAuthor Commented:
After rebooting the server, everything is working again.

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
Storage Software

From novice to tech pro — start learning today.